[Bug]: GitHub App authentication breaks on self-hosted runners

[DF1229]

AL-Go version

9.0

Describe the issue

Without adjusting any settings - other than githubRunner - our authentication with github through our own GH-app fails, simply by switching from windows-latest to self-hosted. Specifically during the "Download Project Dependencies" step, which we have setup to download one of our AppSource apps from its own repo (from the same GH org) using the probing paths setting.

The error seems to be an authorization error at first (http 401), but when running the connection from my own PC (on the same network as the self-hosted runners) the github API returns unprocessable (http 422).

This has always worked just fine in the past but since we have moved from GH-hosted to self-hosted this has been giving us trouble. Up until about last week I thought this might be a network-related issue, but we have since gotten a new (way faster) connection and we experience no other networking issues.

Our ghTokenWorkflow secret is stored in Azure, but this connection seems to be fine, and the secret value is retrieved and masked.

Attaching two log downloads, one for a failed self-hosted PR build, and one for a cancelled GH-hosted PR build where the download dependencies succeeded.

Expected behavior

Authentication should keep working... right?

Steps to reproduce

1. PTE repo with probing paths setup to another AL-Go repo for an AppSource app
2. Setup authentication to GH through a GH-app, with the ghTokenWorkflow secret stored in Azure
3. Have a successful CI/CD run in the AppSource app repo
4. Create a PR for the PTE, and run the PR build

Additional context (logs, screenshots, etc.)

Self-hosted PR build (failed).zip

GH-hosted PR build (cancelled).zip

Also see #2271

[aholstrup1]

And when you switch back to GitHub hosted runners again, it starts working again? Just to rule out that something expired JUST as you switched. Unlikely... But just making sure :)

We might need a little more information from your side to get to the bottom of this. Can I ask you for two things

1. Can you rerun the self-hosted PR Build with debug logging enabled? And then attach those logs here

2. When I ask copilot the only guess it has is that maybe it's due to clock drift. The GitHub documentation around generating the JWT we use in AL-Go does actually mention clock drift as a concern. Could you try running the following on your self-hosted runner? To rule out whether clock drift could be the culprit in this case

function Test-ClockDrift {
    1..5 | ForEach-Object {
        $resp = Invoke-WebRequest -Uri "https://api.github.com/zen?cb=$([guid]::NewGuid())" `
                  -Method Get -UseBasicParsing -Headers @{ 'Cache-Control' = 'no-cache' }
        $offset = ([datetime]::UtcNow - [datetime]::Parse($resp.Headers['Date']).ToUniversalTime()).TotalSeconds
        'GitHub offset (s): {0:N1}' -f $offset
        Start-Sleep -Milliseconds 500
    }
    w32tm /stripchart /computer:time.windows.com /samples:3
}

Test-ClockDrift

[DF1229]

And when you switch back to GitHub hosted runners again, it starts working again? Just to rule out that something expired JUST as you switched. Unlikely... But just making sure :)

It does indeed just start working again, I have also wondered about expirations but I renewed the ghTokenWorkflow based on a new private key file just 16 days ago. Azure connection seems to be working just fine too, since all the secrets and whatnot are loaded in.

1. Can you rerun the self-hosted PR Build with debug logging enabled? And then attach those logs here

logs_74744111884.zip

3. When I ask copilot the only guess it has is that maybe it's due to clock drift. The GitHub documentation around generating the JWT we use in AL-Go does actually mention clock drift as a concern. Could you try running the following on your self-hosted runner? To rule out whether clock drift could be the culprit in this case>

function Test-ClockDrift {
    1..5 | ForEach-Object {
        $resp = Invoke-WebRequest -Uri "https://api.github.com/zen?cb=$([guid]::NewGuid())" `
                  -Method Get -UseBasicParsing -Headers @{ 'Cache-Control' = 'no-cache' }
        $offset = ([datetime]::UtcNow - [datetime]::Parse($resp.Headers['Date']).ToUniversalTime()).TotalSeconds
        'GitHub offset (s): {0:N1}' -f $offset
        Start-Sleep -Milliseconds 500
    }
    w32tm /stripchart /computer:time.windows.com /samples:3
}

Test-ClockDrift

Interesting, haven't considered this before... Also not sure how to interpret the results, but the offsets don't seem outrageous to me (assuming milliseconds).

EDIT: Just saw it uses seconds, but even then I doubt the JWT expires that quickly... right?

[aholstrup1]

I think the acceptable range is Now - 10 seconds to Now + 10 minutes, so it depends if your clock is ahead or behind. In this case it seems like it's ahead by around 6 seconds. That's close to the 10 second limit but shouldn't be enough for the clock drift to be an issue.

Just to rule it out completely, can you try running the following in an elevated shell? This will force a resync of the clock.

w32tm /query /status
w32tm /resync /force
w32tm /query /status

And then run this again to see if it closes the offset

function Test-ClockDrift {
    1..5 | ForEach-Object {
        $resp = Invoke-WebRequest -Uri "https://api.github.com/zen?cb=$([guid]::NewGuid())" `
                  -Method Get -UseBasicParsing -Headers @{ 'Cache-Control' = 'no-cache' }
        $offset = ([datetime]::UtcNow - [datetime]::Parse($resp.Headers['Date']).ToUniversalTime()).TotalSeconds
        'GitHub offset (s): {0:N1}' -f $offset
        Start-Sleep -Milliseconds 500
    }
    w32tm /stripchart /computer:time.windows.com /samples:3
}

Test-ClockDrift

Can you include the logs for that here? And then retry your build to see if the authentication passes now?

If that doesn't work, we'll have to look elsewhere :)

[DF1229]

Every time I open an issue here, I learn something 😄

Turns out the W32Time service on our runner was disabled, causing the time to drift. I couldn't access the runner last Friday (or over the weekend), and by the time I got to it this morning the offset was averaging roughly 7.5 seconds.

GitHub offset (s): 7.6
GitHub offset (s): 7.3
GitHub offset (s): 7.9
GitHub offset (s): 7.6
GitHub offset (s): 7.3
Tracking time.windows.com [104.40.149.189:123].
Collecting 3 samples.
The current time is 22/06/2026 09:37:39.
09:37:39, d:+00.0053303s o:-07.1619123s  [       *                   |                           ]
09:37:41, d:+00.0059209s o:-07.1618236s  [       *                   |                           ]
09:37:43, d:+00.0050714s o:-07.1618391s  [       *                   |                           ]

Re-enabled the service with the default startup parameters, forced a sync, and the builds are working again!

The offset with Github is still between 0.1-1.0 seconds, which I find curious, but the important thing is that it's all working again.

GitHub offset (s): 0.7
GitHub offset (s): 0.5
GitHub offset (s): 0.2
GitHub offset (s): 0.9
GitHub offset (s): 0.5
GitHub offset (s): 0.2
GitHub offset (s): 0.9
GitHub offset (s): 0.6
GitHub offset (s): 0.3
GitHub offset (s): 1.0
Tracking time.windows.com [104.40.149.189:123].
Collecting 5 samples.
The current time is 22/06/2026 09:56:13.
09:56:13, d:+00.0051705s o:+00.0089932s  [                           *                           ]
09:56:15, d:+00.0055944s o:+00.0094485s  [                           *                           ]
09:56:17, d:+00.0127161s o:+00.0056549s  [                           *                           ]
09:56:19, d:+00.0050325s o:+00.0095797s  [                           *                           ]
09:56:21, d:+00.0051394s o:+00.0093428s  [                           *                           ]

Thanks again for your help @aholstrup1!

[aholstrup1]

@DF1229 Thanks for checking! I'm happy that turned out to be the reason, otherwise I didn't really have any other theories 😆

I also learned that GitHub recommends allowing Now - 60 seconds rather than Now - 10 seconds when creating JWTs. I think that's a small improvement we could make in AL-Go to minimize the risk of running into these situations in the future (https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/generating-a-json-web-token-jwt-for-a-github-app#about-json-web-tokens-jwts).

I'll prepare a change for that 👍