After updating to 0.1.6 pods are deploying with JIT_CONFIG_ENABLED false intermittently.

[Hdom]

Hello, since we updated to v0.1.6, intermittently, runners have been deploying with the JIT_CONFIG_ENABLED environment variable set to false.

I believe this flag is true by default. I've been looking through the code and I cannot find anywhere that it gets disabled other than if you purposely set DisableJITConfig in the provider or if the provider is nil.

https://github.com/cloudbase/garm/blob/release/v0.1/runner/providers/v0.1.1/external.go#L381

Any ideas?

Here is my current provider config:

[[provider]]
name = "k8s-garm"
description = "K8s garm provider"
provider_type = "external"
  [provider.external]
  config_file = "/tmp/garm/providers.d/k8s/garm.yaml"
  provider_executable = "/tmp/garm/providers.d/k8s/garm-provider-k8s"
  environment_variables = ["KUBERNETES_"]

and here is the external provider config

kubeConfigPath: ""
runnerNamespace: "<namespace>"
flavors:
  nano:
    requests:
      cpu: 63m
      ephemeral-storage: 20Gi
      memory: 512Mi
    limits:
      ephemeral-storage: 20Gi
      memory: 512Mi
  tiny:
    requests:
      cpu: 125m
      ephemeral-storage: 20Gi
      memory: 1023Mi
    limits:
      ephemeral-storage: 20Gi
      memory: 1024Mi
  small:
    requests:
      cpu: 250m
      ephemeral-storage: 20Gi
      memory: 2047Mi
    limits:
      ephemeral-storage: 20Gi
      memory: 2048Mi
  medium:
    requests:
      cpu: 500m
      ephemeral-storage: 20Gi
      memory: 4095Mi
    limits:
      ephemeral-storage: 20Gi
      memory: 4096Mi
  large:
    requests:
      cpu: 1000m
      ephemeral-storage: 20Gi
      memory: 8191Mi
    limits:
      ephemeral-storage: 20Gi
      memory: 8192Mi
  xlarge:
    requests:
      cpu: 2000m
      ephemeral-storage: 40Gi
      memory: 16383Mi
    limits:
      ephemeral-storage: 40Gi
      memory: 16384Mi
podTemplate:
  spec:
    terminationGracePeriodSeconds: 120
    securityContext:
      fsGroup: 1001
      fsGroupChangePolicy: OnRootMismatch
    containers:
      - name: runner
        env:
          - name: HOSTNAME_LIST
            value: gcr.io,us-docker.pkg.dev
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
          - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
            value: /home/runner/k8s/job-template.yaml
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: NODE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP
          - name: ACTIONS_RUNNER_HOOK_JOB_STARTED
            value: /home/runner/k8s/pre-job-hook.sh
        volumeMounts:
          - mountPath: /runner/_work
            name: work
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            metadata:
              creationTimestamp: null
            spec:
              accessModes:
              - ReadWriteOnce
              resources:
                requests:
                  storage: 20Gi
              storageClassName: standard-rwo
              volumeMode: Filesystem

[Hdom]

I tried hardcoding the JIT_CONFIG_ENABLED to true and that seems to have made it worse, seeing this message as a result:

2025-08-14 19:11:38.017 GMT
time=2025-08-14T19:11:38.017Z level=ERROR source=/workspace/runner/metadata.go:115 msg="failed to get instance params" error="instance not configured for JIT: not found"
2025-08-14 19:11:38.017 GMT
time=2025-08-14T19:11:38.017Z level=ERROR source=/workspace/apiserver/controllers/metadata.go:63 msg="getting JIT config file" error=Unauthorized

[Hdom]

This might also be related to rate limits.

[gabriel-samfira]

GARM falls back to registration token if the GH API errors out during the creation of a JIT config for a runner. It most likely is because of rate limits. In the main branch I added the ability to show the current rate limit count and next reset time for credentials, but I don't think I added it to v0.1.x. You can check via curl, though.

[Hdom]

Hello @gabriel-samfira we requested an increase to our rate limit and we are no longing hitting any reported rate limits (core, actions etc) and yet I am still seeing tons of github operation errors happening per hour, I assume due to secondary rate limits:

and I am still seeing a decent amount of runners failing because they start up with JIT disabled, it seems this fallback mechanism is not working as expected.

[gabriel-samfira]

Then we need to see the nature of those errors. In the logs you should see an error message. Could you share those messages if they're not related to rate limits?

If we're still hitting rate limits, the fallback mechanism won't work. We could remove the fallback if the error is rate limit related.

[gabriel-samfira]

if you wish, you can join our slack channel and we can have a more interactive debug session to determine the root cause of the issues you're seeing and see if we can fix it.

[Hdom]

Then we need to see the nature of those errors. In the logs you should see an error message. Could you share those messages if they're not related to rate limits?

Sure, although I don't think I've ever seen these sort of errors output to logs, I'm going to look through the logs to see if I can find something.

If we're still hitting rate limits, the fallback mechanism won't work. We could remove the fallback if the error is rate limit related.
Yeah when it comes to rate limits a fallback mechanism should be more of a wait and retry.

I think what's happening is that it fails to retrieve a JIT configuration and then falls back to runner registration token, but, for the same rate limit reason, its unable to get one. It should then not proceed to attempt to create a runner if its not able to get either.

if you wish, you can join our slack channel and we can have a more interactive debug session to determine the root cause of the issues you're seeing and see if we can fix it.

I had no idea you had a slack, I must have scrolled right past that so many times, I will join it now.

[gabriel-samfira]

I think what's happening is that it fails to retrieve a JIT configuration and then falls back to runner registration token, but, for the same rate limit reason, its unable to get one. It should then not proceed to attempt to create a runner if its not able to get either.

The registration token doesn't get generated when we add the runner. It gets generated when the runner we spin up requests it. We never save it anywhere for security reasons. So the flow is:

• Try to generate a JIT config, if it fails, the instance object just gets a nil value JIT config
• Create the instance in the DB with pending_create status
• Consume pending_create instances and send them to the provider
• If the instance has a JIT config, we set the JitConfigEnabled flag to true
• If the JitConfigEnabled is set, the provider knows to generate proper install script. If no JIT config is available, it will request a registration token

What we can do is we can remove the fallback. The fallback was there to make GARM compatible with GHES < 3.10. But I think nobody will be running versions that old at this point. And if we fail to get a JIT config, we can bail out and avoid the above flow.

I had no idea you had a slack, I must have scrolled right past that so many times, I will join it now.

Check out the README.md file :D

[gabriel-samfira]

To recap:

• Once rate limits are reached, sending requests to the API to determine runner groups or to create JIT configs can fail. Upon failure, the runner would be created anyway without JIT config, which would prompt the instance to requests a registration token. At which point, the same rate limits would kick in and the runner would fail
• Additionally, as discussed in Idle runner scaledown causing existing jobs to be canceled #455, in large deployments with many pages of runners, we may have the situation that as we're paginating, some runners will shift from one page to another and we can get duplicates or we can omit runners. If we omit them, GARM will think we have an orphan runner and remove it, potentially canceling jobs.

A number of PRs were merged that attempt to address this:

• Fallback to registration token has been removed (Ensure scale set exists #493)
• API call to determine the runner group ID has been cached for 1 hour (Ensure scale set exists #493). If you spin up 1000 runners in a runner group per hour, you will only consume 1 API call for that runner group. This should help with rate limits.
• Listing all runners now tries to use the scaleset API on GitHub (which is not paginated), with a fallback to the regular API (Attempt to use the scalset API and caching #495).

We need to move away from attempting to keep state in sync by abusing the github APIs. I am considering injecting a GARM agent in the final runners and use it as a wrapper for the runner service. The agent would stay connected to GARM via gRPC/WebSockets and report back to GARM when the runner exits due to a job being run. When/If we go that route, we will no longer need to poll the github API. Added bonus is that we would also be able to get some real time metrics from the actual instances.