vSphereCloud: add optional vCenter session keep-alive / connection pool

[jimklimov]

Augmented by AI (Claude)

Every vSphere API call — find template, clone VM, configure, power on, wait for tools, revert snapshot, delete clone — previously performed a full login → operation → logout cycle. With multiple agents being provisioned concurrently this produced dozens of authentication round-trips per second against vCenter, increasing latency and load.

Three call sites additionally leaked sessions by never calling disconnect() at all:

• vSphereCloudSlave.doTestConnection,
• vSphereCloudSlaveComputer.getVMInformation,
• vSphereCloudLauncher.revertVM

A single long-lived vSphere session per vSphereCloud instance. Callers continue to call vSphereInstance() / disconnect() exactly as before; when the pool is active, disconnect() becomes a no-op and the pool manages the real session lifecycle via background maintenance:

• Health check (configurable interval): issues a lightweight currentTime() SOAP call; reconnects automatically on failure.
• Session age limit: proactively restarts the session after N seconds to comply with vCenter session-duration policies.
• Use-count limit: restarts after N acquisitions, independently of age.
• Idle disconnect: logs out after N seconds of inactivity; the next acquire() transparently re-establishes the session.

Each feature is independently enabled/disabled via its own integer setting; 0 means the feature is disabled. The pool is disabled entirely by default (useConnectionPool = false) to preserve the existing stateless behaviour.

The pool holds ONE connection per cloud instance (not a pool of N connections), because VSphere itself is stateless — its session token is immutable and all operations create fresh ServiceInstance objects, so concurrent callers share the same instance safely.

• volatile boolean pooled field: when true, disconnect() returns immediately instead of calling logout().
• markAsPooled() / forceDisconnect(): package-private methods called only by VSphereConnectionPool to control the real logout.
• isSessionAlive(): public; issues currentTime() and returns true/false — used by the pool health-check thread. Note: the correct yavijava 9.0.2 method name is currentTime(), not getCurrentTime() which does not exist in this version.

Five new DataBoundSetter fields (all default 0 / false):

| useConnectionPool | boolean | master toggle |
| poolHealthCheckIntervalSecs | int | seconds; 0 = no health checks |
| sessionMaxAgeSecs | int | seconds; 0 = never restart |
| sessionMaxUses | int | count; 0 = never restart |
| poolIdleTimeoutSecs | int | seconds; 0 = keep alive forever |

vSphereInstance() routes through getOrCreatePool() when the toggle is on. Each setter calls resetPool() so that live reconfiguration via JCasC or the UI immediately takes effect with a fresh session.

• config.jelly: five new entries inside <f:advanced>
• help-useConnectionPool.html: overview + full JCasC YAML example
• help-poolHealthCheckIntervalSecs.html
• help-sessionMaxAgeSecs.html
• help-sessionMaxUses.html
• help-poolIdleTimeoutSecs.html

clouds:

  - vSphere: vsDescription: "My vCenter" vsConnectionConfig: vsHost: "https://vcenter.example.com" credentialsId: "vcenter-creds" useConnectionPool: true
      poolHealthCheckIntervalSecs: 60    # 0 = disabled
      sessionMaxAgeSecs: 3600            # 0 = never restart on age
      sessionMaxUses: 500                # 0 = never restart on count
      poolIdleTimeoutSecs: 300           # 0 = keep alive indefinitely

Three call sites that obtained a VSphere instance without ever calling disconnect() have been fixed with try/finally blocks:

• vSphereCloudSlave.DescriptorImpl.doTestConnection() — was creating up to two sessions (one for VM lookup, one for snapshot lookup) and leaking both; refactored to a single session covering both lookups.

• vSphereCloudSlaveComputer.getVMInformation() — leaked the session after fetching VM properties for the computer detail panel.

• vSphereCloudLauncher.revertVM() — leaked the session used to resolve the snapshot reference before reverting.

• VSphereConnectionPoolSettingsTest (9 unit tests, no Jenkins needed): verifies pool is off by default, integer settings default to zero, setter round-trips, pool shutdown is safe/idempotent, negative constructor arguments are clamped to zero.

• configuration-as-code-with-pool.yml: new JCasC fixture with all five pool fields set to non-default values.

• ConfigurationAsCodeTest: two new methods — pool_is_disabled_by_default_when_not_specified_in_yaml and should_load_pool_configuration_from_yaml.

Testing done

Will be tested in-house in the coming weeks/months.

Submitter checklist

• Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
• Ensure that the pull request title represents the desired changelog entry
• Please describe what you did
• Link to relevant issues in GitHub or Jira
• Link to relevant pull requests, esp. upstream and downstream changes
• Ensure you have provided tests that demonstrate the feature works or the issue is fixed

[jimklimov]

Ran some experiments with this PR's result, seems promising:

• auto-logout / re-login works
• according to vCenter Events log for the user account, a lot more API operations are grouped (seen with vim.event.UserLogoutSessionEvent entries)
• a typical VM clone/reconfigure/start workflow no longer involved a dozen sessions with 5 operations each over a few seconds (or same second) taking a screenful of vCenter log :)
• "proactive reconnect" and "session established" events are seen in Jenkins log, and correlate with vCenter log entries

Some disappointments were also there:

I've initially tried to enable several options at once:

• Pool health check interval in seconds: 10
• Pool session max age in seconds: 60
• Pool session max uses: 0 (default)
• Pool idle disconnect timeout in seconds: 120

With these settings, it reconnected every 70-75 seconds (not 60), consuming about 20 API operations during each session even if there were no requests from the pipelines. It never went into idle disconnect handling.

• I suppose the API operations were due to health check, and never-idle because the session was always younger than 120s?
  • TODO: For these timeout purposes, account only operations received via plugin API, not those initiated internally like login or health polling
  • TODO: Follow the requested timeouts more strictly (e.g. via alarms) if possible?

I then toned down the settings to:

• Pool health check interval in seconds: 0
• Pool session max age in seconds: 360
• Pool session max uses: 0 (default)
• Pool idle disconnect timeout in seconds: 120

However it seems to have been ignored during the continuing lifetime of the Jenkins server. While there were logged o.jenkinsci.plugins.vSphereCloud#InternalLog: STARTING VSPHERE CLOUD messages for every edit of Cloud Provider settings, there were still reconnections every ~70s with ~20 API operations each.

• TODO: Take configuration changes into account more actively

Ultimately I've restarted the Jenkins instance, and it seemed to have the desired effect:

• First there was no log from the plugin
• I've opened its cloud config page and applied it (no changes)
• It logged "STARTING VSPHERE CLOUD"
• No proactive sessions appeared
• Later I started the test job dealing with a VM
• It logged "session established" at 14:21:53
• ...worked with the VM until 14:22:01 (per vCenter log)
• ...and o.j.p.v.t.VSphereConnectionPool#scheduledMaintenance: vSphere connection pool [https://vcenter.dayjob.com]: disconnecting - idle for 142s (limit 120s) at 14:24:23 - so time accounting matches

Again, the requested idle time (120s) was exceeded, but at least did work this time. There was a logout event in vCenter log, and there were no subsequent interactions.

One nuance that needs more attention here - when Jenkins was being restarted, there was no logout from vCenter. However as the Jenkins JVM was SIGTERM'ed and by itself just died and did not log any graceful teardown log lines, I can't say yet whether the plugin handles it cleanly or not when all goes well.

[jimklimov]

Continuing with the experiment, I've updated the configuration with a health-check again:

• Pool health check interval in seconds: 10
• Pool session max age in seconds: 360
• Pool session max uses: 0 (default)
• Pool idle disconnect timeout in seconds: 120

Again, in Jenkins log the "STARTING VSPHERE CLOUD" line appeared and nothing else per se. Only after I re-ran the test job, it added "session established" at 14:49:09, and "disconnecting - idle for 143s (limit 120s)" at 14:51:38. This was without the Jenkins controller restart (so maybe the plugin effectively ignored the config change).

A subsequent safeExit did log Jenkins teardown, but no messages from this plugin (no active session at that time either).

[jimklimov]

Retested with starting a job (and so a vCenter session) and subsequent safeExit - there was no logout.

• TODO: Handle Jenkins shutdown in the plugin

[jimklimov]

Retrying with shorter-lived settings:

• Pool health check interval in seconds: 10
• Pool session max age in seconds: 120
• Pool session max uses: 0 (default)
• Pool idle disconnect timeout in seconds: 60

Ran the test job, "session established" at 15:00:57, last operation in vCenter logged at 15:01:04 and job finished at 15:01:09.
Idle timeout honoured: "disconnecting - idle for 82s (limit 60s)" at 15:02:26.

Notably, the settings were modified and applied (logging "STARTING VSPHERE CLOUD") after starting Jenkins but before running the job, and took hold in the same server uptime.

The session ended on vCenter side of the log too, with 40 "API invocations" vs. 52 at 14:58:38 or 22 at 14:24:23 with earlier tests. No subsequent re-login logged - by neither vCenter nor Jenkins.

[jimklimov]

Updated the above settings to poll every 20 seconds, restarted Jenkins and initiated the test job right away (without any visit to the cloud config page in this new uptime).

This time there is no "STARTING VSPHERE CLOUD" in the log at all; "session established" at 15:11:31, job ended at 15:11:43, and "disconnecting - idle for 83s (limit 60s)" at 15:13:00 with 30 logged API calls (so apparently health-check polling counts among those, probably). No subsequent re-login.

So for the idle time vs. session max time (and probably max operations per session), we need to be careful about accounting the last request from plugin consumers, and ignore the (sub-)session start time - if it is not the first loop cycle.

[jimklimov]

Modified the config to check also API usage (max ops per session):

• Pool health check interval in seconds: 20
• Pool session max age in seconds: 120
• Pool session max uses: 12
• Pool idle disconnect timeout in seconds: 60

As before, this change did not take hold in the same Jenkins uptime (since a job was executed).

After a restart of Jenkins, started the test job right away.

• "session established" at 15:21:16
• job ended at 15:21:30
• no Jenkins nor vCenter logs about logging out (due to API operations)
• "disconnecting - idle for 82s (limit 60s)" at 15:22:45, after 30 API invocations

[jimklimov]

Retry without max age, only max usage:

• Pool health check interval in seconds: 25
• Pool session max age in seconds: 0
• Pool session max uses: 8
• Pool idle disconnect timeout in seconds: 60

Apply, restart,.. but before I re-ran there was some delay due to a coffee break - and vCenter logged that "user ... logged out (login time: Tuesday, July 14, 2026 4:55:51 PM CEST, number of API invocations: 29, user agent: Java/25.0.2)" at "07/14/2026, 5:26:40 PM" (local time). This probably corresponds to earlier test with safeExit - vCenter did reap the un-utilized session after some half an hour. Good :)

• TODO: Now need to test how the plugin behaves against that, with a longer-lived session permission; would it reconnect?

Re-running the test job again did not interrupt sessions due to API invocations, plugin disconnected itself some exactly 90 seconds (idle 83 of 60) after establishing the session, consuming 28 API invocations (probably one less health-check than before with 30 seen several times).

• --- [ ] TODO: Fix max-calls-per-session constraint, does not seem to work at all. ---

UPDATE: Retrying with extreme low value of "Pool session max uses: 2" to check whether it is not the case of the job does not fit into the allowance while polling or whatever is not accounted into it. And bumped health check interval to 35. Hooray:

• "session established" at 15:39:47.317
• "restarting session - max uses reached" at 15:39:48.462 with 8 API invocations logged by vCenter (2 logical operations as plugin step calls: powered off and deleted the old VM)
• "session established" at 15:39:48.690
• job ended at 15:40:00.110
• "disconnecting - idle for 82s (limit 60s)" at 15:41:16.543 (hard to say if this was considered after the initial job start, or after reconnection... them being under 2s apart), consumed 21 more API invocations for two more plugin step calls to clone and reconfigure the VM (logging a lot more lines in vCenter to assign BIOS and instance UUIDs, clone data on the storage LUN, apply reconfiguration, etc.)

Not looking at code, I suppose this session constraint check happens then a plugin operation starts, so code proceeds atomically in same session, regardless of the API consumption (which is probably a good thing so it is not broken mid-way). The next operation may find that the allowance is overspent, and would start another session.

• TODO: Check if the implementation is about vCenter API calls or plugin API/step calls? Probably it is worth leaning toward the latter, and updating the documentation/help messages accordingly.

[jimklimov]

At some point I've checked that running a job, letting the session expire by timeout (or now also call count), and starting another in the same Jenkins uptime is seamless, jobs succeed. What about running several jobs while the same session is up?

With same uptime as the previous test, but some 17 minutes after that ended and I was documenting the results, I've gone to edit cloud config to disable the call count constraint. Either the edit worked without Jenkins restart, or the constraint only works once, but subsequent job runs did not reconnect due to "max uses".

• earlier disconnection at 15:41:16.543
• config applied and "STARTING VSPHERE CLOUD" at 15:58:07.545
• job started, "session established" at 15:58:26.014
• job ended at 15:58:37.548
• another job started, ended at 15:58:59.976
• another job started, ended at 15:59:20.160 (last operation in vCenter at 15:59:14)
• "disconnecting - idle for 71s (limit 60s)" at 16:00:25.777, consuming 66 API invocations overall per vCenter logs