[FEATURE] Extend virtual thread support to I/O-bound Kyuubi Server executors

[wangzhigang1999]

Code of Conduct

• I agree to follow this project's Code of Conduct.

Search before asking

• I have searched the existing issues and found no similar issue.

Describe the feature

PR #7656 added optional virtual-thread support to the Thrift Binary frontend. This issue proposes extending it to other I/O-bound Kyuubi Server executors, including HTTP frontends, operation execution, Server-to-Engine RPC, engine lifecycle helpers, metadata workers, batch submission, Data Agent, and Kubernetes operations.

Engine-side executors remain out of scope.

Motivation

Kyuubi Server is an I/O gateway. These executors spend much of their time waiting for network calls, engine responses, subprocess output, or cluster APIs, which makes them suitable for virtual threads.

Benchmark TL;DR:

We compared platform threads with All Virtual Threads (All VT) on Java Development Kit (JDK) 25.0.3 and Spark 4.2 in Kubernetes cluster mode. Each cell used a 60-second warm-up and a 300-second measurement window. Server and Engine ran in separate Pods.

Throughput is reported as completed operations per second. The p50 value is median latency. Native Memory Tracking (NMT) is the JVM facility used to measure committed native thread memory.

Workload and metric	Platform threads	All VT	Difference
Connection churn throughput	558.7 ops/s	745.0 ops/s	+33.4%
Connection churn p50	681.8 ms	411.4 ms	-39.7%
Connection churn NMT thread memory	120.8 MiB	4.6 MiB	-96.2%
Operation lifecycle throughput	108.0 ops/s	104.2 ops/s	-3.5%
Operation lifecycle NMT thread memory	124.1 MiB	5.6 MiB	-95.5%
Metadata throughput	5428.4 ops/s	5430.0 ops/s	Flat
Metadata Server CPU	4.58 cores	4.11 cores	-10.2%
Metadata container memory	3322.6 MiB	3156.8 MiB	-5.0%
Short Spark SQL throughput	13.230 ops/s	13.227 ops/s	Flat
10 GB Spark SQL throughput	0.337 ops/s	0.347 ops/s	+3.0%

All requests completed without failures. The largest gains came from connection-heavy workloads and native thread memory. Spark SQL throughput stayed close to the platform-thread baseline because query computation runs in separate Engine Pods.

The Core configuration performed slightly better than All VT in some workloads. This supports keeping the global option disabled by default and retaining component-level overrides.

Describe the solution

Add a Server-wide opt-in configuration:

kyuubi.server.virtualThreads.enabled=false

Each supported component retains its own option. Component options inherit the global value unless an administrator configures them explicitly.

The implementation preserves concurrency limits, queue capacity, rejection behavior, task serialization, metrics, and shutdown semantics. CPU-bound pools and library-owned event loops remain on platform threads. Enabling virtual threads requires JDK 21 or later.

Additional context

• This is a follow-up to [FEATURE] Support virtual threads in the server binary frontend #7655 and [KYUUBI #7655][SERVER] Support virtual threads in the binary frontend #7656.
• The Server-wide performance benchmark used JDK 25. JDK 21 has different monitor-pinning behavior and requires targeted validation for the converted paths.
• The All VT benchmark enabled every Server virtual-thread option available in that benchmark build. Low-frequency management paths were not stressed independently.

Are you willing to submit PR?

• Yes. I would be willing to submit a PR.