feat(taskdump): sample captures with per-worker budgets

[noxware]

Changes

• Add experimental task sampling before capture using per-worker budgets, alongside the existing task dumps.
• Add TaskSamplingConfig, DIAL9_TASK_SAMPLING_ENABLED, and DIAL9_TASK_SAMPLING_PER_WORKER_HZ. Existing TaskDumpConfig and idle-threshold settings retain their behavior.
• Emit separate TaskSampleEvents with capture probabilities and sampling metadata for future mixed flamegraphs.
• Refine the design doc to account for worker handoffs and the separate configuration.
• Fix cached Tokio worker metrics when a thread switches runtimes.

For reviewers

Just in case, check the changes in the design first, then prioritize:

• Public API and configuration: dial9-tokio-telemetry/src/telemetry/task_sampling_config.rs, dial9/src/env_config.rs.
• Core sampling logic: dial9-tokio-telemetry/src/task_dump/, particularly sampler.rs, worker.rs, and sampled.rs. legacy.rs preserves the existing policy. capture.rs contains shared capture mechanics.
• Worker state and metadata: dial9-tokio-telemetry/src/telemetry/recorder/runtime_context.rs.

Closes #842

[rcoh]

mostly looking at high level stuff. didnt' get super deep in the code itself

[rcoh]

I wonder if we should make TaskDumpConfigV2 no strong preference, but its possible some code would be simpler if we had two independent code paths.

[noxware]

Done. I changed direction by adapting my changes on top of fresh upstream main. That's why history got re-written. I also removed the published demo trace file as this produces a new event that is not consumed yet, and can not be produced at the same time as TaskDumps from the metrics-service example.

The old feature is called TaskDump, and the new one is called TaskSample. That's reflected in all related types, like event types or config types. (I can adjust the name).

Inside the same runtime, task dumps and task samples are exclusive, but you can have 2 different runtimes, configured differently.

[noxware]

Now that I think, maybe we should protect this config API with a experimental-task-sampling flag.

[rcoh]

this is definitely not going to work for production. Setting probability to 1 would be a large bottleneck for a production service. I think you are already looking into some options, but we need to have a cap.

[rcoh]

having this be a btree map is probably going to be a performance problem. Did you benchmark this? This should probably be a Vec instead.

Secondarily, I think having this mutex be on every Pending might end up being too hot. Would be good to have some benchmarks with this on vs. off.

One option could be a TLS that stores the current worker id and has some sort of fast path for when there wasn't a worker handoff?

[noxware]

The Mutex<BTreeMap<...> is not locked/looked-up on every Pending. I cache in TLS the WorkerTaskDumpSampler.

I'm in the middle of moving things around but look for static SAMPLER: RefCell<Option<Arc<WorkerTaskDumpSampler>>>, which is set from the existing register_worker_if_needed.

TaskDumped<F> always obtains its sampler from TLS. It also makes an Arc::clone of it (only if pointers are really different) to keep it stable during the poll.

The only Mutex::lock being called in Pendings is the internal one, inside WorkerTaskDumpSampler, which should be mostly un-contended (except in the block_in_place case which may move the worker's core to a different thread to continue operating, while the original thread also uses that Mutex).

[noxware]

Note: BTreeMap is used because register_worker_if_needed currentely receives dial9's global/displaced worker index, which has no fixed maximum (as runtimes can be dynamically created).

But, there is not reason to actually use that global id for this, I think? I can use Tokio's raw worker index without displacement, and put data into pre-allocated Box<[Option<...>]> using num_workers() as the size.

(I'm trying this change now)

[noxware]

@rcoh Done at e509bd6.

However, I spotted a weird issue with the existing worker_metrics(...) in 9f570fa, and I noticed it was affecting other feature as well, so I fixed it properly here 9b9d9f7. Regression tests validate the problem.

Edit: After history re-write, the fix is in 13f255e