fix(kbagent): reap zombie processes

[leon-ape]

What

• install tini-static in the tools image and keep it at /bin/tini-static
• re-exec kbagent under the sibling tini-static binary when kbagent starts as PID 1
• copy both kbagent and tini for custom action images, while deferring the legacy init image/command pair only on existing Pods
• keep tools-image-only upgrades on the existing image-only in-place update path
• add unit and upgrade compatibility coverage for the bootstrap and rollout behavior

Why

kbagent currently runs as PID 1 and does not provide init-process semantics. Orphaned action descendants can therefore remain as zombie processes. Running kbagent under tini ensures orphan adoption and reaping while preserving the existing Pod command and upgrade behavior.

The desired InstanceSet template always contains the new tools image and two-file copy command. Existing custom-image Pods keep their old init image and command as an atomic pair without being restarted; newly created or recreated Pods use the new template and copy both binaries. This compatibility decision is handled by the InstanceSet update path, so explicit restarts, config restarts, application changes, and other recreate-worthy updates retain their normal behavior.

Verification

• go test ./cmd/kbagent ./pkg/kbagent/service
• component and workload controller envtest suites
• focused InstanceSet image-only and deferred init migration tests
• go vet ./cmd/kbagent ./pkg/controller/component ./controllers/apps/component ./pkg/controller/instanceset
• tools image build
• Docker regression with 2,000 orphan-process actions, timeout process-group cleanup, and SIGTERM forwarding checks
• git diff --check

Closes #10762

[apecloud-bot]

Auto Cherry-pick Instructions

Usage:
  - /nopick: Not auto cherry-pick when PR merged.
  - /pick: release-x.x [release-x.x]: Auto cherry-pick to the specified branch when PR merged.

Example:
  - /nopick
  - /pick release-1.1

CLA Recheck Instructions

Usage:
  - /recheck-cla: Trigger a re-check of CLA status for this pull request.
Example:
  - /recheck-cla

[codecov]

Codecov Report

❌ Patch coverage is 84.21053% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.96%. Comparing base (9ee2a73) to head (6ecf81e).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/controller/instanceset/in_place_update_util.go	79.54%	6 Missing and 3 partials ⚠️
pkg/controller/instanceset/instance_util.go	57.14%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10763      +/-   ##
==========================================
+ Coverage   64.56%   64.96%   +0.39%     
==========================================
  Files         521      506      -15     
  Lines       63343    63393      +50     
==========================================
+ Hits        40897    41181     +284     
+ Misses      18804    18578     -226     
+ Partials     3642     3634       -8     

Flag	Coverage Δ
unittests	64.96% <84.21%> (+0.39%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cjc7373]

maybe it's simpler to use https://github.com/ramr/go-reaper

[leon-ape]

Thanks for the suggestion, @cjc7373. We evaluated go-reaper, but decided to keep tini for this fix.

The in-process Reap() mode calls wait4(-1) and can race with kbagent's existing exec.Cmd.Wait() calls for action processes, producing ECHILD/wait: no child processes. go-reaper documents this caveat for programs that manage child processes directly, and there is a matching report in ramr/go-reaper#2.

Its forked mode avoids that ownership race, but currently does not forward termination signals to the kbagent child and always exits with code 0 instead of propagating the child's exit status. Both behaviors are required for a container PID 1.

Making the in-process approach reliable would require replacing kbagent's action execution with a single global process manager that owns every wait4, dispatches exit statuses by PID, preserves streaming I/O and exit errors, and handles timeout/process-group cancellation. That would be a much larger and riskier change than using a dedicated init process.

tini gives us a clear ownership boundary: tini is PID 1 and handles orphan reaping, signal forwarding, and exit-status propagation, while kbagent keeps its current exec.Cmd.Wait() and process-group timeout behavior.

[cjc7373]

Another question. With tini as pid 1, when a kbagent's child process exits and a grandchild process still exists, the grandchild becomes kbagent's direct child (which is tini's grandchild). Now it is not tini's direct child, can tini reap it?

[leon-ape]

@cjc7373 Yes, tini can reap it. A normal grandparent does not automatically adopt an orphaned grandchild. On Linux, the orphan is reparented to the nearest living child subreaper, or to the PID namespace init process when no such subreaper exists.

kbagent is not registered as a child subreaper, while tini is PID 1, so in this case the orphaned grandchild becomes a direct child of tini, not kbagent. When it exits, tini receives SIGCHLD and reaps it with waitpid(-1, ..., WNOHANG).

kbagent still owns and waits for its direct action child via cmd.Wait(). tini cannot steal that wait because a process can only wait for its own children; it handles the main kbagent process plus orphaned descendants that have been reparented to it.

This is also the behavior described by the Linux child-subreaper semantics and implemented by tini: https://man7.org/linux/man-pages/man2/PR_SET_CHILD_SUBREAPER.2const.html and https://github.com/krallin/tini/blob/master/src/tini.c#L548-L608

[leon-ape]

/approve