[huskcass]

Summary

A helm-backed release (plan ConfigMap) used to vanish from GET /api/deployments as soon as helm uninstall returned, while its pods were still terminating. It now stays listed with phase: uninstalling until the helm release is gone and no pod matches the release, then drops out.

Rebased on master after #30 (PD autoscaling). The uninstalling state now uses #30's per-release apply lock and covers the autoscaler.

flowchart LR
  A[installed<br/>phase applied] -->|DELETE /api/releases/ns/name| B[uninstalling<br/>plan marked, helm uninstall]
  B -->|helm fails| F[failed<br/>plan kept]
  B -->|helm release gone<br/>and no pods left| C[gone<br/>plan and autoscale ConfigMaps deleted]
  B -->|swissd restart| R[rebuilt from<br/>phase label] --> B

Loading

• Before helm runs, the plan ConfigMap gets status.yaml phase: uninstalling and the label swiss.modelsphere.dev/phase=uninstalling. It is kept after helm returns. Archive Secrets are deleted after helm uninstall, and again on a repeat DELETE and from the check loop once helm has dropped the release, so a restart does not leave them behind. They are kept when the uninstall fails and while helm still holds the release.
• Race fix under the apply lock: the uninstall marker is written while holding the release's apply lock, and the uninstalling check runs inside applyLocked, under the same lock, for install, upgrade, rollback and autoscale applies. An apply cannot overwrite the marker, and an uninstall cannot start mid-apply (409 apply in progress). A repeat DELETE on a release that is already uninstalling answers 200, even while the first uninstall still holds the lock.
• swissd keeps the uninstalling set in memory. A loop started from Server.Run (next to the autoscale loop) checks every 5s, and so do list and status reads. After a restart the set is rebuilt from the label.
• The plan is dropped only after exec.Lookup reports the helm release gone and nothing matches app.kubernetes.io/instance=<release> or app=<release>-<chart>. The swiss-autoscale-<release> ConfigMap is removed in the same step. Finished pods (Succeeded/Failed) don't count. A Terminating pod (deletionTimestamp set) still counts during its grace period (deletionGracePeriodSeconds, else terminationGracePeriodSeconds, else 30s). Past that it is ignored if its node is NotReady or missing, and in any case once deletionTimestamp + grace + 5 minutes has passed. If the lookup or pod list fails, the release stays listed. If a ConfigMap delete fails, the next check retries it.
• An LLMService with metadata.deletionTimestamp set (or status.phase: Deleting) is reported as uninstalling too; see the llmsvc section below.
• The autoscaler holds (holdReason: uninstalling, no apply) for a release whose plan is uninstalling or whose LLMService has a deletionTimestamp.

LLMService (llmsvc) backend

Depends on modelsphere/llm-operator#21 (staged LLMService deletion). With that change the operator runs the teardown behind its finalizer: delete the ModelRoute, wait for route settle, helm uninstall, wait for the release pods to terminate (bounded by maxWait, optional force delete), then drop the finalizer. Swiss no longer waits for any of it in the request, and does not scan pods for llmsvc releases.

• DELETE marks the plan ConfigMap uninstalling when there is one, deletes the LLMService, records the uninstall as started, and returns 200 right away with the in-progress body (output: uninstall already in progress, same fields as before). The 15-minute WaitGone in the request path is gone, so a repeat DELETE during teardown gets 200 in progress instead of 409. NotFound on delete is still 404, and the plan is put back.
• The reap loop treats an llmsvc release as done when the LLMService is NotFound. Then it removes archive Secrets, the swiss-autoscale-<release> ConfigMap and the plan ConfigMap, and records the final audit row once. A restart rebuilds the set from LLMServices that have a deletionTimestamp. If the object is recreated, tracking stops and nothing is deleted.
• List and status: an LLMService with a deletionTimestamp or status.phase: Deleting is uninstalling. status.deletion is passed through as deletion (stage, outcome, podsRemaining, deadline, startedAt, message, pods, forceDeleted) on the deployments row and in llmservice.deletion on status. llmservice.operatorPhase is the raw status.phase. Drift names the stage, e.g. uninstall in progress: PodsTerminating. All of these are optional, and helm rows are unchanged.
• The autoscaler still holds on a deletionTimestamp, and now also on status.phase: Deleting.
• Swiss has no force or skip action. To force or shorten a teardown, annotate the LLMService: serving.modelsphere.dev/deletion-max-wait, serving.modelsphere.dev/route-settle, serving.modelsphere.dev/force-delete-after, or serving.modelsphere.dev/deletion-policy: Orphan.
• With an operator that does not stage deletion yet, the object goes almost at once and the same reap path cleans up.

Status contract (additive only): the deployments row phase, helm planStatus.phase and llmsvc llmservice.phase can now also be uninstalling. DELETE /api/releases/{ns}/{release} keeps its route and body. A repeat DELETE on an uninstalling release answers 200 with output: uninstall already in progress. openapi.yaml now lists the 409 a DELETE gets while an apply holds the lock. The deployments row omits revision when there is none (it is no longer required in openapi.yaml), so an uninstalling row does not report 0. The chart is unchanged because it already grants pod list and ConfigMap delete.

UI: modelsphere/console#96 · Refs modelsphere/console#94

Testing

• internal/server/testdata/uninstalling/*.yaml has one case per file, run by uninstalling_cases_test.go. Cases: pods remain, pods gone, engine-only and instance-only pods, restart (rebuild), restart while the helm release remains, destroy fails, second uninstall, retry while helm remains, install/upgrade/rollback 409, pod list error, ConfigMap delete retry, LLMService with deletionTimestamp, finished pods ignored, terminating within grace (kept), terminating past grace / on a NotReady node / on a missing node (ignored), revision omitted, and archive cleanup on repeat DELETE and after restart.
• New cases for the lock and autoscale changes:
  • uninstall while an apply holds the lock (409, phase unchanged), and install/upgrade/rollback refused while helm runs;
  • a repeat DELETE while helm runs (200);
  • the autoscale ConfigMap is removed on reap;
  • a NotReady or missing node within grace keeps the release uninstalling;
  • archives are kept when the uninstall fails and while helm still holds the release.
• Autoscale cases: internal/autoscale/testdata/decide/guard-uninstalling.yaml, guard-llm-deleting.yaml, and internal/server/testdata/autoscale-loop/helm-uninstalling.yaml, llm-deleting.yaml (hold, no apply).
• llmsvc cases (testdata/uninstalling/llmsvc-*.yaml): DELETE returns at once in progress, repeat DELETE 200, reap waits while the LLMService exists with status.deletion and passes it through to list and status, reap cleans up on NotFound with one final audit, delete NotFound is 404 and restores the plan, restart rebuilds from a deleting LLMService, a recreated object is not cleaned up, and status.phase: Deleting passthrough. The old synchronous TestLLMUninstall and llmsvc TestWaitGone are removed. Autoscale: guard-llm-phase-deleting.yaml, autoscale-loop/llm-phase-deleting.yaml.
• Ran locally, all passing: gofmt (clean), go build ./..., go vet ./..., go vet -tags=e2e ./... and go test -race -count=1 ./....
• test/e2e: the helm deploy-and-remove spec checks that the row is listed as uninstalling while pods exist, then gone. The llmsvc spec passes whether the operator drops the LLMService at once or stages it: while the object exists, the row must be uninstalling and a repeat DELETE returns 200. The gone timeout for llmsvc is 6 minutes.

[huskcass]

Blocker

• internal/server/uninstalling.go:330-345 (podsRemain): every pod that matches either selector is counted, whatever its phase, and this check is the only thing that ever ends uninstalling. Helm does not delete some pods: a Succeeded or Failed helm test pod, a hook Job pod labelled app.kubernetes.io/instance=<release>, or a pod stuck Terminating on a lost node. Any one of these keeps the release uninstalling for good. After that, install, upgrade and rollback return 409 (refuseUninstalling, uninstalling.go:193), and a repeated DELETE only answers uninstall already in progress (backend_helm.go:181). The API gives no way out, so someone has to delete the pod or the plan ConfigMap by hand. Reproduced with a copy of testdata/uninstalling/pods-gone.yaml that has a single phase: Succeeded pod: after dropHelm the row is still listed as uninstalling, and a second uninstall does not change that. Suggested fix: ignore pods in a terminal phase, and give the state an exit. That could be a deadline that moves the plan to failed with a message, or a repeated DELETE that drops the plan once helm reports the release gone.

Verdict: the design and test coverage are good, and the restart, destroy-fails and retry paths hold up. It can merge once the stuck-uninstalling path has an exit.

— Review Golden

[huskcass]

Thanks. b0f53f8: podsRemain now ignores pods in Succeeded or Failed phase, so leftover helm test and hook pods no longer hold a release in uninstalling; new cases finished-pods-ignored (only Succeeded/Failed pods left, plan dropped) and live-pods-keep-uninstalling (a Pending pod next to a Succeeded one keeps it uninstalling). The exit for a release that stays stuck (deadline vs. repeated DELETE) is still open and will follow in a separate commit.

[huskcass]

Pushed 09b0ef5 with the agreed review fixes:

• podsRemain: a Terminating pod still counts during its grace period. It is ignored only when stuck: past deletionTimestamp + grace (deletionGracePeriodSeconds, else terminationGracePeriodSeconds, else 30s) + 5m, or its node is NotReady or missing. The decision lives in the pure podStuckTerminating, which takes the clock and node readiness as inputs.
• Archive Secrets are now also deleted on a repeat DELETE and from the check loop once helm has dropped the release, so the path after a restart cleans them up too.
• revision in the deployments row is omitempty, and openapi.yaml no longer lists it as required.
• New YAML cases: terminating within grace, past grace, on a NotReady node, on a missing node, archive cleanup on repeat DELETE and after restart, and revision omitted.

The check-then-act 409 race and the writes during GET /api/deployments are left for the rebase onto #30's per-release lock.

[huskcass]

No blockers. The stuck-uninstalling blocker from the previous review is resolved.

Verdict: good to merge after the #30 rebase. Finished pods are ignored, stuck Terminating pods are released by the grace + 5m rule or a NotReady/missing node, archive cleanup only touches this release's owner=swiss Secrets, and every fix has a case that fails without it.

— Review Golden

[huskcass]

Rebased on master after #30 (new head 9cc0284). Changes:

• The uninstalling check now runs inside applyLocked, under the same per-release lock as the uninstall marker. An apply can't overwrite the marker, and an uninstall can't start mid-apply (409). A repeat DELETE on an uninstalling release still returns 200.
• swiss-autoscale-<release> is removed in reapRelease.
• The autoscaler holds (uninstalling) for an uninstalling release or an LLMService being deleted.
• A Terminating pod on a NotReady or missing node is ignored only after deletionTimestamp + grace.
• New YAML cases cover these, plus archives being kept on failure and while helm holds the release. openapi.yaml documents the DELETE 409.

[huskcass]

No blockers. The earlier stuck-uninstalling blocker stays fixed after the rebase.

Verdict: good to merge. The uninstall marker and the refusal for apply, rollback and autoscale now happen under one per-release TryLock, so neither path can block the other. The autoscaler holds on uninstalling releases and on LLMService deletions, and reap removes swiss-autoscale-<release>.

— Review Golden

[huskcass]

Pushed 22c0740 and f2dccc2. These change only the llmsvc backend and depend on modelsphere/llm-operator#21 (staged LLMService deletion). Nothing changes for helm.

22c0740: feat(server): return from an LLMService delete at once and reap on NotFound

• DELETE marks the plan uninstalling, deletes the LLMService, records the uninstall as started and returns 200 in progress right away. The body has the same fields as before. A repeat DELETE during teardown is now 200 in progress, where it used to be 409 behind the 15-minute WaitGone. NotFound is still 404, and the plan is restored.
• Reap: an llmsvc release is done when the LLMService is NotFound. There is no pod scan and no helm lookup. Then it removes archives, swiss-autoscale-<release> and the plan, and writes one final audit row. A restart rebuilds the set from deleting LLMServices. A recreated object is left alone.
• List and status pass status.deletion through as deletion (stage, outcome, podsRemaining, deadline, startedAt, message, pods, forceDeleted). Status also adds llmservice.operatorPhase. Deleting counts as uninstalling, and the autoscaler holds on it.
• Removed: WaitGone and its test, and the synchronous TestLLMUninstall. New YAML cases are under testdata/uninstalling/llmsvc-*.yaml.
• Force or skip is done by annotating the LLMService (deletion-max-wait, route-settle, force-delete-after, deletion-policy: Orphan). Swiss has no API for it.

f2dccc2: test(e2e): accept an instant or staged LLMService delete

• While the LLMService exists, the row must be uninstalling and a repeat DELETE returns 200. An object that is gone at once passes too. The gone timeout for llmsvc is 6 minutes.

Checks: gofmt, go build, go vet (also -tags=e2e) and go test -race -count=1 ./... pass locally. CI go, e2e and publish pass.

[huskcass]

No blockers.

Verdict: good to merge. An LLMService DELETE now returns 200 once the delete is accepted, a repeat DELETE also gets 200, the reap runs once the object is NotFound, and it still works against an operator without staged deletion.

— Review Golden