[huskcass]

Deleting an LLMService used to uninstall the release and drop the finalizer immediately, so the CR vanished while pods were still terminating and the ModelRoute was removed at the same moment as the Services it routes to. This stages the teardown so new traffic stops before the stack goes, and the CR stays until the release's pods are gone.

Stages

flowchart TD
  D[deletionTimestamp set] --> S{spec.suspend?}
  S -- yes --> X[no-op, keep finalizer]
  S -- no --> O{Orphan or foreign release?}
  O -- yes --> F[drop finalizer]
  O -- no --> M{ModelRoute in release manifest?}
  M -- no, outcome NoRoute --> U
  M -- yes --> R[RouteRemoving: delete ModelRoute]
  R --> RG{NotFound?}
  RG -- no, before deadline --> R
  RG -- no, past deadline: RouteStuck, its finalizer kept --> U
  RG -- yes --> ST[RouteSettling: wait routeSettle]
  ST --> U[Uninstalling: helm uninstall]
  U --> P[PodsTerminating: list release pods]
  P -- none left --> REL[Released] --> F
  P -- past forceDeleteAfter --> FD[ForceDeleting: grace 0, UID precondition] --> P
  P -- past maxWait: TimedOut --> REL

Loading

• Every wait is a 5s requeue (settle requeues for the exact remainder); nothing blocks.
• Status is written before the action it records, stages only move forward, and only stage and timestamps persist, so a restart resumes on the same clock.
• The ModelRoute is found by parsing kind: ModelRoute (routing.modelsphere.dev/*, routing.gpucluster.io/*) from the new helm.Release.Manifest. The operator never strips the route's finalizer.
• Release pods: app.kubernetes.io/instance=<release> ∪ app=<release>-<chart>, ignoring Succeeded/Failed. A pod within its grace always counts; past grace it is gone if its node is NotReady or missing, otherwise after grace + 5m.
• No in-flight guard: the charts' preStop drains already handle that inside the grace period.
• RBAC: adds delete on pods (only used for the opt-in force delete).

Knobs

Precedence: annotation > spec.deletion > default. Annotations are re-read on every pass, including ones added after the delete. An invalid annotation emits Warning InvalidAnnotation and falls back. 0s is a real zero in both layers.

spec	annotation	default	meaning
spec.deletion.maxWait	serving.modelsphere.dev/deletion-max-wait	1h	overall budget from startedAt; past it the finalizer is dropped
spec.deletion.routeSettle	serving.modelsphere.dev/route-settle	60s	wait after the ModelRoute is NotFound
spec.deletion.forceDeleteAfter	serving.modelsphere.dev/force-delete-after	unset (off)	grace-0 delete of pods terminating longer than this
—	serving.modelsphere.dev/deletion-policy: Orphan	—	unchanged: skips every stage

No other spec field changes.

Status

status.phase: Deleting, a Deleting condition (reasons RouteRemoving, RouteSettling, RouteStuck, NoRoute, Uninstalled, WaitingForPods, PodsForceDeleted, TimedOut, Released), one event per transition, plus Warning PodForceDeleted per pod.

deletion:
  stage: RouteRemoving|RouteSettling|Uninstalling|PodsTerminating|ForceDeleting|Released
  outcome: Clean|NoRoute|RouteStuck|PodsForceDeleted|TimedOut
  startedAt, deadline, routeRemovedAt, uninstalledAt, podsGoneAt: <time>
  podsRemaining: <int>
  pods: [{name, phase, node, deletionTimestamp, grace, stuck}]   # max 20
  forceDeleted: [<pod name>]                                      # max 20
  message: "waiting for 3 pods (2 stuck on NotReady node gpu-7)"

Tests

• 18 envtest controller cases, one YAML file each under internal/controller/serving/testdata/controller/: route deleted before uninstall, NotFound then settle, NoRoute, RouteStuck keeps the route finalizer, pod wait, Succeeded/Failed ignored, NotReady node, force delete, mid-deletion annotation, TimedOut, restart resume, Orphan, foreign release, suspend, annotation-over-spec, invalid annotation, 60s default, 0s settle.
• Helper tables under testdata/helpers/{deletionKnobs,modelRoutesFromManifest,nextStage,podCounts}/.
• kind e2e (test/e2e/deletion_test.go, e2e build tag): ModelRoute removed before the Deployment, a 20s preStop holds PodsTerminating, force-delete-after: 5s on a pod that ignores SIGTERM (grace 600), and RouteStuck with the fake autoconfig stopped. It runs in CI (make test-e2e); it was not run locally.
• Locally: gofmt, go vet (with and without -tags e2e), make manifests generate clean, go test -race on all non-e2e packages.

[huskcass]

No blockers.

Verdict: good to merge. The staged finalizer only moves forward, picks up at the recorded stage after a restart, keeps the finalizer while live pods remain unless maxWait or force-delete applies, and the generated CRD and RBAC files are in sync.

— Review Golden

[huskcass]

No blockers in this change.

Verdict: the route-settle fix is correct. The settle wait now ends at min(routeRemovedAt + routeSettle, deadline), the stage moves on at the deadline, and the new case fails without the fix.

— Review Golden