In-cluster HA: host fencing + automatic VM restart - #137
Open
epheo wants to merge 5 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vSphere HA analog, GitOps-shaped. Enabling stages the Medik8s Node Health
Check operator install + a NodeHealthCheck CR into the platform repo; on merge,
unresponsive workers are fenced (Self Node Remediation, resource-deletion
strategy) and KubeVirt restarts their VMs on surviving hosts.
Mirrors the DRS tier exactly, one file set per layer:
hagen: renders the NHC file set (namespace, all-namespaces OperatorGroup,NHC + SNR Subscriptions, CR). SNR is subscribed explicitly: its OLM
dependency from NHC does not resolve on every catalog (observed live), and
the NodeHealthCheck is dead weight without the SelfNodeRemediationTemplate
CRD. Control-plane nodes are excluded from fencing (etcd quorum). The SNR
resource-deletion template also deletes VolumeAttachments, which is what
frees RWO volumes for restart.
nodehealth: discovery-gated SA reflector over NodeHealthCheck; absencereads as "not installed", never an error loop.
changeset/draft: declarative HA staging (wholesale replace, base-equalfiles skipped, re-submit-committed = cancel), atomic unstage.
/api/ha: GET view across git/draft/live planes, POST/DELETE gated on thecaller's nodehealthchecks-create SSAR;
nodehealthchecksjoins theplatform-author family.
fencing in progress, operator self-disable reason (e.g. a conflicting
MachineHealthCheck), and the reboot consequence before staging.
Spec knobs kept minimal: detection patience (60-3600s, default 300) and
minimum-healthy brake (default 51%). SelfNodeRemediation needs no BMC
hardware; Fence Agents Remediation stays a follow-up (its credentials must
not live in git; NHC would then switch to escalatingRemediations - FAR
first, SNR fallback).
Deliberate skip: no issues-plane rule. Issues derive from the streamed
Inventory only; HA degradation surfaces in the card, same as DRS.
Tests: hagen render/validate/round-trip incl. the explicit SNR Subscription,
nodehealth live-state (remediating sort, Disabled reason), changeset staging
flows incl. atomic unstage.