serviceoffer-controller silently reverts per-agent Hermes config (provider, MCP servers, turn cap) on every reconcile/upgrade

[bussyjd]

Summary

The serviceoffer-controller silently reverts each Agent's hermes-config (its LLM inference provider, MCP-server wiring, and turn cap) on every reconcile and every controller restart — including every obol stack up that bumps the controller image. Operator configuration that isn't expressible in the Agent CRD is destroyed with no warning.

This is not theoretical: after a routine silvernuc3 upgrade it took a paying agent endpoint down with a 502 (agent reverted from its configured provider to a backend that had no healthy deployment; the stale API server then returned EOF → the x402 verifier surfaced it as 502).

Root cause

Three compounding facts (all at internal/serviceoffercontroller/):

1. renderHermesConfig(model, litellmKey) (agent_render.go:120) emits a fixed template. Only the model name is variable — provider: custom→litellm, max_turns: 30, disabled_toolsets: [memory, web] are hardcoded, and no mcp_servers block is ever emitted. So an operator's grok/xai-oauth provider, raised turn cap, or paid MCP tools exist only until the next render overwrites them.

2. The hermes-config ConfigMap is full-replaced on every reconcile (applyAgentObject, agent.go:449 — get-then-full-Update, despite a stale "server-side-applied" comment). There is no content-hash skip on this path (one exists for the skill catalog, catalog.go, but was never applied here).

3. The informer's initial List on every controller-pod start re-emits Add for every Agent → a full re-provision of every agent. Since the controller image is digest-pinned and bumped per release, every upgrade restarts the controller → re-provisions every agent → reverts operator config.

The AgentSpec also has no way to express provider / MCP servers / turn cap, so there is no supported path for an operator to make this config survive.

Reproduce

1. Configure an agent with a non-custom provider (e.g. xai-oauth) and/or an mcp_servers block by editing the hermes-config ConfigMap directly.
2. Restart serviceoffer-controller (or obol stack up).
3. Observe the ConfigMap reverts to the template (gemma/litellm, no MCP), and the agent breaks.

Proposed fix

Make the Agent CR the source of truth, and stop rewriting unchanged config:

• A — Additive CRD fields + render passthrough. Add optional modelProvider, mcpServers, maxTurns, disabledToolsets to AgentSpec (keep model string; no version bump/conversion) and render them. Unset → byte-identical to today.
• B — Hash-skip. Stamp obol.org/hermes-config-hash on the live ConfigMap and skip the apply when the freshly-rendered desired hash matches — survives controller restarts, so unchanged config is never rewritten. Mirrors the existing catalog.go pattern.
• C — ConfigDrift status condition. Surface out-of-band ConfigMap edits instead of silently clobbering them (the safety net for B, which intentionally stops self-healing drift).

PR implementing A→B→C follows and has been validated live on silvernuc3 (controller restart no longer changes the ConfigMap resourceVersion; grok+MCP config preserved; agent healthy).