fix(tpd): land a busy visor's full transport list via a targeted, bounded discovery-leaf fetch - #4046
Merged
Conversation
…nded discovery-leaf fetch
A visor publishes its entire transport set as one inlined `tp-list` leaf on
its CXO feed, alongside a bulky per-transport telemetry subtree. TPD's
aggregator only reconciled discovery once CXO finished filling the WHOLE
Root (OnRootFilled) or, best-effort, when a fill broke (OnFillingBreaks).
For a busy visor the Root carries hundreds-to-thousands of telemetry
objects, and the whole-Root fill can't complete within MaxFillingTime over
the short-lived delivering conn — so OnRootFilled rarely fires and TPD
reflected only ~10% of the visor's transports, starving the route-finder.
Decouple discovery from the telemetry fill. On OnRootReceived, do a
bounded, targeted fetch of just the tp-list leaf path (registry + root
TreeNode + the small top-level Children index + the tp-list TreeEntry) over
the delivering conn and reconcile the reporter's full transport set
immediately — independent of the 90s whole-Root fill and OnFillingBreaks.
Because the snapshot is a single inlined TreeEntry, landing that handful of
path objects lands the ENTIRE list, and it survives the same short conn a
whole-tree fill can't.
The fetch reuses the existing (*Container).Preview + a new public
(*Conn).Getter() — objects are read from the local store if the concurrent
fill already has them, else pulled from the peer on demand and held in
memory only (never written to CXDS), so it neither competes with nor
disturbs the whole-Root fill or the cleanup sweep. It is:
- Bounded: a boundedGetter caps objects (64) and wall-clock (30s) per
Root so a hostile/oversized Root can't drive unbounded requests or hang.
- Coalesced per feed so a rapidly republishing visor can't pile up fetches.
- Idempotent vs the OnRootFilled path: both call the declarative
ReconcileTransportsFromCXO (register-all + deregister-absent).
- Backed by a last-good per-reporter cache re-applied when a fetch fails,
holding transports against the redis TTL until a fresh Root lands;
dropped when the feed is reclaimed.
Tests: a Root with a large telemetry subtree whose objects are withheld
still reconciles ALL of its transports via the targeted path; the fetch is
proven bounded to the tp-list path (telemetry never requested); missing
leaf is a clean miss; boundedGetter honors both caps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A visor publishes its entire transport set as one inlined
tp-listleaf on its CXO feed, alongside a bulky per-transport telemetry subtree (transports/<uuid>/current, timelines, bitmaps). TPD's aggregator only reconciled discovery once CXO finished filling the whole Root (OnRootFilled) or, best-effort, when a fill broke (OnFillingBreaks).For a busy visor the Root carries hundreds-to-thousands of telemetry objects, and the whole-Root fill can't complete within
MaxFillingTime(90s) over the short-lived delivering conn before it churns. SoOnRootFilledrarely fires and TPD reflected only ~10% of the visor's transports (measured live: 850 local transports → ~51-121 in TPD, decaying), starving the route-finder.Fix
Decouple discovery from the whole-Root telemetry fill. On
OnRootReceived, do a bounded, targeted fetch of just the tp-list leaf path — registry + rootTreeNode+ the small top-levelChildrenindex page(s) + thetp-listTreeEntry— over the delivering conn, and reconcile the reporter's full transport set immediately. This is independent of the 90s whole-Root fill and ofOnFillingBreaks; the telemetry subtree fill can still run and fail as before.Because the tp-list snapshot is a single inlined
TreeEntry, landing that handful of path objects lands the entire list — and it survives the same short conn a whole-tree fill can't.How
(*node.Conn).Getter()exposes the connection's existing on-demand single-object getter (oneRqObjectperGet, hash-verified, bounded byResponseTimeout). No change to the fill path.(*skyobject.Container).Previewpack: objects are found in the local store if the concurrent fill already has them, else pulled from the peer on demand and held in memory only (never written to CXDS) — so this neither competes with nor disturbs the whole-Root fill or the cleanup sweep.Safety / bounds
boundedGettercaps objects (64) and wall-clock (30s) per Root, so a hostile/oversized Root can't drive unbounded per-object requests or hang the walk (budget-exceeded unwinds cleanly —childByNametreats an unreadable child as absent).OnRootFilledpath: both call the declarativeReconcileTransportsFromCXO(register-all + deregister-absent).Tests
TestTargetedFetchLandsFullListDespiteTelemetry: a Root with a large telemetry subtree whose objects are withheld still reconciles all 200 of its transports via the targeted path, and the fetch is proven bounded to the tp-list path (telemetry objects never requested).TestTargetedFetchMissingLeafCleanMiss: a Root with no tp-list leaf is a clean miss (falls back to cache, no panic/partial).TestBoundedGetterBudget: object-count and deadline caps both enforced.go build ./...,go vet,go test ./pkg/cxo/... ./pkg/transport-discovery/...(incl.-raceon the aggregator) all pass; existing aggregator/fill tests unchanged.