feat(solana-indexer): PR 4 — Component structs (skeleton declarations) by tilacog · Pull Request #4514 · cowprotocol/services

tilacog · 2026-06-10T17:33:46Z

Description

Adds the indexer/ module with skeleton declarations for the four components that will do the actual work of the indexer. Each struct declares its fields, a constructor, and doc comments describing what it will do; the run bodies are unimplemented! and the behavior lands in later PRs.

The four components and their roles:

Ingester: subscribes to the Yellowstone gRPC stream and drains it as fast as updates arrive, forwarding them to the decoder. It does no decoding itself, so the socket never backs up behind slow processing. It is also the single writer of the "latest chain slot" counter that the other components use to know how far the chain has advanced.
Decoder: receives the raw stream updates, picks out transactions belonging to the settlement and SolFlow programs, matches each transaction with its corresponding account-update snapshot, and persists the resulting typed events to the store.
Partial-event watchdog: some events arrive in two halves (a transaction update and an account update) that don't always land together. The decoder parks the half it has in a map shared with the watchdog; the watchdog periodically scans that map and dead-letters any entry whose other half never showed up within the slot window, recording which half went missing.
Finalization worker: rows are first written at the confirmed commitment level. This worker re-checks them against the chain and promotes them to finalized, or marks them rolled back if the transaction disappeared. It uses a cheap batched RPC call for recent rows and falls back to one-call-per-row lookups for rows old enough that the batched method no longer reports them.

The field declarations also pin down how the components talk to each other: the ingester feeds the decoder over a bounded channel, while the decoder and the watchdog share a concurrent map with no message passing between them. That second decision reshaped the partial-event types, so types/channel.rs became types/shared.rs.

Changes

Added indexer/ingester.rs: the Ingester struct. Generic over the connection so unit tests can drive it with a mock (the trait-bound approach chosen in PR 3 instead of a third in-crate trait). Also declares the shared latest-chain-slot counter and the constants for reconnect backoff and channel capacity.
Added indexer/decoder.rs: the Decoder struct, holding the receiving end of the channel, the shared partial-event map, and the two program ids it filters for.
Added indexer/watchdog.rs: the PartialEventWatchdog struct, holding the store and its view of the shared partial-event map.
Added indexer/finalization.rs: the FinalizationWorker struct, with module docs explaining the two promotion flows and the constants that bound them (finalization window, batch size, retention horizon).
Renamed types/channel.rs to types/shared.rs and reworked the partial-event types: instead of payloads sent over a channel, a partial event is now a key (slot + signature) and a value (whichever half arrived first) in the map shared by the decoder and the watchdog.
Dependencies: added yellowstone-grpc-client to the workspace (source of the connection trait the ingester is generic over), plus dashmap and a few tokio features to the crate.

How to test

cargo check -p solana-indexer
cargo clippy --locked -p solana-indexer --all-features --all-targets -- -D warnings

No unit tests are included: the structs declare shape only, and every run body is unimplemented! until the behavior PRs.

This is a follow-up PR to #4508

gemini-code-assist

Code Review

This pull request introduces the skeleton structure for the Solana settlement indexer components, including the Ingester, Decoder, PartialEventWatchdog, and FinalizationWorker, along with necessary dependency updates and shared type definitions. Feedback on the changes suggests replacing the global static LATEST_CHAIN_SLOT with a shared Arc passed via constructors to avoid process-wide shared mutable state and potential race conditions during parallel testing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-10T17:35:21Z

+/// The sole writer is the ingester, on every slot-filter message. Anchors the
+/// partial-event watchdog and the finalization worker. Cold start is zero; the
+/// watchdog skips its comparison on the first tick.
+pub static LATEST_CHAIN_SLOT: AtomicU64 = AtomicU64::new(0);


Using a global static AtomicU64 for LATEST_CHAIN_SLOT introduces shared mutable state across the entire process. This causes race conditions and flakiness when running unit/integration tests in parallel (Cargo's default behavior). It also prevents running multiple indexer instances in the same process.

Actionable Suggestion:
Remove the global static and instead pass an Arc<AtomicU64> (or a shared state struct) to the constructors of Ingester, Decoder, PartialEventWatchdog, and FinalizationWorker.

References

Focus exclusively on identifying missing edge cases, potential race conditions, or logic that deviates from the PR's stated goals. ^(link)

github-actions · 2026-06-18T00:39:55Z

This pull request has been marked as stale because it has been inactive a while. Please update this pull request or it will be automatically closed.

MartinquaXD · 2026-06-23T11:02:50Z

+/// Typical number of slots for a transaction to finalize (~12.8 s). The
+/// promotion pass skips rows fresher than this.


What does typical mean here? In which circumstances are 32 slots NOT correct?

MartinquaXD · 2026-06-23T11:04:52Z

+
+/// Typical number of slots for a transaction to finalize (~12.8 s). The
+/// promotion pass skips rows fresher than this.
+#[allow(dead_code)]


Instead of using #[allow(lint)] please use #[expect(lint)]. The reason is that clippy will automatically generate a warning when the lint is no longer violated which forces you to clean up stale lint exceptions.

MartinquaXD · 2026-06-23T11:07:01Z

+
+/// Transaction finalization worker. See the module docs for the two flows it
+/// runs.
+pub struct FinalizationWorker<St: Store, R: SolanaClient> {


IMO going down the route of generics instead of Arc<dyn Trait> is a slippery slope. Type parameters need to be propagated throughout the code base and the tiny overhead of chasing a pointer when using dyn trait hardly matters if the component issues network requests.

MartinquaXD · 2026-06-23T11:09:26Z

+/// The sole writer is the ingester, on every slot-filter message. Anchors the
+/// partial-event watchdog and the finalization worker. Cold start is zero; the
+/// watchdog skips its comparison on the first tick.
+pub static LATEST_CHAIN_SLOT: AtomicU64 = AtomicU64::new(0);


What's the reason that this value does not live inside a specific component that gets accessed via a getter?
Also the API is currently fragile. The type is pub and allows anyone to access it. A safer API would be a new type wrapping the counter with an update function that is only accessible in this module for example.

MartinquaXD · 2026-06-23T11:10:36Z

+/// Capacity of the channel from the ingester to the decoder.
+pub const INGEST_TO_DECODER_CAPACITY: usize = 1024;


This and a few other consts seems like it should be configurable by a config file.

MartinquaXD · 2026-06-23T11:12:21Z

+/// Capacity of the channel from the ingester to the decoder.
+pub const INGEST_TO_DECODER_CAPACITY: usize = 1024;


Can you explain why the ingester is necessary? Why is it better to have it forward the events to a different channel? If the other channel it pushed into does not get cleared fast enough the backpressure will end up here anyway.

MartinquaXD · 2026-06-23T11:13:11Z

+    /// Sends `StreamUpdate` to the decoder. Should be bounded to
+    /// `RECONNECT_BACKOFF_CAP` entries.
+    pub tx: Sender<StreamUpdate>,


Comment says the size of the channel should be bounded to a unit of time which does not seem to make sense.

MartinquaXD · 2026-06-23T11:15:06Z

+    /// Shared in-memory map of partial events keyed by `PartialEventKey`.
+    ///
+    /// The decoder holds a clone of this `Arc` and both inserts and removes
+    /// halves as it processes them.
+    pub partials: Arc<DashMap<PartialEventKey, PartialEvent>>,


What was the reasoning for using channels for some of the communication but not for this partials map?

MartinquaXD · 2026-06-23T11:21:21Z

+/// The sole writer is the ingester, on every slot-filter message. Anchors the
+/// partial-event watchdog and the finalization worker. Cold start is zero; the
+/// watchdog skips its comparison on the first tick.
+pub static LATEST_CHAIN_SLOT: AtomicU64 = AtomicU64::new(0);


It should be documented that this slot number is completely unrelated to the indexing progress of the system. Obviously depends on the rest of the indexer logic but it sounds like a counter that tracks fully processed slots might be more useful.

squadgazzz · 2026-06-24T13:13:43Z

+///
+/// The watchdog holds a clone of the same `partials` map, so the two operate on
+/// the same concurrent map without any message passing between them.
+pub struct Decoder<St: Store> {


St looks a bit odd to me.

Suggested change

pub struct Decoder<St: Store> {

pub struct Decoder<S: Store> {

squadgazzz · 2026-06-24T13:16:09Z

+/// Key for the shared decoder↔watchdog partials map: the `(slot, signature)`
+/// pair identifying which on-chain event a `PartialEvent` belongs to.
+#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, PartialOrd, Ord)]
+pub struct PartialEventKey(pub u64, pub Signature);


Already mentioned in previous PRs, but having a global Slot type should probably improve readability.

Suggested change

pub struct PartialEventKey(pub u64, pub Signature);

pub struct PartialEventKey(pub Slot, pub Signature);

squadgazzz · 2026-06-24T13:20:48Z

+/// Cap on the exponential backoff between reconnect attempts.
+#[allow(dead_code)]
+pub const RECONNECT_BACKOFF_CAP: std::time::Duration = std::time::Duration::from_secs(30);
+
+/// Capacity of the channel from the ingester to the decoder.
+pub const INGEST_TO_DECODER_CAPACITY: usize = 1024;


Those should probably be a part of some config with default config values, which aligns with the general codebase.

tilacog and others added 2 commits June 10, 2026 14:21

feat(solana-indexer): PR 4 — Component structs (skeleton declarations)

b29a0b5

docs(solana-indexer): describe component roles in indexer module docs

126a8cd

tilacog requested a review from a team as a code owner June 10, 2026 17:33

gemini-code-assist Bot reviewed Jun 10, 2026

View reviewed changes

github-actions Bot added the stale label Jun 18, 2026

MartinquaXD reviewed Jun 23, 2026

View reviewed changes

tilacog mentioned this pull request Jun 23, 2026

feat(solana-indexer) PR 5.1: ingester drain loop #4549

Open

squadgazzz reviewed Jun 24, 2026

View reviewed changes

		/// Typical number of slots for a transaction to finalize (~12.8 s). The
		/// promotion pass skips rows fresher than this.

		/// Capacity of the channel from the ingester to the decoder.
		pub const INGEST_TO_DECODER_CAPACITY: usize = 1024;

	pub struct Decoder<St: Store> {
	pub struct Decoder<S: Store> {

	pub struct PartialEventKey(pub u64, pub Signature);
	pub struct PartialEventKey(pub Slot, pub Signature);

Conversation

tilacog commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

How to test

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tilacog commented Jun 10, 2026 •

edited

Loading