POC: aggregate post-block sync acknowledgements

.env.example

@@ -22,6 +22,11 @@
 # Block production interval (seconds). Lower = faster blocks, higher CPU.
 CONSENSUS_TIME=10
 
+# Experimental POC: replace each block recipient's all-peer sync-status
+# rebroadcast with one bounded aggregate from the block sender. Keep disabled
+# outside an isolated devnet until convergence and adversarial tests pass.
+BLOCK_SYNC_AGGREGATION_ENABLED=false
+
 # Genesis-state network parameters. Governance proposals can override these
 # at runtime; these values are the chain's bootstrap defaults.
 # Total per-tx flat fee = RPC_FEE + NETWORK_FEE + BURN_FEE (default 1+1+1=3).

.gitignore

@@ -89,6 +89,7 @@ testing/devnet/identities/
 testing/devnet/.env
 testing/devnet/postgres-data/
 testing/runs/
+.poc-results/
 local_tests/
 
 # ---- Local devnet identities (private keys / mnemonics) ----

docs/poc/block-sync-aggregation.md

@@ -0,0 +1,165 @@
+# Block sync aggregation POC
+
+## Purpose
+
+This POC tests one narrow change to the existing PoRBFT network path. It does
+not change transaction validation, committee voting, block construction, or
+finality.
+
+The legacy path invokes block broadcast on every committee member, then makes
+each non-signing block recipient rebroadcast its sync status to every known
+peer. With `N` nodes and `S` block signers, its modeled post-block request
+burst is:
+
+```text
+S * (N - S) duplicate block deliveries
++ N * (N - S) recipient status calls
++ N * S sender status calls
+```
+
+The POC designates the existing committee secretary as the sole block
+publisher and replaces the recipient broadcasts with one acknowledgement
+aggregate:
+
+```text
+(N - S) block deliveries + (N - 1) aggregate deliveries
+```
+
+## Activation
+
+The experiment is disabled by default. Enable it only on an isolated devnet:
+
+```text
+BLOCK_SYNC_AGGREGATION_ENABLED=true
+```
+
+Every node in the experiment must use the same setting. This POC does not
+define a mixed-version activation protocol.
+
+The repository includes a loopback-only, resource-bounded overlay. Generate
+six identities and enable both POC profiles so a four-validator committee has
+two real non-signing block recipients:
+
+```text
+NODE_COUNT=6 testing/devnet/scripts/setup.sh
+docker compose --profile rehearsal --profile scale-poc -p demos-sync-poc \
+  -f testing/devnet/docker-compose.yml \
+  -f testing/devnet/docker-compose.fixture.yml \
+  -f testing/devnet/docker-compose.sync-aggregation-poc.yml up --build
+```
+
+Set `BLOCK_SYNC_AGGREGATION_ENABLED=false` on the compose command to run the
+same resource-bounded topology through the legacy path for comparison.
+
+Six full processes are the safe ceiling for the current 4-CPU production VPS.
+The 20/30/50-node validation must run across dedicated hosts; putting those
+processes on one busy machine would benchmark CPU and memory starvation rather
+than consensus networking.
+
+## Validation performed by recipients
+
+An aggregate is accepted only when:
+
+- it has the bounded version-1 shape;
+- its block number and hash match a locally stored block;
+- its RPC sender signed that block;
+- each claimed identity was committed in the block peerlist or signed the
+  block; and
+- the peer is already known locally.
+
+The aggregate only advances the peer's sync hint to an already verified local
+block. It never marks a peer online. Existing authenticated hello calls and
+peer gossip remain the anti-entropy path for missed aggregate deliveries.
+
+## POC limitation
+
+The aggregate authenticates the block-signing publisher, not each relayed
+peer acknowledgement. A production protocol should either carry a detached
+signature from every acknowledging peer or formally state that a quorum block
+signer is trusted to relay inclusion-only liveness observations. This POC must
+not be deployed until that trust decision and mixed-version activation are
+reviewed.
+
+## Modeled request counts
+
+For a four-validator committee:
+
+| Nodes |  Legacy | Aggregate | Reduction |
+| ----: | ------: | --------: | --------: |
+|     5 |      29 |         5 |     82.8% |
+|     6 |      44 |         7 |     84.1% |
+|    20 |     464 |        35 |     92.5% |
+|    30 |   1,004 |        55 |     94.5% |
+|    50 |   2,684 |        95 |     96.5% |
+|   500 | 251,984 |       995 |     99.6% |
+
+With 500 realistic 66-character public-key identities, the version-1 JSON
+aggregate is about 34.4 KB before its authenticated RPC envelope. A production
+version should encode acknowledgements as a peerlist-indexed bitmap (or use a
+bounded gossip tree) to reduce bytes as well as request count.
+
+The model excludes periodic anti-entropy because it is not triggered once per
+block. A real multi-host run is still required to measure bytes, latency,
+retries, convergence, and failure recovery.
+
+## Six-node VPS result
+
+On 2026-08-18 the legacy and aggregate paths were run on the same isolated
+six-node Docker topology (four-member shard, two non-signing recipients), with
+a 20-second consensus cadence. Each measurement excluded startup traffic and
+covered five complete blocks.
+
+| Mode | Block deliveries | Sender status | Receiver status | Aggregate | Total calls/block |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Legacy | 8 | 24 | 12 | 0 | 44 |
+| Aggregate | 2 | 0 | 0 | 5 | 7 |
+
+The measured reduction was 84.1%, exactly matching the model. All six nodes
+started and ended each sample at the same height (height spread zero).
+
+A missed-update recovery check then stopped node 6 while the chain advanced,
+restarted it, and observed node 6 automatically converge with node 1 at height
+10. This confirms the retained fast-sync/anti-entropy path repairs missed
+aggregate deliveries in this topology.
+
+The host had four CPUs and 15 GiB RAM. Six full nodes saturated the available
+CPU during rounds and used roughly 1 GiB resident memory each, so larger real
+tests must use dedicated multi-host infrastructure. Running 20–50 processes on
+that host would measure resource starvation rather than network scalability.
+
+## Lightweight transport-scale result
+
+`testing/devnet/scripts/run-sync-scale-emulator.ts` exercises the real
+aggregate builder, wire shape, and recipient admission code over loopback HTTP
+with hundreds of virtual identities. It injects 20–100 ms baseline jitter, 5%
+slow peers with another 220 ms delay, and 5% retryable first-attempt failures.
+The VPS wrapper pauses (but does not remove) the six full POC nodes, enforces a
+memory floor, verifies DACS health, and resumes all six nodes after the run.
+
+On 2026-08-18, five post-block bursts were measured at each size:
+
+| Peers | Legacy calls/block | Aggregate calls/block | Reduction | Mean burst | Aggregate | Total wire/block | Peak RSS | Event-loop p99 |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 100 | 10,384 | 195 | 98.1% | 504 ms | 6.8 KB | 0.72 MB | 70 MB | 3.9 ms |
+| 250 | 63,484 | 495 | 99.2% | 528 ms | 17.1 KB | 4.51 MB | 104 MB | 14.7 ms |
+| 500 | 251,984 | 995 | 99.6% | 810 ms | 34.4 KB | 18.10 MB | 149 MB | 55.2 ms |
+
+The observed logical call count matched the linear model at every size. All
+deliveries were admitted after bounded retries; valid aggregates passed, while
+non-signer and wrong-block aggregates failed closed. The six full nodes and
+all four live DACS services were healthy after automatic resume.
+
+Run the guarded VPS test with:
+
+```text
+NODE_COUNTS=100,250,500 ITERATIONS=5 \
+  testing/devnet/scripts/run-sync-scale-vps.sh
+```
+
+This is strong evidence for the transport path, not a substitute for a real
+multi-host validator soak. The emulator multiplexes virtual recipients through
+one Bun process and does not reproduce hundreds of databases, consensus loops,
+OmniProtocol connections, cryptographic acknowledgement signatures, or WAN
+kernel scheduling. The 18.1 MB sender burst at 500 peers also motivates the
+documented bitmap/tree follow-up even though the burst completed in under one
+second here.

src/config/defaults.ts

@@ -67,6 +67,7 @@ export const DEFAULT_CONFIG: AppConfig = {
         restore: false,
         blockWatchdogEnabled: false,
         blockWatchdogTimeoutSeconds: 600,
+        blockSyncAggregationEnabled: false,
     },
 
     tlsnotary: {

src/config/envKeys.ts

@@ -53,6 +53,7 @@ export const EnvKey = {
     RESTORE: "RESTORE",
     BLOCK_WATCHDOG_ENABLED: "BLOCK_WATCHDOG_ENABLED",
     BLOCK_WATCHDOG_TIMEOUT_SECONDS: "BLOCK_WATCHDOG_TIMEOUT_SECONDS",
+    BLOCK_SYNC_AGGREGATION_ENABLED: "BLOCK_SYNC_AGGREGATION_ENABLED",
 
     // --- TLSNotary ---
     TLSNOTARY_ENABLED: "TLSNOTARY_ENABLED",

src/config/loader.ts

@@ -159,6 +159,10 @@ export function loadConfig(): Readonly<AppConfig> {
                 EnvKey.BLOCK_WATCHDOG_TIMEOUT_SECONDS,
                 d.core.blockWatchdogTimeoutSeconds,
             ),
+            blockSyncAggregationEnabled: envBool(
+                EnvKey.BLOCK_SYNC_AGGREGATION_ENABLED,
+                d.core.blockSyncAggregationEnabled,
+            ),
         },
 
         tlsnotary: {

src/config/types.ts

@@ -66,6 +66,11 @@ export interface CoreConfig {
     restore: boolean
     blockWatchdogEnabled: boolean
     blockWatchdogTimeoutSeconds: number
+    /**
+     * POC: replace receiver-side all-peer sync rebroadcasts with one
+     * block-signer aggregate. Disabled by default until multi-node validation.
+     */
+    blockSyncAggregationEnabled: boolean
     rpcFee: number
     networkFee: number
     /** Per-tx burn — sat/lamport-style integer for now.

src/features/metrics/MetricsService.ts

@@ -143,6 +143,11 @@ export class MetricsService {
         this.createCounter("messages_sent_total", "Total messages sent", [
             "type",
         ])
+        this.createCounter(
+            "block_sync_messages_sent_total",
+            "POC block synchronization messages sent",
+            ["kind", "source"],
+        )
         this.createCounter(
             "messages_received_total",
             "Total messages received",

src/libs/blockchain/routines/Sync.ts

@@ -1232,7 +1232,7 @@ async function requestBlocks(): Promise<boolean> {
                 peer = next
                 continue
             }
-            await BroadcastManager.broadcastOurSyncData()
+            await BroadcastManager.broadcastOurSyncData("catchup_complete")
 
             // Trigger L2PS sync
             triggerL2PSSync(peer)
@@ -1528,7 +1528,7 @@ export async function fastSync(
 
             if (difference >= 2) {
                 getSharedState.syncStatus = false
-                await BroadcastManager.broadcastOurSyncData()
+                await BroadcastManager.broadcastOurSyncData("lag_signal")
                 log.debug(
                     "[fastSync] Network highest block is more than 2 blocks ahead of our highest block, setting sync status to false and broadcasting",
                 )
@@ -1591,7 +1591,7 @@ export async function fastSync(
         log.debug("[fastSync] Fast sync routine ended ⚪️⚪️⚪️⚪️⚪️⚪️⚪️⚪️⚪️")
         log.debug("[fastSync] Sync status: " + synced)
         getSharedState.syncStatus = synced
-        BroadcastManager.broadcastOurSyncData()
+        BroadcastManager.broadcastOurSyncData("fast_sync_complete")
 
         log.debug("[fastSync] Broadcasted our sync data 📤📤📤📤📤📤📤📤📤")
         const lastBlockNumber = await Chain.getLastBlockNumber()