bench: expose #1242 update batch-shape measurements

[snissn]

Summary

Draft measurement PR for #1242. This gives the batch-shape and phase timing work a stable home while the preceding optimization stack continues to settle. The diff points at main intentionally; it is expected to shrink as the PR3b/benchmark stack merges.

This PR adds:

• update-batch semantic record build timing
• update-buffer semantic append timing
• indexed flush materialization breakdown timing:
  • semantic plan
  • publish input build
  • plan stats
• benchmark/report propagation for those counters
• direct TreeDB benchmark coverage comparing concurrent Update combiner behavior against explicit UpdateBatch sizes of 1, 8, 64, 512, and 5000

Current Finding

The default concurrent Update path is not reliably producing large batches in the 5k docs / 50k indexed repeated-ID canary.

With 8 writers:

case	main ns/op	aggregate ns/op	actual items/batch
concurrent Update	3607	4013	~7.5
explicit UpdateBatch, batch 64	2620	2913	~64
explicit UpdateBatch, batch 512	2529	2774	~510
explicit UpdateBatch, batch 5000	2540	2800	5000

With 64 writers, the combiner reaches roughly 37-39 items/batch and throughput improves, but it still does not approach the 512+ item regime that best exercises the #1242 memtable/B-tree batching model.

The aggregate branch does reduce root work as expected:

metric	main	aggregate
root_delta_plan_entries/doc	1.2	0.3

But current default batching is too small to fully amortize the semantic/materialization overhead.

Validation

git diff --check

go test ./TreeDB/collections \
  -run 'TestCollectionUpdateBufferBreakdownStatsSnapshotAndAdd|TestCollectionUpdateBatchStatsExposeIndexRunShape|TestPR3bSemantic' \
  -count=1

go test ./cmd/mongo_gateway_bench \
  -run 'TestTreeDBStatsDeltaAndPhaseMetrics|TestDeriveTreeDBPhaseMetricsEmitsZeroValues|TestTreeDBStatsDeltaPreservesZeroMetrics|TestRunTreeDBProfiledPhaseDrainsBeforeStatsSnapshot|TestDeltaCollectionManagerUpdateStats|TestReportCollectionManagerUpdateStats|TestProfileBench' \
  -count=1

Benchmark artifacts from local runs:

• aggregate batch-shape run: /tmp/gomap_batch_shape_aggregate_1778010321
• main batch-shape run: /tmp/gomap_batch_shape_main_1778010457
• aggregate 64-writer run: /tmp/gomap_batch_shape_aggregate_default_w64_1778010572
• main 64-writer run: /tmp/gomap_batch_shape_main_default_w64_1778010590

Notes

This is measurement-first. It should not change collection semantics. The intended follow-up decision is whether #1242 needs a stronger default batching contract above the current opportunistic synchronous update combiner.

[snissn]

Drill-down update on explicit UpdateBatch performance measurement.

I added measurement-only counters for public/scaffolding and flush-preparation buckets, then reran the direct engine benchmark:

MONGO_GATEWAY_PROFILE_BENCH_UPDATE_DOCUMENTS=5000 \
MONGO_GATEWAY_PROFILE_BENCH_BATCH_SIZE=5000 \
MONGO_GATEWAY_PROFILE_BENCH_WRITERS=8 \
MONGO_GATEWAY_PROFILE_BENCH_BUFFERED_INDEXED_WRITE_MAX_DOCUMENTS=100000 \
go test ./cmd/mongo_gateway_bench \
  -run '^$' \
  -bench '^BenchmarkDirectCollectionUpdateBatchBSONIndexes2CityUpdate/batch_(64|512|5000)$' \
  -benchtime=50000x -benchmem -count=3 -p=1

Key means from this run:

batch	ns/op	UpdateBatch call	final FlushAll	flush rotate/freeze	flush materialize	flush publish
64	3305	2432 ns/doc	835 ns/doc	573 ns/doc	127 ns/doc	130 ns/doc
512	3053	2240 ns/doc	781 ns/doc	556 ns/doc	123 ns/doc	99 ns/doc
5000	2955	2161 ns/doc	765 ns/doc	537 ns/doc	111 ns/doc	108 ns/doc

The important finding: the final drain gap is now accounted for. FlushAll wall time is essentially:

indexed_flush_rotate_ns/doc + indexed_flush.duration_ns/doc

rotateIndexedMutableToFlushUnitLocked calls Freeze() on the mutable root tables, so the dominant final-drain cost in this canary is mutable run table freeze/sort, not root apply. Root apply is roughly 90-140 ns/doc here, while flush rotate/freeze is roughly 535-575 ns/doc.

Per-update dominant named costs at batch 5000:

callback/BSON mutation        ~409 ns/doc
BSON _id validation           ~365 ns/doc
index-state extraction        ~296 ns/doc
buffer stage                  ~199 ns/doc
prepare replacement           ~147 ns/doc
semantic record build          ~88 ns/doc
buffered-read snapshot         ~91 ns/doc
current read                   ~76 ns/doc

This makes the next optimization discussion more concrete: for this explicit UpdateBatch canary, the largest collection-side opportunity is the flush-boundary mutable table freeze/sort path. The per-item path still has clear costs, especially BSON mutation/validation and index-state extraction, but root apply is no longer the first bottleneck in this benchmark.

[snissn]

#1242 benchmark acceptance summary

Raw artifacts: /tmp/gomap_1242_acceptance_1778014283

Comparison setup: stack branch has full #1242 measurement counters; main uses a compatible temporary benchmark scaffold. Main indexed_flush_rotate_ns/doc is reported as 0 because that subphase counter does not exist before the stack instrumentation.

Collection flush-after-staging, docs_5000

shape	main ns/doc	stack ns/doc	stack vs main	main root_apply ns/doc	stack root_apply ns/doc	main B/op	stack B/op	main allocs/op	stack allocs/op	main raw entries/doc	stack raw entries/doc	main final entries/doc	stack final entries/doc	stack rotate ns/doc	stack materialize ns/doc	stack publish ns/doc
repeated_same_id_indexed_change_back	2,078	1,659	-20.2%	30.7	7.286	5,338,426	3,481,330	52,736	40,228	4.000	4.000	0.001	0.000	0.142	325	46.6
repeated_same_id_non_indexed_update	818	766	-6.3%	3.217	3.853	1,773,128	1,773,506	20,158	20,159	1.000	1.000	0.000	0.000	0.139	269	33.3
many_ids_indexed_changes	628	1,278	+103.6%	166	180	1,671,485	9,865,162	457	25,775	4.000	4.660	4.660	4.660	0.086	1,062	200
many_ids_non_indexed_changes	295	195	-33.8%	95.0	125	174,314	228,365	196	196	1.000	1.000	1.000	1.000	0.067	49.8	139

DB ordered-root delta-shape, docs_5000

shape	branch	ns/doc	B/op	allocs/op	raw entries/doc	final entries/doc	root_apply ns/doc	root_apply ops/doc
repeated_same_id_indexed_change_back_raw	main	76.6	707,077	98.0	1.000	1.000	57.8	1.000
repeated_same_id_indexed_change_back_coalesced	main	0.003	0.000	0.000	1.000	0.000	n/a	n/a
repeated_same_id_non_indexed_update_raw	main	75.3	707,925	99.0	1.000	1.000	64.8	1.000
repeated_same_id_non_indexed_update_coalesced	main	19.1	641,832	72.0	1.000	0.000	2.428	0.000
many_ids_indexed_changes_raw	main	72.6	742,928	97.0	1.000	1.000	62.4	1.000
many_ids_indexed_changes_coalesced	main	70.0	741,858	99.0	1.000	1.000	54.3	1.000
many_ids_non_indexed_changes_raw	main	74.5	743,733	101	1.000	1.000	56.7	1.000
many_ids_non_indexed_changes_coalesced	main	76.1	708,485	100	1.000	1.000	58.4	1.000
repeated_same_id_indexed_change_back_raw	stack	80.4	708,024	99.0	1.000	1.000	57.7	1.000
repeated_same_id_indexed_change_back_coalesced	stack	0.006	0.000	0.000	1.000	0.000	n/a	n/a
repeated_same_id_non_indexed_update_raw	stack	73.9	707,805	99.0	1.000	1.000	58.0	1.000
repeated_same_id_non_indexed_update_coalesced	stack	25.3	641,885	73.0	1.000	0.000	2.311	0.000
many_ids_indexed_changes_raw	stack	63.6	741,810	95.0	1.000	1.000	52.6	1.000
many_ids_indexed_changes_coalesced	stack	76.0	742,637	101	1.000	1.000	55.8	1.000
many_ids_non_indexed_changes_raw	stack	68.1	743,320	100	1.000	1.000	56.6	1.000
many_ids_non_indexed_changes_coalesced	stack	73.5	707,413	98.0	1.000	1.000	56.4	1.000

Decision read

• Target collection shape raw/final collapse: main final 0.001 entries/doc, stack raw/final 4.000 -> 0.000 entries/doc.
• Target collection shape flush time: main 2,078 ns/doc, stack 1,659 ns/doc (-20.2%).
• DB main non-indexed repeated raw vs coalesced root_apply: 64.8 -> 2.428 ns/doc.
• DB stack non-indexed repeated raw vs coalesced root_apply: 58.0 -> 2.311 ns/doc.
• Preliminary decision: the root-delta shape benchmark proves smaller final deltas are much cheaper to apply. The collection flush benchmark shows a win in the repeated same-ID indexed target and many-ID non-indexed shape, but a severe regression for many-ID indexed changes due to stack-side materialization/allocation overhead. This is not yet a blanket acceptance; next work should target collection freeze/sort/materialization and semantic/allocation overhead before landing complexity-heavy pieces.

[snissn]

#1242 benchmark acceptance summary

Raw artifacts: /tmp/gomap_1242_acceptance_1778014283

Comparison setup: stack branch has full #1242 measurement counters; main uses a compatible temporary benchmark scaffold. Main indexed_flush_rotate_ns/doc is reported as 0 because that subphase counter does not exist before the stack instrumentation. Stack includes the semantic planner gate/fixed single-value diff optimization from this run.

Collection flush-after-staging, docs_5000

shape	main ns/doc	stack ns/doc	stack vs main	main root_apply ns/doc	stack root_apply ns/doc	main B/op	stack B/op	main allocs/op	stack allocs/op	main raw entries/doc	stack raw entries/doc	main final entries/doc	stack final entries/doc	stack rotate ns/doc	stack materialize ns/doc	stack publish ns/doc
repeated_same_id_indexed_change_back	2,078	1,644	-20.9%	30.7	7.442	5,338,426	3,807,101	52,736	40,233	4.000	4.000	0.001	0.000	0.164	320	46.4
repeated_same_id_non_indexed_update	818	746	-8.8%	3.217	3.647	1,773,128	2,101,506	20,158	20,176	1.000	1.000	0.000	0.000	0.139	263	49.8
many_ids_indexed_changes	628	352	-43.9%	166	157	1,671,485	1,747,626	457	462	4.000	4.660	4.660	4.660	0.086	163	174
many_ids_non_indexed_changes	295	181	-38.6%	95.0	107	174,314	529,720	196	211	1.000	1.000	1.000	1.000	0.069	64.2	119

DB ordered-root delta-shape, docs_5000

shape	branch	ns/doc	B/op	allocs/op	raw entries/doc	final entries/doc	root_apply ns/doc	root_apply ops/doc
repeated_same_id_indexed_change_back_raw	main	76.6	707,077	98.0	1.000	1.000	57.8	1.000
repeated_same_id_indexed_change_back_coalesced	main	0.003	0.000	0.000	1.000	0.000	n/a	n/a
repeated_same_id_non_indexed_update_raw	main	75.3	707,925	99.0	1.000	1.000	64.8	1.000
repeated_same_id_non_indexed_update_coalesced	main	19.1	641,832	72.0	1.000	0.000	2.428	0.000
many_ids_indexed_changes_raw	main	72.6	742,928	97.0	1.000	1.000	62.4	1.000
many_ids_indexed_changes_coalesced	main	70.0	741,858	99.0	1.000	1.000	54.3	1.000
many_ids_non_indexed_changes_raw	main	74.5	743,733	101	1.000	1.000	56.7	1.000
many_ids_non_indexed_changes_coalesced	main	76.1	708,485	100	1.000	1.000	58.4	1.000
repeated_same_id_indexed_change_back_raw	stack	80.4	708,024	99.0	1.000	1.000	57.7	1.000
repeated_same_id_indexed_change_back_coalesced	stack	0.006	0.000	0.000	1.000	0.000	n/a	n/a
repeated_same_id_non_indexed_update_raw	stack	73.9	707,805	99.0	1.000	1.000	58.0	1.000
repeated_same_id_non_indexed_update_coalesced	stack	25.3	641,885	73.0	1.000	0.000	2.311	0.000
many_ids_indexed_changes_raw	stack	63.6	741,810	95.0	1.000	1.000	52.6	1.000
many_ids_indexed_changes_coalesced	stack	76.0	742,637	101	1.000	1.000	55.8	1.000
many_ids_non_indexed_changes_raw	stack	68.1	743,320	100	1.000	1.000	56.6	1.000
many_ids_non_indexed_changes_coalesced	stack	73.5	707,413	98.0	1.000	1.000	56.4	1.000

Decision read

• Target collection shape raw/final collapse: main final 0.001 entries/doc, stack raw/final 4.000 -> 0.000 entries/doc.
• Target collection shape flush time: main 2,078 ns/doc, stack 1,644 ns/doc (-20.9%).
• DB main non-indexed repeated raw vs coalesced root_apply: 64.8 -> 2.428 ns/doc.
• DB stack non-indexed repeated raw vs coalesced root_apply: 58.0 -> 2.311 ns/doc.
• Current decision: the stack now shows the intended target-shape win and avoids the prior many-ID indexed regression. This is much closer to acceptance, but the repeated same-ID target still carries high allocation count from staging semantic records; next polish should reduce that allocation cost or explicitly document it as outside the flush timer before treating the whole stack as final.

[snissn]

#1242 benchmark acceptance summary

Raw artifacts: /tmp/gomap_1242_acceptance_1778014283

Comparison setup: stack branch has the #1242 measurement counters and semantic publish gating. Main uses a compatible temporary benchmark scaffold. Main indexed_flush_rotate_ns/doc is reported as 0 because that subphase counter does not exist before the stack instrumentation.

Values below are medians of the saved -count=3 runs for docs_5000.

Collection flush-after-staging, docs_5000

shape	main ns/doc	stack ns/doc	stack vs main	main root_apply ns/doc	stack root_apply ns/doc	main B/op	stack B/op	main allocs/op	stack allocs/op	main raw entries/doc	stack raw entries/doc	main final entries/doc	stack final entries/doc	stack rotate ns/doc	stack materialize ns/doc	stack publish ns/doc
repeated_same_id_indexed_change_back	2,078	1,651	-20.5%	30.7	7.197	5,338,426	3,800,957	52,736	40,232	4.000	4.000	0.0008	0.0004	0.1527	335	42.8
repeated_same_id_non_indexed_update	818	754	-7.7%	3.217	3.233	1,773,128	1,773,336	20,158	20,159	1.000	1.000	0.0002	0.0002	0.1195	262	46.5
many_ids_indexed_changes	628	361	-42.4%	166	166	1,671,485	1,634,386	457	463	4.000	4.660	4.660	4.660	0.1278	160	186
many_ids_non_indexed_changes	295	151	-48.9%	95.0	91.0	174,314	226,882	196	193	1.000	1.000	1.000	1.000	0.0695	46.0	103

DB ordered-root delta-shape, docs_5000

shape	branch	ns/doc	B/op	allocs/op	raw entries/doc	final entries/doc	root_apply ns/doc	root_apply ops/doc
repeated_same_id_indexed_change_back_raw	main	76.6	707,077	98.0	1.000	1.000	57.8	1.000
repeated_same_id_indexed_change_back_coalesced	main	0.0027	0.0000	0.0000	1.000	0.0000	n/a	n/a
repeated_same_id_non_indexed_update_raw	main	75.3	707,925	99.0	1.000	1.000	64.8	1.000
repeated_same_id_non_indexed_update_coalesced	main	19.1	641,832	72.0	1.000	0.0002	2.428	0.0002
many_ids_indexed_changes_raw	main	72.6	742,928	97.0	1.000	1.000	62.4	1.000
many_ids_indexed_changes_coalesced	main	70.0	741,858	99.0	1.000	1.000	54.3	1.000
many_ids_non_indexed_changes_raw	main	74.5	743,733	101	1.000	1.000	56.7	1.000
many_ids_non_indexed_changes_coalesced	main	76.1	708,485	100	1.000	1.000	58.4	1.000
repeated_same_id_indexed_change_back_raw	stack	80.4	708,024	99.0	1.000	1.000	57.7	1.000
repeated_same_id_indexed_change_back_coalesced	stack	0.0055	0.0000	0.0000	1.000	0.0000	n/a	n/a
repeated_same_id_non_indexed_update_raw	stack	73.9	707,805	99.0	1.000	1.000	58.0	1.000
repeated_same_id_non_indexed_update_coalesced	stack	25.3	641,885	73.0	1.000	0.0002	2.311	0.0002
many_ids_indexed_changes_raw	stack	63.6	741,810	95.0	1.000	1.000	52.6	1.000
many_ids_indexed_changes_coalesced	stack	76.0	742,637	101	1.000	1.000	55.8	1.000
many_ids_non_indexed_changes_raw	stack	68.1	743,320	100	1.000	1.000	56.6	1.000
many_ids_non_indexed_changes_coalesced	stack	73.5	707,413	98.0	1.000	1.000	56.4	1.000

Decision read

• Target collection shape raw/final collapse: stack raw/final 4.000 -> 0.0004 entries/doc; main final was 0.0008 entries/doc.
• Target collection shape flush time: main 2,078 ns/doc, stack 1,651 ns/doc (-20.5%).
• DB main non-indexed repeated raw vs coalesced root_apply: 64.8 -> 2.428 ns/doc.
• DB stack non-indexed repeated raw vs coalesced root_apply: 58.0 -> 2.311 ns/doc.
• Many-ID indexed guardrail: main 628 ns/doc, stack 361 ns/doc (-42.4%), with stack final/raw entries unchanged at 4.660/4.660 entries/doc.
• Many-ID non-indexed guardrail after zero-delta semantic-record pruning: main 295 ns/doc, stack 151 ns/doc (-48.9%); allocation count remains comparable (196 vs 193 allocs/op), while B/op is now in the same rough range across noisy runs.

Current decision: the benchmark suite now demonstrates the intended root-delta/root-apply collapse and a measurable collection flush-after-staging win for the target repeated-ID indexed workload. The stack also no longer shows the earlier many-ID indexed slowdown. Remaining follow-up is to keep reducing allocation cost in the repeated same-ID indexed path, but the acceptance evidence is now strong enough to treat the stack as a real, attributable inner-engine win rather than only an e2e/noise story.

[snissn]

Update pushed: 4ecfcacfe1 (collections: trim UpdateBatch hot allocations).

Local validation:

go test ./TreeDB/collections -run 'TestDirectBufferedRootEntriesOwnKeysAndRetainDocumentArena|TestPR3bSemantic|TestCollectionUpdateBatchStatsExposeIndexRunShape|TestCollectionUpdateBufferBreakdownStatsSnapshotAndAdd|TestCollectionBufferedTemplateV1|TestTemplateV1|TestUpdateBatch|TestUpdateCombiner|TestCollectionUpdateCombiner|TestFreezeSortRunTable' -count=1
PASS

go test ./TreeDB/collections ./TreeDB/db -count=1
PASS

Benchmark comparison against the pre-patch branch head 01c0f05c02, same machine/command:

go test ./TreeDB/collections -run '^$' \
  -bench '^BenchmarkCollectionUpdateBatchDirectBufferedTemplateV1NewShape/batch_(80|16000)$' \
  -benchmem -benchtime=5000x -count=5

Key allocation results:

case	before	after
batch_80 allocs/op	22	13
batch_80 B/op	~3210-3222	~3070-3079
batch_16000 allocs/op	19	11
batch_16000 B/op	~3108-3110	~2232-2233

Timing is still noisy, but the direction is favorable in the large-batch canary: branch-head batch_16000 runs were mostly ~2371-2488 ns/doc, and the patched runs were mostly ~2128-2238 ns/doc. The main intended improvement here is allocation pressure reduction in the explicit UpdateBatch path.

Changes made:

• Arena-clone UpdateBatchItem.DocumentID values during public batch ownership cloning.
• Avoid []byte -> string allocation in duplicate-ID validation; use exact hash plus byte-compare collision handling.
• Avoid BSON _id snapshot/validation work on non-BSON formats.
• Skip per-document overlay-root checks when the collection primary root has no overlays.
• Use array-keyed TemplateV1 resolver caches instead of string(id[:]) map keys.
• Arena-copy semantic index value sets instead of cloning each encoded value separately.
• Pre-grow freezeSortRunTable entry storage only for large ApplyStealEntryFunc calls; a small-batch regression was observed with unconditional preallocation, so it is threshold-gated.

CI is running on the pushed commit now.