[quinnj]

Summary

• replace the Avro.jl 1.x internals with the planned 2.0 schema, value, binary, container, resolution, typed, projection, and sort-order architecture

• enforce one bounded resource-accounting model across parsing, encoding, decoding, codecs, parallel reads, defaults, and public operations

• add prepared datum APIs, streaming rows, materialized tables, projection, single-object encoding, legacy-file support, codec extensions, and complete migration documentation

• add generated and Apache fixtures, Java and fastavro differential tests, cross-version exchange, performance and RSS gates, Aqua, JET, smoke tests, and the supported Julia CI matrix

• use registered DataDecimals 1.0 for ordinary decimal results and accept fixed-scale shared decimal inputs

• accept Durations 1.0 values with checked millisecond conversion; retain the full unsigned wire range in generic Avro duration results

Takeover audit

The recovered implementation had completed phases 0 through 5. A final independent audit fixed these release blockers:

• validate defaults only after recursive schema graphs are complete, using logical-type value domains
• compose max_depth across reader defaults, JSON encoding, and OCF writer staging
• keep single-object encode/decode, tojson, and compare under one public-operation work budget
• enforce max_bytes while skipping inactive sized-map keys
• track src/span.jl and enable the required Arrow 2 and DataFrames smoke job
• restore the aligned container writer's transactional fast path without weakening exact work limits or error diagnostics
• make the cross-platform release matrix portable across Julia versions, operating systems, architectures, and instrumented coverage

Breaking changes

• Avro.write(dst, x) now means container output. Use Avro.encode or Avro.encode! for datum encoding.
• Avro.Table replaces readtable for materialized columns. Avro.Rows provides streaming rows.
• generic enums, fixed values, records, unions, timestamps, decimals, and durations use the new 2.0 value types.
• schema derivation uses StructUtils customization.
• strict validation and fixed resource limits are enabled by default.
• readtable and writetable remain as one-cycle deprecation shims. The migration guide covers legacy Avro.jl 1.x files.

Shared-value validation

• 16 focused shared-value tests passed on Julia 1.10 and 1.12, including exact typed conversion and range errors.
• 62,477 full-suite assertions passed locally. One latency assertion exceeded 10 seconds under concurrent load; all 15 latency assertions passed separately, with a maximum of 7.49 seconds.
• Julia 1.10 and 1.11 package manifests and the documentation manifest resolve the registered dependencies.

Validation before shared-value adoption

• hosted CI for the prior rewrite head at e5c3509: 26 jobs passed, 0 failed, and the disabled Arrow 3 candidate job was the only planned skip
• supported matrix: Julia 1.10, 1.11, and current on Ubuntu, macOS, and Windows with 1 and 4 threads; prerelease, nightly, coverage, quality, docs, trim, cross-version, and interop jobs also passed
• Julia 1.12.6 local release suite with 4 threads, Aqua, JET, Arrow 2, and DataFrames smoke: 62,494/62,494
• compile-cost gate: 46,214 specializations, 0 new, 13.8 MB RSS growth
• latency gate: 6/6; slowest 4.61 seconds against 10 seconds
• performance gate: 16/16; one-million-row write 0.185 seconds, 5.1x Avro.jl 1.1.2
• parallel RSS gate: 6/6; 3,035.8 MB peak against 4,096 MB
• Apache datum differential: 58/58
• complete Java and fastavro interop matrix: 1,586/1,586
• cross-version exchange: 32/32 for Julia 1.10.11 to 1.12.6 and 32/32 in reverse
• strict Documenter build: passed

Writer destination stability

Opening a writer for out.avro and changing the working directory before close previously left the intended file unchanged and could replace out.avro in the new directory. The writer now resolves the physical parent when it opens and uses that same destination for temporary-file creation and publication. Parent symlinks and .. keep their filesystem meaning. Atomic writes still replace a final symlink while preserving its referent.

Focused regressions cover atomic and in-place writes, abort cleanup, and symlink paths. The API and container manual describe the directory selection at open time.

Validation for this correction:

• The unchanged head reproduced the wrong-folder publication on Julia 1.10.12 and 1.13.1, with all other controls passing.
• An independent snapshot passed 27 checks covering Unicode SubString paths, changes to the working directory or parent symlink, atomic and in-place writes, aborts, do-blocks, and caller-owned streams.
• Full Julia 1.10.12 suite: 62,476 checks passed. Full Julia 1.13.1 suite with Aqua, JET, and Arrow 2/DataFrames smoke tests: 62,530 checks passed. All normal fuzz, compile-cost, and latency gates remained enabled; the slowest current-runtime latency case was 5.39 seconds against 10 seconds.
• The first local current-runtime command exposed a JET dependency-activation issue reproduced identically on the unchanged head. Both heads passed analysis of all 2,582 definitions after explicit package activation with the same captured dependencies; the complete current suite then passed in that context without source or analysis-setting changes.
• Strict Documenter build and doctests passed. All 733 tracked-file hashes matched the independently reviewed snapshot.

The writer destination correction was researched and implemented with Codex. This bounded review covers the writer correction; review of the complete rewrite remains separate.

Quality analysis environment and current CI repairs

JET evaluates Avro in a virtual module during the quality gate. The package project remains available only during that analysis and is removed in finally. The analysis target and the requirement of zero reports are unchanged.

The test dependency range now admits JET 0.12 alongside the previous versions. This resolves the JuliaInterpreter precompilation failure before nightly native tests. JET 0.12 explicitly supports full analysis on stable Julia 1.12/1.13 and provides dependency-loading stubs on newer Julia. The stable quality job still performs real package analysis; the nightly native job never requested analysis.

The newer stable checker found captured local variables and implicit private input contracts that the prior checker did not report. Prepared readers, the enum constructor, and inspection diagnostics now use separate closure-local names. Nullable documentation values are read once. Binary comparison and row decoding express their existing decoder/schema contracts. Unnamed tuple fields and nongeneric column materialization receive the package's existing error types; metadata keys are validated before copying. All resource-accounting and decode paths are retained.

The overlapping-reservation test now compares the current unreserved OS-memory reading with Avro's available-memory reading after the reservation worker is parked. Its original 64 MiB payload, 24 MiB noise tolerance, lower bound, resident-storage checks, and failing-admission checks remain. Additional checks verify the exact pending counter, an upper bound, settlement restoration, and the resident buffer's lifetime.

A bounded Linux control reproduces the stale historical-baseline failure when an unrelated 96 MiB process exits normally. Its historical delta was -132,935,640 bytes although the current subtraction and pending counter both measured 67,108,904 bytes; all nine positive controls passed. This establishes a cause of the test's false failure, without identifying the external activity responsible for the earlier hosted observation.

Coverage uses the verified v7 uploader with job-scoped, conditional OIDC authentication and strict upload-error reporting. The previous successful job's logs recorded token-required upload failures. Coverage percentages remain informational; no coverage threshold, native gate, or quality assertion is changed.

Validation for this bounded correction:

• Stable Julia 1.13.1 Aqua: 10 checks passed. JET: all 2,818 definitions analyzed with zero reports, preserving the strict existing gate.
• Focused real-memory Limits checks: 144 passed on macOS Julia 1.13.1 and 143 on Linux Julia 1.13.0, with the original tolerance.
• Public constructor, generic/typed byte and IO reader, positional read, comparison, row/projection/partition, and unsupported-record controls: 32 passed on Julia 1.10.12 and installed nightly 1.14.0-DEV.3316.
• Avro and JET 0.12.3 precompile and load on the installed nightly. This is a dependency/native-load result, not a supported nightly analysis result.
• Strict current Documenter build and doctests passed.
• Full current Julia 1.13.1 four-thread, bounds-checked native suite: 62,505 checks passed, with all normal fuzz, compile-cost, and latency gates enabled. The slowest latency case was 4.85 seconds against the unchanged 10-second limit.
• Full minimum Julia 1.10.12 four-thread, bounds-checked native suite: 62,503 checks passed, including all normal fuzz, compile-cost, and latency gates. Compile accounting recorded 44,415 specializations, zero new specializations, and 17.1 MiB RSS growth; all six latency controls passed, with a maximum of 6.83 seconds against 10 seconds.
• Full installed-nightly 1.14.0-DEV.3316 native log: 62,505 checks passed in 63m30s, with 66,812 specializations, zero new specializations, and 6.8 MiB RSS growth. All six latency controls passed; the maximum was 4.73 seconds against 10 seconds. All 79 frozen source/test hashes still match. The inherited runner's shell exit status is unavailable, so this reports the complete successful native test log.
• The source-coverage upload for 42713fe3ceeac0d8e67feca2fc1bdbeb134b2012 was accepted and processed as one report at 90.04%. Its Windows Julia 1.11 four-thread job exposed the separate sampling-test issue below; its compile-cost and all six latency gates passed.

This bounded correction does not constitute approval of the complete rewrite. The required independent review remains a separate gate.

Consistent host-memory sample checks

The Windows Julia 1.11 test compared a host-memory result of 13,201,567,744 bytes with a later OS reading of 13,201,600,512 bytes. On Windows the helper returns the clamped OS reading, so memory released between those calls invalidates the test's inequality. The macOS comparison also used readings taken at different moments.

The internal helper now accepts optional sampled values while preserving its zero-argument OS sampling, integer saturation, and macOS reclaimable-memory maximum. Deterministic checks cover those same policies, and live platform queries still run. Admission limits, pending reservation subtraction, memory bounds, the 24 MiB noise allowance in the separate resident-memory test, and latency gates are unchanged.

The full Limits suite passes 148 assertions on actual Julia 1.10.12, 1.13.1, and installed nightly 1.14.0-DEV.3316. Stable quality validation passes Aqua's 10 checks and real JET analysis of all 2,820 definitions with zero reports. The full Julia 1.13.1 native suite at a285ada58da9fdb55568f2c774a52f1626360493 passed 62,509 checks, including compile-cost and all six latency gates (maximum 4.90 seconds against 10 seconds).

Worker tests under admitted memory budgets

Parallel decoding correctly falls back to serial operation when the available-memory ceiling cannot reserve the head block and a worker block. The worker-specific tests used 8 KiB data blocks with a 256 MiB output cap; a controlled planner fixture measured a minimum worker ceiling of 1,192,102,937 bytes. The hosted macOS Julia 1.10 test selected the serial path, so its worker-count and injected worker-failure assertions did not exercise their intended branches.

Worker-specific fixtures now use a 16 KiB block cap, 1 MiB output cap, and the existing 16 MiB minimum codec cap. Their measured worker reservation is 55,479,321 bytes, below the unchanged 88,113,152-byte first-unit guard. Every admitted reader therefore has enough budget for those fixture workers. The 64 MiB datum and byte defaults, wide-limit acceptance cases, 375 MB total-limit fixture, production admission policy, and native memory and latency gates remain unchanged.

Permanent controls cover the old worker-admission boundary one byte below and exactly at the threshold, worker admission at the fixture's minimum reader budget, and restoration of pending reservations. All existing worker, failure-ordering, and scheduling assertions remain.

Validation for this test-only correction:

• Full actual macOS Julia 1.10.12 native suite with four threads and bounds checking: 62,512 checks passed in 18m36s. Compile accounting recorded 44,419 specializations, zero new specializations, and 13.8 MiB RSS growth. All six latency controls passed; the maximum was 5.55 seconds against the unchanged 10-second limit.
• Focused parallel suite: 133 checks passed with four threads on Julia 1.10.12, 1.13.1, and installed nightly 1.14.0-DEV.3316. Current Julia with one thread passed 117 checks.
• All 733 tracked-file hashes match the independently reviewed snapshot.
• The complete hosted run for 0eb1e12a5e22dd383d7bc3a2abe375ae937dac0c passed all 26 active jobs. Raw logs confirm the native minimum/current/1.11 matrix, prerelease and nightly tests, real stable Aqua/JET and Arrow 2 smoke, Java/fastavro interoperability, cross-version exchange, strict docs, and trim smoke completed without errors. The nightly native suite passed 62,469 checks in 92m21s; the macOS minimum four-thread lane that exposed this fixture issue now passes 62,512 checks.
• The upload for that exact commit was accepted by JuliaData/Avro.jl and processed as the expected one coverage report, covering 13,776 of 15,298 lines (90.05%). Both coverage status checks pass. The existing disabled Arrow 3 candidate remains the active run's only planned skip; the duplicate same-repository pull-request run skips its jobs by the existing workflow condition.
• The repository still requires an independent approving review before merge. These bounded repairs and green checks do not constitute review of the complete rewrite.

Co-authored by Codex

AI disclosure: This work was prepared with assistance from OpenAI Codex.

Maintenance sweep (Oct 15, 2026)

Changes pushed to this branch as separate commits; every behavior fix has a regression test:

• Development material (Codex review rounds, plan, status record, benchmark baselines) and the pinned root and test Manifests are out of the repository. CI resolves dependencies like a registered package.
• Deflate blocks with bytes after the final block, which the Apache Python writer and fastavro 1.13.1 and earlier write, are read as Java and Python read them.
• Field, symbol, and branch indexes are sorted once, so wide schemas and tables work under default limits (before: 150 columns named x1…x150 failed Avro.write; 1,000 reverse-ordered fields failed parseschema).
• Encoding picks the union record branch a value fills completely (it dropped fields), and scalars are no longer encoded as one-element arrays.
• Typed decoding converts timestamp units exactly, rejects scalar-into-collection conversions that produced empty values, prefers a union branch's own Julia type, and decodes records into Dict targets.
• Streaming readers check exact block consumption in fast mode; extra columns in later partitions raise; IOBuffers over offset vectors read from their first byte; atomic writes keep normal file modes.
• select= on a recursive record raises instead of mis-describing nested values; a failed union branch inside a recursive record no longer recurses until a stack overflow; minsize is exact for mutually recursive records (files Avro.jl wrote were unreadable); record.schema/record.values return fields of those names when the record has them; Base.write(::Writer, x) and the empty TimeZones extension are gone.
• README, manual, and docstrings corrected against probes; dead code removed.

Closes #6
Closes #13
Closes #15
Closes #17
Closes #18
Closes #19

🤖 Generated with Claude Code

[quinnj]

[Reviewed by AI] Reviewed container framing at e9691e7. No actionable findings in this scope. All-prefix truncation and sync-corruption checks passed across four built-in codecs and byte/stream sources on Julia 1.10 and 1.12 (7,888 checks each). Existing binary, container, and parallel-read tests also passed. This review does not cover the entire rewrite.

[codecov]

Codecov Report

❌ Patch coverage is 90.53390% with 867 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.66%. Comparing base (7c8e4b1) to head (a72b123).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/parallel.jl	51.23%	217 Missing ⚠️
src/plan_write.jl	87.21%	129 Missing ⚠️
src/resolution.jl	89.89%	102 Missing ⚠️
src/plan_read.jl	91.23%	49 Missing ⚠️
src/span.jl	93.68%	48 Missing ⚠️
src/frozen.jl	83.41%	34 Missing ⚠️
src/jsonreader.jl	93.85%	34 Missing ⚠️
src/tables.jl	94.58%	33 Missing ⚠️
src/jsonencoding.jl	95.00%	26 Missing ⚠️
src/limits.jl	94.36%	26 Missing ⚠️
... and 15 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #21       +/-   ##
===========================================
+ Coverage   77.43%   90.66%   +13.23%     
===========================================
  Files          13       33       +20     
  Lines         811    15411    +14600     
===========================================
+ Hits          628    13973    +13345     
- Misses        183     1438     +1255     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.