fee/aesstream: chunked AES-256-GCM STREAM encrypt/decrypt (FIL-470)#4
Conversation
Implement the FEE body cipher: fixed-size plaintext chunks each sealed
with AES-256-GCM under a per-segment CEK, using the STREAM nonce layout
fixed by the FilOne encryption RFC and FIP discussion #1253:
nonce[12] = baseNonce[7] || chunkIndex[4, big-endian] || lastFlag[1]
This is just the raw body cipher (key + 7-byte base nonce + AAD + chunk
size in, ciphertext stream out); the COSE envelope, IV generation and
CEK key-wrapping live in the parent fee package (FIL-469/471/547).
- Writer/Reader are streaming io wrappers with O(chunk-size) memory; a
full chunk is held back until more plaintext arrives so the final
chunk can be flagged last (Close flushes it, mirroring age).
- The Reader detects the final chunk by retrying a full-size chunk with
the last flag set, converts a chunk-boundary EOF into a truncation
error, and rejects trailing data after a full-size final chunk.
- Truncation, chunk reordering and any bit flip all fail authentication
rather than yielding plaintext; the empty plaintext is one empty final
chunk (16-byte tag), so ciphertext is never zero-length.
- Default chunk size 256 KiB (configurable 4 KiB..16 MiB); the 4-byte
chunk counter caps a stream at 2^32 chunks rather than wrapping.
Includes Config validation, NewBaseNonce, EncryptedSize and one-shot
Seal/Open helpers, plus black-box and white-box tests covering round
trips across chunk boundaries, truncation/reorder/tamper/trailing-data
detection, the exact nonce wire format, the counter-overflow guard,
and O(chunk-size) memory under a 128 MiB stream (93% coverage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
7541546 to
83a1ff8
Compare
There was a problem hiding this comment.
Pull request overview
Adds the fee/aesstream package implementing FilOne’s chunked AES-256-GCM STREAM body cipher (Writer/Reader + one-shot helpers) as the foundation for the higher-level FEE envelope work.
Changes:
- Introduces core streaming encryption/decryption types (
Writer,Reader) and the sharedConfig/nonce construction. - Adds convenience helpers (
Seal,Open,EncryptedSize,NewBaseNonce) for one-shot usage and sizing. - Adds comprehensive black-box and internal white-box tests covering boundary cases, tampering/truncation, and allocation/memory behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| fee/aesstream/aesstream.go | Defines the AES-GCM STREAM format, config/validation, nonce layout, helpers (Seal/Open/EncryptedSize/NewBaseNonce), and exported errors/constants. |
| fee/aesstream/writer.go | Implements chunked streaming encryption and ciphertext writing. |
| fee/aesstream/reader.go | Implements chunked streaming decryption with truncation/tamper/trailing-data detection. |
| fee/aesstream/aesstream_test.go | Black-box tests for round-trips, boundary conditions, tamper/truncation/reorder, sizing, and bounded-memory streaming. |
| fee/aesstream/internal_test.go | Internal tests pinning nonce wire format, chunk-count guards, and per-chunk allocation behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Respond to PR review feedback on the new package: - NewWriter/NewReader now copy cfg.AAD (append([]byte(nil), ...)) instead of aliasing it. AAD is authenticated on every chunk, so a caller that mutates or reuses the slice after construction must not change a stream's authentication. Key and BaseNonce were already copied; this closes the gap and removes a latent data-race footgun. - writeAll returns io.ErrShortWrite on a zero-progress, no-error write rather than looping forever. A compliant io.Writer never does this, but the interface doesn't strictly forbid it, and a hang in the encrypt path is worth guarding against. Tests: TestAADIsCopiedOnConstruction (mutates the caller's AAD right after construction and confirms the stream still authenticates under the original value) and TestWriteAllShortWriteGuard (would hang without the fix). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Fold TestRoundTripDefaultChunkSize into TestRoundTrip as t.Run subtests over a cases slice that carries each chunk size alongside its boundary-tuned plaintext sizes. The default-chunk case keeps ChunkSize 0 to exercise the 0 -> DefaultChunkSize resolution, and now also gets the EncryptedSize length cross-check (EncryptedSize resolves a 0 chunk size to the default). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Replace the hand-rolled if/t.Fatalf/t.Errorf assertions in the aesstream test files with testify require.* — house style in other fil-forge repos. testify was already an indirect dependency; go mod tidy promotes it to direct. The allocation-measurement closures in the steady-state tests keep their plain error checks so require's per-call bookkeeping cannot perturb AllocsPerRun. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Fold the empty-plaintext and full-size-final-chunk tests into TestRoundTrip's cases; inline the mustSeal helper (errors handled at each call site); assert io.CopyBuffer's returned byte count; add a one-byte-at-a-time decrypt pass that exercises the Reader assembling chunks from short underlying reads; and merge the five *Fails tests into one table-driven TestDecryptFailures, where each case either corrupts the ciphertext or overrides the decrypt config and all share the released-bytes-are-a-genuine-prefix invariant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Rename openChunk's input from last to mustBeLast (the caller's constraint: a short read means the chunk must be the final one) and return wasLast (the determined fact), so the parameter and result no longer share one overloaded name. Pass the chunk index as a parameter too, so openChunk's inputs are the per-chunk (index, ciphertext, mustBeLast) and the receiver supplies only the stream's fixed parameters. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Seal pre-grows the output buffer to the exact ciphertext size, but on a 32-bit platform a near-2 GiB plaintext can make the int64 to int conversion overflow negative, which would panic bytes.Buffer.Grow. Skip the (optional) pre-grow when the size does not fit in int. No behavior change on 64-bit. Raised by Copilot review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
frrist
left a comment
There was a problem hiding this comment.
I'm going to sound like a broken record across all these crypto PRs; I lack the depth of knowledge to determine is this is "correct", though it looks good to me:
Approving so this isn't blocked. But flagging, as a follow-up rather than an objection: this is hand-rolled cryptography, and the blast radius if any of it is subtly wrong is large.
This is the stream encryption. The FEE FIP specifies two algorithms for encryption: AES-256-GCM, which is well-known and IANA-registered, and Chunked AES-256-GCM with STREAM construction, which is defined in the FIP in terms of AES-256-GCM over a chunking scheme, to allow for random access. We're using the latter, which isn't implemented in any off-the-shelf package yet. Luckily, the chunking is pretty straightforward, and AES-256-GCM itself is supported by Go's
crypto/cipher.Unfortunately, being a bespoke configuration, even over standard AES, there are no published fixtures to test against, but the tests are pretty thorough, with lots of round-tripping.
Implements FIL-470 — the raw chunked AES-256-GCM STREAM body cipher for the FilOne File Encryption Envelope (FEE), in
fee/aesstream/. The parentfeepackage (COSE envelope, key-wrapping) will build on this.Wire format
Per the FilOne encryption RFC (fil-one/RFC#9) and FIP #1253:
baseNonce[7] ‖ chunkIndex[4, big-endian] ‖ lastFlag[1]Enc_structure); chunks are differentiated by the counter, not per-chunk AADScope
Raw body cipher only —
Config{Key, BaseNonce, AAD, ChunkSize}in, ciphertext stream out. The COSE_Encrypt envelope (FIL-469), IV generation, and CEK key-wrapping (FIL-471/547) live in the parentfeepackage.API
Writer/Reader— streamingio.WriteCloser/io.Reader, O(chunk-size) memorySeal/Open— one-shot convenience helpersConfig(+ validation),NewBaseNonce,EncryptedSizeAcceptance criteria
io.EOF)Testing
Black-box + white-box suites: round-trips across chunk boundaries (empty, ±1 chunk, exact multiples, odd remainders), truncation / reorder / tamper / trailing-data detection, the full-size-last-chunk retry path, the exact nonce wire format, the counter-overflow guard, a 128 MiB bounded-memory stream, and zero-allocation steady state.
-raceclean, 93.4% statement coverage (remainder is unreachable defensive code).A design seam for FIL-472 (range decryption) is intentionally left in place: the nonce/chunk math is isolated so range reads can determine the last-chunk flag from the envelope's byte length rather than the streaming reader's retry path.
🤖 Generated with Claude Code
https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Generated by Claude Code