Skip to content

fee/aesstream: chunked AES-256-GCM STREAM encrypt/decrypt (FIL-470)#4

Merged
Peeja merged 10 commits into
mainfrom
fil-470-fee-go-chunked-aes-256-gcm-stream-encryptdecrypt
Jun 26, 2026
Merged

fee/aesstream: chunked AES-256-GCM STREAM encrypt/decrypt (FIL-470)#4
Peeja merged 10 commits into
mainfrom
fil-470-fee-go-chunked-aes-256-gcm-stream-encryptdecrypt

Conversation

@Peeja

@Peeja Peeja commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

This is the stream encryption. The FEE FIP specifies two algorithms for encryption: AES-256-GCM, which is well-known and IANA-registered, and Chunked AES-256-GCM with STREAM construction, which is defined in the FIP in terms of AES-256-GCM over a chunking scheme, to allow for random access. We're using the latter, which isn't implemented in any off-the-shelf package yet. Luckily, the chunking is pretty straightforward, and AES-256-GCM itself is supported by Go's crypto/cipher.

Unfortunately, being a bespoke configuration, even over standard AES, there are no published fixtures to test against, but the tests are pretty thorough, with lots of round-tripping.


Implements FIL-470 — the raw chunked AES-256-GCM STREAM body cipher for the FilOne File Encryption Envelope (FEE), in fee/aesstream/. The parent fee package (COSE envelope, key-wrapping) will build on this.

Wire format

Per the FilOne encryption RFC (fil-one/RFC#9) and FIP #1253:

  • AES-256-GCM, 32-byte CEK, 16-byte tag
  • Default 256 KiB chunks (configurable 4 KiB – 16 MiB)
  • Nonce: baseNonce[7] ‖ chunkIndex[4, big-endian] ‖ lastFlag[1]
  • AAD is constant per stream (the COSE Enc_structure); chunks are differentiated by the counter, not per-chunk AAD
  • Empty plaintext → one empty final chunk (16-byte tag), so the ciphertext is never zero-length
  • The 4-byte counter caps a stream at 2³² chunks rather than silently wrapping a nonce

Scope

Raw body cipher only — Config{Key, BaseNonce, AAD, ChunkSize} in, ciphertext stream out. The COSE_Encrypt envelope (FIL-469), IV generation, and CEK key-wrapping (FIL-471/547) live in the parent fee package.

API

  • Writer / Reader — streaming io.WriteCloser / io.Reader, O(chunk-size) memory
  • Seal / Open — one-shot convenience helpers
  • Config (+ validation), NewBaseNonce, EncryptedSize

Acceptance criteria

  • ✅ Round-trips arbitrary-size streams; chunk size defaults to 256 KiB
  • ✅ Truncation → error rather than partial plaintext (callers trust only a clean io.EOF)
  • ✅ Reordered chunks → error
  • ✅ Bit flip anywhere in a chunk → error
  • ✅ Encrypt larger than memory without OOM — heap stays O(chunk size), not O(stream size)

Testing

Black-box + white-box suites: round-trips across chunk boundaries (empty, ±1 chunk, exact multiples, odd remainders), truncation / reorder / tamper / trailing-data detection, the full-size-last-chunk retry path, the exact nonce wire format, the counter-overflow guard, a 128 MiB bounded-memory stream, and zero-allocation steady state. -race clean, 93.4% statement coverage (remainder is unreachable defensive code).

A design seam for FIL-472 (range decryption) is intentionally left in place: the nonce/chunk math is isolated so range reads can determine the last-chunk flag from the envelope's byte length rather than the streaming reader's retry path.

🤖 Generated with Claude Code

https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL


Generated by Claude Code

Implement the FEE body cipher: fixed-size plaintext chunks each sealed
with AES-256-GCM under a per-segment CEK, using the STREAM nonce layout
fixed by the FilOne encryption RFC and FIP discussion #1253:

    nonce[12] = baseNonce[7] || chunkIndex[4, big-endian] || lastFlag[1]

This is just the raw body cipher (key + 7-byte base nonce + AAD + chunk
size in, ciphertext stream out); the COSE envelope, IV generation and
CEK key-wrapping live in the parent fee package (FIL-469/471/547).

- Writer/Reader are streaming io wrappers with O(chunk-size) memory; a
  full chunk is held back until more plaintext arrives so the final
  chunk can be flagged last (Close flushes it, mirroring age).
- The Reader detects the final chunk by retrying a full-size chunk with
  the last flag set, converts a chunk-boundary EOF into a truncation
  error, and rejects trailing data after a full-size final chunk.
- Truncation, chunk reordering and any bit flip all fail authentication
  rather than yielding plaintext; the empty plaintext is one empty final
  chunk (16-byte tag), so ciphertext is never zero-length.
- Default chunk size 256 KiB (configurable 4 KiB..16 MiB); the 4-byte
  chunk counter caps a stream at 2^32 chunks rather than wrapping.

Includes Config validation, NewBaseNonce, EncryptedSize and one-shot
Seal/Open helpers, plus black-box and white-box tests covering round
trips across chunk boundaries, truncation/reorder/tamper/trailing-data
detection, the exact nonce wire format, the counter-overflow guard,
and O(chunk-size) memory under a 128 MiB stream (93% coverage).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
@Peeja Peeja force-pushed the fil-470-fee-go-chunked-aes-256-gcm-stream-encryptdecrypt branch from 7541546 to 83a1ff8 Compare June 23, 2026 19:56
@Peeja Peeja marked this pull request as draft June 23, 2026 19:59
@Peeja Peeja requested a review from Copilot June 23, 2026 21:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the fee/aesstream package implementing FilOne’s chunked AES-256-GCM STREAM body cipher (Writer/Reader + one-shot helpers) as the foundation for the higher-level FEE envelope work.

Changes:

  • Introduces core streaming encryption/decryption types (Writer, Reader) and the shared Config/nonce construction.
  • Adds convenience helpers (Seal, Open, EncryptedSize, NewBaseNonce) for one-shot usage and sizing.
  • Adds comprehensive black-box and internal white-box tests covering boundary cases, tampering/truncation, and allocation/memory behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
fee/aesstream/aesstream.go Defines the AES-GCM STREAM format, config/validation, nonce layout, helpers (Seal/Open/EncryptedSize/NewBaseNonce), and exported errors/constants.
fee/aesstream/writer.go Implements chunked streaming encryption and ciphertext writing.
fee/aesstream/reader.go Implements chunked streaming decryption with truncation/tamper/trailing-data detection.
fee/aesstream/aesstream_test.go Black-box tests for round-trips, boundary conditions, tamper/truncation/reorder, sizing, and bounded-memory streaming.
fee/aesstream/internal_test.go Internal tests pinning nonce wire format, chunk-count guards, and per-chunk allocation behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread fee/aesstream/writer.go Outdated
Comment thread fee/aesstream/reader.go Outdated
Comment thread fee/aesstream/writer.go
Respond to PR review feedback on the new package:

- NewWriter/NewReader now copy cfg.AAD (append([]byte(nil), ...)) instead
  of aliasing it. AAD is authenticated on every chunk, so a caller that
  mutates or reuses the slice after construction must not change a
  stream's authentication. Key and BaseNonce were already copied; this
  closes the gap and removes a latent data-race footgun.
- writeAll returns io.ErrShortWrite on a zero-progress, no-error write
  rather than looping forever. A compliant io.Writer never does this, but
  the interface doesn't strictly forbid it, and a hang in the encrypt
  path is worth guarding against.

Tests: TestAADIsCopiedOnConstruction (mutates the caller's AAD right after
construction and confirms the stream still authenticates under the
original value) and TestWriteAllShortWriteGuard (would hang without the
fix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Peeja and others added 4 commits June 24, 2026 09:39
Fold TestRoundTripDefaultChunkSize into TestRoundTrip as t.Run subtests over a cases slice that carries each chunk size alongside its boundary-tuned plaintext sizes. The default-chunk case keeps ChunkSize 0 to exercise the 0 -> DefaultChunkSize resolution, and now also gets the EncryptedSize length cross-check (EncryptedSize resolves a 0 chunk size to the default).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Replace the hand-rolled if/t.Fatalf/t.Errorf assertions in the aesstream test files with testify require.* — house style in other fil-forge repos. testify was already an indirect dependency; go mod tidy promotes it to direct. The allocation-measurement closures in the steady-state tests keep their plain error checks so require's per-call bookkeeping cannot perturb AllocsPerRun.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Comment thread fee/aesstream/aesstream_test.go Outdated
Comment thread fee/aesstream/aesstream_test.go
Comment thread fee/aesstream/aesstream_test.go Outdated
Comment thread fee/aesstream/aesstream_test.go Outdated
Comment thread fee/aesstream/aesstream_test.go Outdated
Comment thread fee/aesstream/aesstream_test.go Outdated
claude and others added 3 commits June 24, 2026 14:59
Fold the empty-plaintext and full-size-final-chunk tests into TestRoundTrip's cases; inline the mustSeal helper (errors handled at each call site); assert io.CopyBuffer's returned byte count; add a one-byte-at-a-time decrypt pass that exercises the Reader assembling chunks from short underlying reads; and merge the five *Fails tests into one table-driven TestDecryptFailures, where each case either corrupts the ciphertext or overrides the decrypt config and all share the released-bytes-are-a-genuine-prefix invariant.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
Rename openChunk's input from last to mustBeLast (the caller's constraint: a short read means the chunk must be the final one) and return wasLast (the determined fact), so the parameter and result no longer share one overloaded name. Pass the chunk index as a parameter too, so openChunk's inputs are the per-chunk (index, ciphertext, mustBeLast) and the receiver supplies only the stream's fixed parameters. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Comment thread fee/aesstream/reader.go
Comment thread fee/aesstream/writer.go
Comment thread fee/aesstream/reader.go
Comment thread fee/aesstream/aesstream.go Outdated
Seal pre-grows the output buffer to the exact ciphertext size, but on a 32-bit platform a near-2 GiB plaintext can make the int64 to int conversion overflow negative, which would panic bytes.Buffer.Grow. Skip the (optional) pre-grow when the size does not fit in int. No behavior change on 64-bit. Raised by Copilot review.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014EgQ1SYk7fMEtwoEdppPqL
@Peeja Peeja marked this pull request as ready for review June 24, 2026 19:52
@Peeja Peeja requested review from alanshaw and frrist June 24, 2026 19:52

@frrist frrist left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to sound like a broken record across all these crypto PRs; I lack the depth of knowledge to determine is this is "correct", though it looks good to me:

Approving so this isn't blocked. But flagging, as a follow-up rather than an objection: this is hand-rolled cryptography, and the blast radius if any of it is subtly wrong is large.

@Peeja Peeja merged commit 03e30e5 into main Jun 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants