Repository navigation
Conversation
Store entry bundles (tile/entries/* and tile/data/*) in GCS with Content-Encoding: gzip to reduce storage and egress bandwidth by ~55-60%. - Compress entry bundle payloads with gzip in gcsStorage.setObject and set w.ContentEncoding = "gzip" while keeping checkpoints and hash tiles uncompressed. - Transparently gunzip in gcsStorage.getObject when r.Attrs.ContentEncoding == "gzip" && !r.Attrs.Decompressed so both the HTTP GCS client (which auto-decompresses via decompressive transcoding) and the gRPC GCS client (which returns raw compressed bytes) always return uncompressed logical bytes to internal callers (updateEntryBundles, antispam, deduplication, migration, and 412 precondition-failed idempotency checks). - Remain fully backward-compatible with pre-existing uncompressed entry bundles in GCS.
phbnf
reviewed
Oct 7, 2026
phbnf
left a comment
phbnf
left a comment
Contributor
There was a problem hiding this comment.
Specs say that we should do this, so we should certainly do it! Maybe we need an issue to also make sure that we do this on POSIX? I can't remember what / if current CT operators have done something about this already.
However, I wonder what the CPU impact would be both for writes and reads. Should we test this? I'd like to make sure that this change won't cause unexpected CPU surges.
| // returns an error if the named object exists _and_ contains different data to what's | ||
| // passed in here. | ||
| if err := s.objStore.setObject(ctx, objName, bundleRaw, &gcs.Conditions{DoesNotExist: true}, logContType, logCacheControl); err != nil { | ||
| if err := s.objStore.setObject(ctx, objName, bundleRaw, &gcs.Conditions{DoesNotExist: true}, logContType, logCacheControl, bundleContEnc); err != nil { |
Contributor
There was a problem hiding this comment.
Might be worth popping a comment explaining that if a client doesn't set Accept-Encoding, then GCS (I think?) will decompress data.
patflynn
marked this pull request as draft
October 7, 2026 16:50
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Store entry bundles (
tile/entries/*andtile/data/*) in GCS withContent-Encoding: gzipto reduce storage and egress bandwidth by ~55-60%.Details & Safety
setEntryBundle) are compressed withgzip(Content-Encoding: gzip). Checkpoints and Merkle hash tiles (tile/<level>/*) remain uncompressed.gcs.NewClient) transparently decompressesContent-Encoding: gzipobjects (settingr.Attrs.Decompressed = trueand clearingr.Attrs.ContentEncoding).gcs.NewGRPCClient, used by TesseraCT) does not perform decompressive transcoding and returns the raw compressed bytes withr.Attrs.ContentEncoding == "gzip"andr.Attrs.Decompressed == false.gcsStorage.getObjectchecksr.Attrs.ContentEncoding == "gzip" && !r.Attrs.Decompressedand gunzips before returning, ensuring all internal callers (updateEntryBundles,antispam, deduplication, migration, and 412 precondition-failed idempotent write recovery) always receive uncompressed logical bytes regardless of client transport.Content-Encodingand continue to be read without change.Accept-Encoding: gzipreceive compressed bytes directly from GCS/CDN; clients omittingAccept-Encoding: gzipreceive decompressed bytes via GCS decompressive transcoding.