History rewrite: 502 MB of deleted binaries still in every clone

[iyashjayesh]

What

Thirteen compiled example binaries are tracked in git, totalling 502 MB on disk. They are all present in HEAD, introduced by #38.

 41.8 MB  example/router-integration/gin-integration/gin-integration-example
 40.8 MB  example/router-integration/fiber-integration/fiber-integration-example
 39.1 MB  example/router-integration/echo-integration/echo-integration-example
 39.0 MB  example/security-examples/echo/echo-secured-example
 38.3 MB  example/security-examples/chi/chi-secured-example
 38.0 MB  example/function-trace-example/function-trace-example-monigo
 37.9 MB  example/monigo-example
 37.8 MB  example/security-examples/ip-whitelist-example/ip-whitelist-example
 37.8 MB  example/security-examples/basic-auth/basic-auth-example
 37.8 MB  example/security-examples/api-key/api-key-example
 37.8 MB  example/router-integration/gorilla-mux-integration/gorilla-mux-integration-example
 37.7 MB  example/security-examples/custom-auth/custom-auth-example
 37.7 MB  example/router-integration/standard-mux-integration/standard-mux-integration-example

GitHub reports the repository as 240 MB. That is what every contributor downloads to work on a library whose source is a few hundred kilobytes.

They are not useful to anyone

$ file example/monigo-example
example/monigo-example: Mach-O 64-bit executable arm64

macOS arm64. They will not run on Linux CI, on Windows, on an Intel Mac, or in any container. They are rebuildable in seconds with go build. There is no scenario in which a checked-in copy is the thing someone needs.

There is an irony worth naming: the dashboard payload was just cut from 7.9 MB to 360 KB because every byte ships in consumers' binaries. Half a gigabyte sits in the repo that every contributor pays for.

Two halves, with different costs

Half one: stop it growing — safe, do now

git rm --cached <the 13 files>
# .gitignore: ignore built example binaries

No history rewrite, nothing breaks, no clone is invalidated. But the clone stays 240 MB forever, because the blobs remain in history.

The .gitignore needs a rule that actually catches them. They have no common suffix (-example, -monigo, …-example), so the reliable approach is to ignore each example directory's build output by name, or adopt a convention (bin/) and have the examples build there.

Half two: shrink the clone — effective, and it breaks things

git filter-repo --strip-blobs-bigger-than 5M followed by a force-push takes the clone from 240 MB to roughly 15 MB (current size-pack without the binaries is 14.9 MiB).

The cost is real and should not be waved away:

• Every one of the 19 forks is invalidated. Their history diverges from the rewritten one; merging becomes painful.
• Every existing clone must git fetch --all && git reset --hard origin/main, and any open work rebased.
• Tags must be re-pushed. v1.4.1 and v1.5.0 are already published to the Go module proxy — the proxy caches by content hash, so rewriting the commits those tags point at means the proxy's copy no longer matches the repo. Existing consumers keep working from the proxy cache; this needs checking before anything is rewritten, because getting it wrong breaks go get for everyone.

Recommendation

Do half one now. It is safe, it is a five-line change, and it stops the problem compounding.

Treat half two as a scheduled event, not a cleanup. If it happens: announce it on the repo first, give fork owners a window, do it at a quiet moment, and verify the module proxy still serves v1.4.1 and v1.5.0 afterwards. 240 MB is a genuine barrier to contribution for a 410-star project, so it is worth doing — but on purpose, not as a side effect of tidying.

Also worth folding in

.gitignore currently has /bin and nothing that catches these. A convention of building examples into an ignored directory would prevent recurrence better than listing thirteen paths.

[iyashjayesh]

Correction: it is 14 files, not 13 — monigo.gif is the second-biggest

I sized every tracked file rather than trusting the earlier count. The working tree is 527.3 MB across 160 tracked files, and 526 MB of that is 14 files:

13 Mach-O arm64 binaries	502 MB
monigo.gif	24.1 MB
the other 146 tracked files	~1.3 MB

So the repo is not "240 MB with some binaries in it" — it is 14 files plus a rounding error.

monigo.gif needs different handling from the binaries. It is in use — README.md:17 embeds it — and the CHANGELOG shows it was already re-encoded once, 65 MB → 24 MB, with the reason recorded as "the module zip is downloaded on every go get". So the cost is understood and it has had one pass already. It should not simply be deleted. Options, cheapest first:

1. Re-encode again. A 24 MB GIF of a dashboard is mostly redundant frames; MP4/WebM in a <video> tag, or a heavily quantised WebP, typically lands 10–40× smaller. GitHub renders <video> in READMEs.
2. Move it out of the module. Host it as a release asset or on a branch that is not part of the module zip, and reference it by URL. Removes it from every go get without touching history.

Either way it is independent of the binary removal and carries none of the fork risk, so it can go first.

The .gitignore half needs care

The existing .gitignore covers *.exe, *.dll, *.so, *.dylib, /bin and *.test. None of those match a Go binary on macOS or Linux, which has no extension and is named after its directory — monigo-example, api-key-example, and so on. That is exactly why these 13 were never caught.

A pattern cannot reliably distinguish "compiled binary" from "source file with no extension", so the safe form is explicit paths, one per example, added in the same commit as the git rm --cached.

The new examples CI job in #77 builds with go build -o "$(mktemp -d)" specifically so it does not recreate these files.

[iyashjayesh]

Untracking is done — #81 is merged

The 13 compiled binaries are no longer tracked, and TestNoCompiledBinaryIsTracked reads the magic number of every tracked file so they cannot come back unnoticed.

Tracked working tree	527.3 MB → 25.3 MB
.git directory	503 MB — unchanged

That gap is the whole of what is left here. The files are gone from HEAD; every byte is still in history, so a fresh git clone still transfers ~500 MB.

The GIF is deferred, not forgotten

monigo.gif is 24.1 of the 25.3 MB still tracked — 95% of what remains. It is also showing the pre-rebuild dashboard: the nav in it reads "Dashboards / Function Metrics / Go Routines Stats / Reports", against the current "Overview / Functions / Goroutines / Reports / Exporters".

Re-encoding was measured and works — WebP at 5.3 MB, visually equivalent, 4.5× smaller — but it would only make a stale image load faster. New assets are planned, so this waits for them rather than being optimised twice. When they land, the encoding question is already answered: animated WebP, -q:v 40, referenced from an ordinary <img> tag.

What remains: the rewrite

Removing the objects from history means git filter-repo and a force-push. That is not a task, it is a scheduled event:

• 19 forks are invalidated. Every fork's history diverges and cannot be merged back without manual work.
• Published versions. v1.4.1, v1.5.0 and v1.6.0 are on the module proxy with recorded hashes. The proxy serves its own cached copies, so existing go get keeps working — but this needs confirming against the actual proxy behaviour before anyone runs the rewrite, not assumed.
• Every open PR and local clone needs re-cloning.

Expected result is roughly 503 MB → ~15 MB.

The prerequisite is that nothing new is being added to the pile, and that is now true. This can happen whenever an announcement is ready.