feat(http): add authenticated Git hosting and collaboration - #150
Merged
Conversation
forhappy
force-pushed
the
codex/http-server-product
branch
2 times, most recently
from
September 6, 2026 02:47
f2e33f9 to
661bb6b
Compare
forhappy
changed the base branch from
codex/remote-git-http-qualification
to
main
September 6, 2026 02:49
forhappy
marked this pull request as ready for review
September 6, 2026 02:52
forhappy
force-pushed
the
codex/http-server-product
branch
from
September 6, 2026 06:32
01c48af to
4b87472
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build an object-storage-backed repository application from the remote-Git examples: one Rust HTTP server embeds a React frontend and serves authenticated browsing, native Git fetch/push, issues, pull requests, releases, and commit-bound review decisions. The server reads Crab repositories directly and does not invoke Git, clone repositories, or maintain a local Git object database. Native receive uses private temporary pack files and shared publication machinery.
Rebased onto
main. Draft: this is progress toward the complete self-hosted GitHub replacement, not a production-readiness claim.Latest changes
crab-http-serveras the product composition boundary and matchescrab-write’s production use ofcrab-remote-git. The local architecture gate passes all checks (103 direct workspace dependency edges).Behavior and ownership
--initializemode, using the same canonical initializer ascrab init.default_branchselects the unborn HEAD and defaults tomain. Repeated or concurrent initialization adopts the validated canonical roots; a nonempty prefix without the canonical layout fails closed without conversion or overwrite. The bucket must already exist, and the normal server path never mutates repository roots implicitly.crates/crab-http-serverowns the configured bucket catalog, HTTP policy, OIDC sign-in, repository membership, session-bound Git tokens, issues, pull requests, releases, repository labels, commit statuses, detailed check runs and embedded assets.packages/repositoryowns the React application, using Primer primitives/components and Pierre Trees/Diffs. Node is needed only to build the assets.--healthcheckmode, and drains onSIGTERM. Configuration, identity secrets and object-storage credentials remain operator-mounted inputs..git, matching GitHub and allowing native Git LFS endpoint discovery without client-specific configuration. Git LFS Basic transfers provide authenticated batch, upload and download endpoints backed directly by object storage; payload size and SHA-256 are verified, request bodies and multipart work are bounded, disconnected uploads drain before capacity is released, and locking reports the protocol-defined unsupported response. Browser reads reopen journal-aware snapshots; pending committed work runs under a generation-owner lease and both GC fence domains before becoming readable.receive-packaccepts atomic branch/tag creation, fast-forward updates and deletions while preserving submitted OIDs. Non-fast-forward batches and deletion of the current symbolic HEAD are rejected atomically. Other branches and tags remain deletable; policy follows the recorded HEAD rather than a fixed branch name. Exact configured protected branches reject direct changes after initialization, including an entire atomic batch, while pull request merge retains the canonical publication path. Protection rules can require 0–20 approvals and up to 50 required status contexts. Only each reviewer's latest decision on the exact current head counts, a current change request blocks admission, and every required context must have a latest successful commit status or check run on that same head. Intake has a five-minute cooperative deadline and bounded wire, object, inflation, graph and dependency work. Four transfer slots are shared with fetch; disconnected receive workers drain before releasing capacity or closing the runtime.crab-gitowns receive framing, private pack quarantine/preparation and exact graph/ref/visibility validation.crab-readproves Crab/LFS pointer dependencies from a pinned snapshot.crab-write, metadata and coordination own canonical journal/catalog publication, ref/namespace serialization, GC fencing and cleanup. CLI consumers share these engines; protected-view commit translation remains independently owned.git ls-remote --symref. Git tokens remain limited to explicit read or write scope. Writers can explicitly generate read/write tokens for one repository. Read tokens retain read scope even for a writer. Repository-scoped write tokens can reportpending,success,failureanderrorcommit statuses through the canonical GitHub-style route; readers can inspect the combined latest state. Status contexts are case-insensitive, retries are immutable and ordered, and stale retries cannot replace newer results. Detailed check runs add queued/in-progress/completed lifecycle, bounded Markdown output, plain-text step logs, file annotations and external detail links; the newest status or run for a required context controls merge admission. Authorization is rechecked before intake and publication. The catalog exposes effective access; changing selection or permission clears the displayed secret. Revocation invalidates subsequent requests.owner/name, following GitHub’s archive confirmation flow. The versioned lifecycle object rejects stale updates and reloads across servers. Archived repositories remain browsable and searchable while repository API mutations, native pushes and Git LFS uploads return read-only errors; native Git and LFS recheck immediately before publication. The React application shows a global Archived label and warning, hides code and collaboration write controls, and restores them after unarchive.c9b4190resolves those tips before checking their prior closure.e86129eadds the HTTP composition, token UI and qualification.The added Rust dependency edges reuse pinned workspace packages;
Cargo.lockadds only dependency references, with no external version changes. The implementation does not add a second publication format. Browser-created blob, tree and commit objects are packed in process, then pass through the same quarantine, graph validation, ref lease, GC fences, dependency proof, visibility evidence and journal publication path as native push.Verification
cmd/kubelet/kubelet.goblob directly from the existing RustFS repository without a checkout: two fresh runtimes returned HTTP 200 in 32.12 and 32.70 seconds, with 18 ranges covering all 39 lines at generation 10. The accepted 2,048-commit wave recorded 125,471 logical objects, 103,768 storage requests and 164,389,032 fetched bytes. An 8,192-commit candidate fetched 412,666,490 bytes and was slower, so it was rejected. The same endpoint is fast once the immutable blame cache is warm. This reproduces the previously failing UI request and proves the deep-history path is active in the binary now serving port 8787.crab-http-serverbinary with unbornrefs/heads/trunkin 0.84 seconds; the same command adopted it in 0.34 seconds. The resulting server passed readiness, advertised the exact unborn branch, accepted the first native smart-HTTP push in 0.15 seconds, served the exact 150-byte Markdown blob through the API, and produced a byte-identical independent depth-one clone in 0.07 seconds at commit341f2d14f8b2548e9a42a9cb61bcbc556101b2a0. The source repository path was never provided to the server. Three shared initializer tests, 30 existing CLI initialization tests, the complete 21-testcrab-writesuite, 57 HTTP-server tests, strict all-target Clippy, formatting, CLI check and release build pass.git ls-remote --symreftest fails before the fix with an all-zero HEAD, then passes for empty and tag-only advertisements. All 126 remote-helper unit tests and seven optionalgix-transportcontract tests pass. The typed formatter applies the same resolved-target rule; shared metadata, HTTP APIs and both protocol-v2 implementations retain unborn HEAD. Live RustFS checks through the rebuilt helper verify exact annotated-tag/peeled OIDs over protocols 0 and 2, with an explicit unborn-main packet in v2 and no zero HEAD. This matches Git’s helper-list parser and protocol-v2 unborn contract. Formatting and the CLI correctness/suspicious Clippy gate pass withgix-transportenabled. An additional strict all-warning lint attempt fails on eight existing VFS warnings in files unchanged fromorigin/main; no lint suppression or VFS changes were added.-D warnings, formatting, frontend TypeScript/build and the Rust release build pass. Nine frontend unit tests and all 30 Chromium browser flows pass. The focused default-branch flow covers stale-state handling, confirmation, dark theme, mobile layout and axe-core. Theme changes now suppress Primer button color interpolation only until the replacement tokens have painted, preventing the transient 4.01:1 foreground/background state while retaining normal interaction transitions. The picker regression covers case-insensitive filtering, empty results, branch/tag switching, keyboard focus and long branch names at a 360px viewport. The rebuilt release server also passes real RustFS tag selection and file browsing in light/dark/mobile layouts; native controls follow the selected color scheme. Browser branch creation and deletion pass writer authorization, CSRF, canonical ref-name, protected/default-branch, exact-source/tip, duplicate, stale-state and native Git advertisement coverage. Archive coverage persists the versioned lifecycle, rejects stale unarchive, browser collaboration changes, native pushes and Git LFS uploads, keeps repository and LFS reads available, restores the same push after unarchive, and checks the GitHub-referenced Danger Zone in dark 390-pixel and axe accessibility runs. Deletion reuses the native ref lease, validation, journal publication and cache invalidation path. A signed-in Kubernetes/RustFS run createdqualification/browser-create-20260904from the visible commit in 49 ms of server handling and 422 ms from click to selected URL; the picker showed the new branch as checked without reloading repository content. Against the real Kubernetes repository, one click oncmd/dependencycheckselected and expanded the row, exposed both children, and updated the directory pane without clearing or refetching the root tree. The rebuilt release also separated the live default branch from three qualification branches, exposed exact commit and compare links, rendered the empty Tags state, and had no horizontal overflow in the 1,280-pixel dark browser. The signed-in refs page deleted one disposable qualification branch in 820 ms from confirmation click to row removal; a fresh RustFS-backed page read still omitted it and kept the default branch without a delete control. A live two-file Kubernetes/RustFS upload committed Markdown and exact binary bytes in 771 ms; the signed-in browser immediately displayed the new commit, retained the selected branch, and rendered the Markdown preview. A separate blob read matched all six binary bytes. A second signed-in run proposed an edit from that source branch in 887 ms of server publication and reached the prefilled pull-request composer 1.77 seconds after clicking. The source ref stayed unchanged; an independent protocol-v2 fetch verified proposal commitd582073d614d070f5bec5c7c78710b4ddfbc585b, parentf1b63c52755b1c27befccd5bad654a5d564d7d79, its message and all 95 Markdown bytes.git ls-remote, and an independent fetch. A fresh RustFS lifecycle prefix published a release in 141 ms, edited its title, notes and prerelease state in 1.6 ms, and returned the exact edit after restart in 2.8 ms. Deletion took 8.4 ms, the same-version recovery replay took 1.9 ms, list/detail omitted the tombstone, and native Git retained the exact tag. A new request reused that tag in 40.7 ms with a new release number; after another restart, the list returned only the replacement in 3.5 ms. The GitHub-referenced Releases/Tags/edit flow passes light, dark, 390px, axe and no-overflow browser checks.README.mdmatched the exact Git bytes. Against the canonical Kubernetes bucket, the pinned archive returned HTTP 200 with 106 ms time to first byte and streamed 4.06 MB in five seconds before a deliberate disconnect; the remote operation reported cancellation and/readyzremained healthy. The server keeps at most two decoded archive entries queued, preserves file modes and symlinks, sanitizes byte paths, and caps the encoded response at 3 GiB without creating a checkout or local Git object database.Tshortcut restores file-search focus. Search uses a debounced accessible combobox with keyboard selection, loading, empty, error and truncation states; choosing a result restores the expanded tree at that path. Against the RustFS Kubernetes snapshot, an uncachedkubelet.gosearch traversed 6,100 tree objects in 2.22 seconds, transferred 1.42 MB from RustFS, returned 50 ranked paths without blob bodies, and openedcmd/kubelet/kubelet.goby keyboard. A repeat on the same runtime completed in 45 ms with no storage requests. Batched raw first-parent history now deduplicates repeated directory tree OIDs while retaining exact request-order results. The rebuilt release resolvedcmd/kubelet/kubelet.goto5e83b9c2c24636e06158f257113f378bcf360da7, matching nativegit log --first-parent, in 11.1 seconds with 7,742 logical objects and 9,419 storage requests; the previous reader exhausted the 10,000-object budget. The 390px document remains exactly viewport-width.5e83b9c), a cold authenticated blame returned all 18 ranges in 36.1 seconds; the previous combined reachability and blame request exhausted its two-minute budget. An unreachable full SHA still returned 404 in 89 ms.cmd/kubelet/kubelet.goline attributions in 44.45 seconds (125,525 logical objects, 108,831 storage requests, 109,141,824 fetched bytes and 155,228,135 inflated bytes); all 39 owners and 18 contiguous ranges match nativegit blame --first-parent. The immutable warm result returned in 123 ms with two storage requests. The reader now uses the same bounded Myers line engine and slider heuristics as Gitoxide blame instead of an O(rows × columns) matrix whose tie-breaking misattributed four repeated lines. The prior unindexed traversal reached only 24,429 logical objects in 103 seconds before cancellation.160bd16d, the rebuilt release server returned the exact native-Git README change2eec33d9in 950 ms of cold server work and 10 ms warm, using 226 logical objects instead of the previous 444. The full remote-reader suite (175 tests), HTTP suite (55 passed, three documented ignores), strict all-target Clippy, release build, nine frontend unit tests and 28 Chromium flows pass.refs/heads/mainat160bd16d98b7f688ce4f3b5ab0c5e4c045f36233torefs/heads/qualification/browser-proposal-20260905atd582073d614d070f5bec5c7c78710b4ddfbc585bin 406.3 ms; the refs API and independent nativegit ls-remote --symref HEADreturned the exact new symbolic ref and OID. Restoringmaintook 362.9 ms and a fresh read returned the original symbolic ref and tip. Integration coverage also proves admin-only authorization, invalid input rejection, stale current-HEAD and stale target-tip conflicts, and the native/browser authority boundary.feature/policy, the next native push was rejected with the protected-branch reason, a stale version returned 409, and removing the rule allowed the exact same commit to publish and appear throughgit ls-remote. The persisted object reloads as the effective runtime state. A second repository instance then wrote version 3 while the serving instance remained at version 2; the next native push refreshed the remote policy, rejected the update, and exposed version 3 through the catalog. Integration coverage also proves invalid-rule rejection, administrator-only authorization and repository hiding for outsiders. The Chromium settings flow creates, edits and deletes a versioned rule, verifies each request body, passes dark-theme and 390px no-overflow checks, and passes axe-core.logo/logo.pngasimage/pngin 95.6 ms; Chromium decoded it at 3,600×3,493, all 302,153 bytes matched native Git, and requesting README text through the endpoint returned 415.9f96fd6. Native RustFS qualification verifies--healthchecksucceeds while repository metadata is reachable, fails after the listener stops, andSIGTERMexits cleanly.mainandtrunk. A fresh RustFS fixture confirms rejection of both single deletion and an atomic deletion/create batch, permitted non-default branch/tag deletion, exact API HEAD and independent fetch. The server suite, all-target Clippy and release build pass with this change.0b807e1946dbb3774439173544b2938c0630ea01in 541.8 ms and replayed the same merge in 3.1 ms. Its Commits tab returned only head commit14439070de6b58d66d9cafbc3c16174671e5de31in 121.4 ms, excluded the base ancestry, opened the exact immutable commit view, and had no horizontal overflow at 610px. An independent protocol-v2 fetch verified its exact first parentbe2090b49a5aee1d819d8bdabd70f6b4e614e527, second parent14439070de6b58d66d9cafbc3c16174671e5de31, messageMerge RustFS qualificationandMERGE_RESULT.mdbytes. Pull creation took 51.2 ms and the independent native fetch took about 230 ms on localhost. The integration suite also proves protected direct push rejection, fast-forward merge/replay, two-parent publication through a protected base, and a same-file conflict that returns 409 without moving the base ref.main. The merged pull and its exact one-file comparison survived source-branch deletion and server restart. Pull creation, merge and idempotent replay took 34.1/391.2/1.0 ms; post-restart detail and changes took 25.4/55.8 ms of server work on localhost. Withmainprotected, a native atomic push touching it failed in 126.7 ms and left both refs unchanged; pull request 4 published that exact head in 259.6 ms. A fresh depth-one clone read the exact commit/file in 163.5 ms, and source deletion plus restart retainedmainand the exact comparison. The authenticated two-user native-Git test now proves merge is blocked before approval, another member's current approval unlocks it, a head advance invalidates the approval, a current change request blocks again, and the same reviewer can replace that decision with a current approval. The rebuilt server loaded the existing RustFS repository with the structured protection config; catalog, merged pull, exact comparison and native refs remained readable. A fourth RustFS flow proved required checks end to end: missing and pendingci/teststatuses blocked pull request 5, a repository-token success unlocked it, and replaying the older pending request returned its original result without regressing the combined success. Merge took 297.9 ms and an independent depth-one clone read the exact merged commit/file in 723.1 ms after source removal. After restart, pull detail, combined status, exact changes and native ref discovery took 45.8/42.4/21.1/42.7 ms. Light, dark and 390px check-panel screenshots were inspected without horizontal overflow. The rebuilt server also filtered the six persisted pull records directly from object storage: a title/description query returned pulls 6 and 5 in 8.4 ms, an uppercase author query returned all six in 2.0 ms, and a miss returned an empty page in 1.5 ms. The GitHub-referenced pull-list search retained state and pagination in the URL and had no horizontal overflow in light, dark or 390px screenshots.64135b2fixes their source: helper-list serialization omits unresolved HEAD instead of giving Git a null object ID. Both original assertions remain intact and pass. Full CI for 64135b2 passed build, formatting and Clippy, then failed one unchangedcrab-stagingconcurrent prepared-write test; the affected source is identical toorigin/main. The Git LFS commitbb95348has scoped local and live RustFS proof. Full CI for bb95348 passed build, formatting and Clippy, then failed the existingsigint_is_forwarded_to_child_and_run_terminatesworkflow test after its 10-second exit deadline; this PR does not touch that test or workflow runner.The architecture guardrail now documents the HTTP server’s expanded shared-crate composition and the production
crab-write→crab-remote-gitedge. The local gate passes with the updated policy; hosted CI should rerun from this commit.Remaining requirements
Wildcard rules and rulesets beyond exact branch protection, protected-view/active-active publication coexistence, durable push receipts and abrupt process-crash qualification remain unfinished. Git LFS HTTP locking and resumable range downloads also remain pending; interrupted Basic downloads currently restart. On a lost response or 503, inspect remote refs before retrying: a journal commit may already exist. Later indexing or cleanup failures never become an invented per-ref rejection. Missing visibility reconstruction and index receipts also need completion.
Team membership administration, multi-instance session/identity and publication admission, and production backup/recovery remain part of the full objective. The merge-base search selects one closest common ancestor and does not yet synthesize recursive virtual merge bases for pathological criss-cross histories. Repositories currently require initialization before serving. Tag-only initialization now preserves an unborn branch through shared manifest/journal validation, readers, CLI/protected candidate publication, HTTP APIs and the browser. Deploy these updated components together: tagged v1.0.1 and v1.1.0 readers reject this newly supported state. Existing resolved-HEAD repositories need no migration. Sessions/tokens are in memory; restart revokes them, membership changes require restart and provider revocation is not immediately propagated. Browser pointer views and raw downloads expose exact Git pointer blobs; native Git LFS clients hydrate through the new transfer endpoints. Manual assistive-technology and complete keyboard accessibility audits, plus external identity-provider qualification, remain pending.
The OIDC dependency includes RUSTSEC-2023-0071 (
rsaprivate-key timing); this server verifies signatures and holds no RSA private key. No dependency overrides or advisory suppressions were introduced.