Repository navigation
New connector source linear - #5264
Merged
Merged
Conversation
Contributor
Strix Security ReviewNo security issues found. Review summaryReviewed the new Updated for Reviewed by Strix |
Adds the estuary-cdk skeleton for a Linear capture connector, plus its
authentication.
Linear is GraphQL-only over a single POST endpoint, so `api.py` carries just the
endpoint constant — streams differ by query document, not by path.
Auth is a personal API key modelled as an `ApiKey(AccessToken)` subclass behind a
`credentials_title` discriminator, so adding an OAuth arm later is not a breaking
change. Linear expects the key as a bare `Authorization: <key>` with no `Bearer`
prefix; `_token_source` sets both an empty token type and a lowercase header name
to get that on the wire without the CDK logging a missing-prefix warning on every
request.
`validate_credentials` probes `{ viewer { id } }`, which returns HTTP 401 on a bad
key, so the status code alone is a sufficient signal.
All four are Relay connections on the same endpoint taking an identical argument set, so they share one parameterised engine; per-stream identity (root field, selection set, cursor fields, sort capability) rides on the document model as ClassVars. Two API behaviours shape the design, both established by probe rather than docs: `orderBy` is descending-only and its enum members carry null descriptions, so direction was undocumented. Three of the four root fields also accept `sort`, which permits an ascending walk; `issueLabels` rejects it. Windows are therefore expressed as `(cursor, horizon]` over a fully-elapsed tick, which keeps the exclusive lower bound safe in both directions: every row sharing the boundary tick is drained before the checkpoint moves, so Linear's common bulk-stamped `updatedAt` ties cannot be split across a resume. Archiving does NOT advance `updatedAt`, so archival is invisible to an `updatedAt`-cursored sweep. `IssueFilter` is the only filter type exposing an `archivedAt` comparator, so Issues runs a second pass over that clock; both share one checkpoint, which is sound because the checkpoint is a horizon and means "both clocks swept through H". Projects, Initiatives and Labels have no such escape and cannot observe archival at all — a documented limitation, not an oversight. Backfills resume from a value watermark rather than a positional offset, so a row deleted mid-sweep renumbers nothing and cannot cause an untouched row to be skipped. Boundaries are enforced server-side. Selection sets contain only universally-available fields: in GraphQL an un-entitled field errors on every page rather than arriving absent, so `identifier` is omitted from Projects and Initiatives, where it sits behind a paid add-on. Nested connections are excluded to stay well inside the 10,000-point per-query cap; issue<->label membership belongs in its own child stream.
The Bruno collection is the evidence record behind the connector's design decisions: 21 read-only requests covering pagination, ordering direction, cursor filter boundaries, the archival capability matrix and the page-size ceiling, plus the seeding mutations used to populate an otherwise-empty test workspace. Account-wide constraints live in collection.bru; per-endpoint constraints sit on the request that proves them. The pre-request script decrypts the sops config per request and never persists the key. The README carries the limitations a user would otherwise meet as data corruption: archival of Projects, Initiatives and Labels is not captured and those records will remain in the destination indefinitely with a null archivedAt; Issues archival is captured but must be read from the tombstone rather than inferred from cursor movement; Initiatives requires a paid plan. Also documents the rate-limit model, including that Linear signals throttling as HTTP 400 rather than 429 and that HTTP 200 does not imply success.
The backfill drained its entire window inside one invocation and yielded a single watermark only after the walk completed, so a crash at any point restarted from `start_date`. Labels was worse: its watermark update was gated behind the ascending branch, so `backfill_labels` never yielded a cursor at all. Split the pager out of `_walk` so page boundaries are visible, and checkpoint at each one. The checkpoint is the last cursor value proven complete — the previous value once a different one has been seen — because ordering only guarantees a tie group is fully drained once a row outside it arrives. Checkpointing the last-seen value instead would split a tie group and drop its remainder, and Linear ties are common: a bulk edit stamps one timestamp across every row it touches. A page whose rows all share one value checkpoints nothing and the walk continues, so a tie group wider than one page cannot stall it. Resumption bounds whichever end of the window the walk consumes from, so descending streams resume correctly too rather than restarting. Verified against a simulated pager: no rows dropped and termination reached for a tie group straddling pages, a window that is entirely one tie value, 20 distinct values, a single row, and an empty window. Twenty rows at three per page now checkpoint ten times instead of once.
Cuts explanation that restated what the code already shows, or that belonged in the connector's documentation rather than beside its field lists. The facts that are not recoverable from reading the code are kept: which fields are excluded and why, and that the tombstone is the only archival signal because archiving does not advance `updatedAt`.
Adds the connector's reference page and indexes it, so the archival limitations reach the people affected by them: a Project, Initiative or Label archived in Linear stays in the destination indefinitely with a null `archivedAt`, because Linear exposes no `archivedAt` filter for those types and archiving does not advance `updatedAt`. Issues are unaffected and carry a real tombstone. Also documents the two-budget rate limit model and that Initiatives need a paid plan. The connector README now points at the reference page instead of repeating it, and keeps only development and design notes.
Registers the connector in the Python workflow's path filters and build matrix. Without this it is never built or published, so the connector cannot ship.
jonwihl
force-pushed
the
wihl/source-linear
branch
from
September 21, 2026 09:53
c7d769d to
e1c7162
Compare
|
Docs preview: https://connectors-pr-5264-estuary-docs.mike-39e.workers.dev Changed pages:
Rebuilt on every push, and the link stays the same. |
The throttle watched only the request budget, but Linear charges complexity per row returned rather than per request, so which of the two hourly budgets binds depends on the workspace: sparse records exhaust requests first, records with many populated relations exhaust complexity first while thousands of requests remain. Airbyte's Linear connector documents complexity as the usual constraint, and our own measurement was taken on a workspace whose rows have almost no relations set, so it could not have surfaced this. Exceeding either budget returns HTTP 400, which the CDK treats as terminal, so the pre-emptive pause is the only real defence and it has to cover both. Also drops the `ascending` argument threaded through the query builder and both walkers. It was always `entity.supports_sort`, so it was a parameter that could never vary independently of the entity already being passed.
Alex-Bair
requested review from
Alex-Bair
and removed request for
Alex-Bair
September 24, 2026 13:43
Applying a label stamps its `lastAppliedAt` but never advances its `updatedAt`, so an `updatedAt` cursor missed label activity and every label untouched since `start_date`. Labels are now re-read in full each interval, keyed on `/_meta/row_id`, which also captures archival and deletion. With labels gone from the incremental set, every remaining stream sorts ascending, so the descending `orderBy` walk in the query builder and the backfill is removed.
A page holding two or more distinct `updatedAt` values checkpointed its second-to-last value even when no page followed. The next invocation then re-fetched and re-emitted that page's final tie group before observing an empty window. Every backfill finished with one redundant request and a duplicate of its newest rows. `_walk_pages` now reports whether another page follows, and the backfill checkpoints only between pages. All three incremental streams share this backfill.
Contributor
Author
main moved the Python connector build matrix out of python.yaml and into .github/python-connectors.yaml (c47b34a). The conflict is resolved by taking main's python.yaml and listing source-linear in the new data file, at the position the old matrix entry held.
…rink oversized pages Review follow-ups on #5264: - `pageInfo` and `hasNextPage` are required, so a response missing them fails validation instead of ending a walk after one page. - A `RATELIMITED` rejection waits for the latest budget reset Linear announced, falling back to exponential backoff, for up to five attempts. One 60s retry against an hourly budget would fail again. - A page rejected as too complex is retried at half the size. Linear documents no distinct error for its per-query cap, so detection matches the message and is marked unverified. - Any GraphQL error now raises. No selected field is gated behind a paid add-on, so no partial error is expected, and emitting the rows that resolved would hide a resolver failure. - Issues carry `labelIds`, so they join to the `labels` collection. - The fetcher wiring map uses a frozen dataclass, the archival query asserts its clock is set, and private helpers take `log` before `http`.
- The docs description no longer says every stream is incremental. - The docs say what a backfill does and does not reconcile, how deleted issues behave, which relationships are not captured, and that the Initiatives response on an unentitled workspace is unverified. - The changelog's "Known limitations" section is not a Keep a Changelog category; its notes move to a Limitations section in the docs. - The Bruno README and MANIFEST describe the collection where it now lives, with correct request counts and no links to files outside the PR.
The repo's GraphQL guidance is to start with a minimal field set and add fields as users ask for them. Issues drop the sort orders, triage, snooze, SLA, auto-close/archive and added-to timestamps, customerTicketCount, branchName, and the delegate, milestone, snoozedBy and template relations. Projects and Initiatives drop `content`, which can be large.
Contributor
|
LGTM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
(Describe the high level scope of new or changed features)
Workflow steps:
(How does one use this feature, and how has it changed)
Notes for reviewers:
updatedAt, so Issues run a secondarchivedAt-cursored pass. Projects/Initiatives/Labels have no such filter and cannot capture archival at all — a permanent limitation to sign off on.Checklist:
CHANGELOG.mdhas been updated (see CONTRIBUTING.md)