Define a minimum `CvBackend` interface (pluggable, stateless CV contract for a high-level WebAR layer)

[kalwalt]

Summary

Important

The interface draft below has been amended. Implement the merged version, not this body verbatim:

• feat(cv_backend): descriptor selection & capability declaration #128 — descriptor selection & capability declaration (DescriptorKind, DescribeOptions, Descriptors.kind/norm, BackendCapabilities)
• feat(cv_backend): optional filterMatches step (GMS seam) #129 — optional filterMatches step (GMS seam) + capabilities.matchFilters

Note

Update 2026-08-26 — contract location (from the purecv ↔ jsfeatNext design discussion).
The contract is expected to live in webarkit/webarkit, likely as a sub-package — a leaning, not a settled fact (it was raised, not finalised).

• It is NOT imported into jsfeatNext, nor into PureCV (this part is certain). The dependency runs the other way: the AR layer in webarkit/webarkit holds the contract and imports the backends (jsfeatNext, PureCV) to wrap them — the backends stay contract-agnostic CV libraries.
• Open question this raises: if neither backend imports the contract, then the CvBackend adapter (the "jsfeatNext deliverable" below) also cannot live in jsfeatNext without pulling the contract in. So the adapter likely lives in webarkit/webarkit too, next to the contract — which would mean Define a minimum CvBackend interface (pluggable, stateless CV contract for a high-level WebAR layer) #96's concrete jsfeatNext work item shrinks to making the modules adapter-ready rather than shipping the adapter. Not yet decided — flagged for a follow-up.
• Source of truth for the signatures/types (a TS definition, the Rust side, or a language-neutral spec) is not yet decided.
• No plan changes to milestone or sequencing; only the location of the interface and adapter sharpens.

Note

Update 2026-08-27 — RESOLVED. The open questions above are now decided.

• Contract location — confirmed. The contract lives in the webarkit/webarkit monorepo as the package @webarkit/cv-backend-spec (packages/cv-backend-spec), authored on branch feature-webarkit-cv-backend. It is a workspace of the @webarkit/webarkit npm-workspaces monorepo.
• Source of truth — decided: TypeScript. The interface is authored as packages/cv-backend-spec/src/cv_backend.ts and published as @webarkit/cv-backend-spec (types + a trivial JS entry). PureCV (Rust) conforms to it on the WASM side; the TS definition is canonical.
• Adapter location — decided. The jsfeatNext CvBackend adapter does NOT live in this repo. It lives in the same webarkit/webarkit monorepo, as a sibling package parallel to cv-backend-spec (e.g. packages/cv-backend-jsfeatnext). That package depends on both @webarkit/cv-backend-spec (the contract) and @webarkit/jsfeat-next (this library, contract-agnostic), and does the matrix_t/keypoint_t ↔ neutral-type mapping. This keeps jsfeatNext free of any contract dependency, exactly as required.
• Consequence for Define a minimum CvBackend interface (pluggable, stateless CV contract for a high-level WebAR layer) #96's jsfeatNext work item: it shrinks from "ship the adapter" to "keep the modules adapter-ready" — i.e. ensure the primitives (fast_corners/yape, orb, bfmatcher, motion_estimator+homography2d, pose_estimator) expose what the adapter needs (public inputs/outputs, no hidden state). The adapter itself is tracked and built in webarkit/webarkit.

Define a minimum, stateless CvBackend interface — the contract that a high-level WebAR project depends on, so that the AR orchestration is written once against the interface and the actual computer-vision backend is pluggable.

The interface exposes only stateless CV primitives:

detect · describe · match · estimateHomography · poseFromHomography

Everything stateful and AR-specific (target training, tracking loop, pose refinement, geometric validation, temporal filtering, renderer adapters) lives above this line — see the WebAR roadmap issue.

This mirrors the jsartoolkitNFT architecture (a WASM core + a TypeScript orchestration wrapper), but made abstract so the core is swappable.

Why — an "elastic", multi-backend design

The high-level AR layer stays backend-agnostic. Any implementation that satisfies CvBackend can plug in. Three candidate backends:

Backend	What it is	Status
standard (jsfeatNext)	pure TS/JS — ORB, homography and pose are all present in the library	available now
WASM (PureCV)	Rust → WASM, SIMD performance for heavy 30–60 fps mobile loads	in development
WebARKitLib (OpenCV/WASM)	the existing WebARKit C++ tracker (OpenCV → emscripten/WASM), driven by webarkit-testing	candidate — needs the same rewrite (see caveat); also prior art, see #97
jsartoolkitNFT	the existing ARToolKit NFT / KPM WASM tracker	candidate — needs a valuable rewrite to fit this contract

Honest caveat — applies to both WASM trackers (jsartoolkitNFT and WebARKitLib): their public APIs are today bundled end-to-end pipelines (WebARKitLib: processFrameData() → pose matrix; jsartoolkitNFT KPM: detect + match + pose fused), not the granular detect / describe / match / estimateHomography / poseFromHomography steps this contract needs. Exposing those individually is exactly the "valuable rewrite" — real work — which is why both are future candidates, not ready backends.

This also suggests a second, coarser seam worth considering: a whole-pipeline TrackerBackend (frame in → pose out) that an existing complete tracker like WebARKitLib could satisfy as-is, with no rewrite. The high-level layer could then either compose primitives itself via CvBackend, or delegate wholesale. Not designed here — noted as an option (see #97).

Because all backends share the same signatures (aligned with OpenCV / PureCV naming), they are interchangeable, and jsfeatNext's role as a numeric reference oracle (same input → same keypoints/descriptors/homography, to validate a Rust port) becomes a bonus of the standard backend, not its only purpose.

Boundary contract (matters for the WASM implementations)

1. Neutral types — typed arrays and plain structs, never jsfeatNext's matrix_t/keypoint_t. Each backend converts internally (jsfeatNext wraps to matrix_t; a WASM backend maps to its heap).
2. Compute methods are SYNCHRONOUS — pure CPU functions over buffers; a loaded WASM module runs them synchronously too. To offload, offload a whole pipeline step to a Worker — don't make individual primitives async and pay a boundary crossing per call.
3. Returned typed arrays are owned by the caller (copied out of any WASM heap) to avoid aliasing bugs where the next call invalidates a view. A zero-copy "borrow" fast path can be added later behind a flag.
4. Geometry is Float64 for precision (H, K, R, t); pixels/descriptors are Uint8; point lists are Float64 interleaved [x0,y0,x1,y1,…].

jsfeatNext deliverable

The interface is neutral and lives in webarkit/webarkit as @webarkit/cv-backend-spec — not imported into jsfeatNext or PureCV (see the 2026-08-27 note at the top). The concrete adapter also lives in webarkit/webarkit, as a sibling package to the spec — not in this repo. So jsfeatNext's job here is only to stay adapter-ready: keep the primitives below publicly reachable with neutral-mappable inputs/outputs. The adapter (built in webarkit/webarkit) maps the neutral types ↔ jsfeatNext's matrix_t/keypoint_t and delegates to these modules:

• detect/describe → fast_corners/yape + orb.describe
• match → bfmatcher (feat(bfmatcher): brute-force Hamming matcher and match_t #133, merged)
• estimateHomography → motion_estimator.ransac + homography2d
• poseFromHomography → pose_estimator (Add pose_estimator module to complete the natural-feature AR pipeline #83), returning (R, t) in the camera frame only (renderer matrices are a high-level adapter — see Add pose_estimator module to complete the natural-feature AR pipeline #83 correction & the roadmap)

Draft interface — cv_backend.ts (pre-amendment; see #128 and #129 above)

/**
 * Minimum CV backend interface.
 *
 * The contract the high-level AR project depends on. It exposes only the
 * stateless computer-vision primitives; everything stateful and AR-specific
 * (target training, tracking loop, pose refinement, validation, temporal
 * filtering, renderer adapters) lives above this line and is written once
 * against this interface.
 *
 * Interchangeable implementations (all behind the same signatures):
 *   - jsfeatNext (TypeScript) — "standard" backend, portable, no WASM; also a numeric oracle.
 *   - PureCV (Rust -> WASM)   — "WASM" backend, production performance.
 *   - WebARKitLib / jsartoolkitNFT — future candidates (WASM trackers); today they expose
 *     a bundled pipeline, so they need a rewrite to surface these primitives.
 *
 * Boundary contract (matters for the WASM implementations):
 *   1. Types are neutral — typed arrays and plain structs, never jsfeatNext's
 *      `matrix_t`/`keypoint_t`. Each backend converts internally.
 *   2. Compute methods are SYNCHRONOUS. They are pure CPU functions over
 *      buffers; a loaded WASM module runs them synchronously too. If you want
 *      to offload work, offload a whole pipeline step to a Worker — do not make
 *      individual primitives async and pay a boundary crossing per call.
 *   3. Returned typed arrays are OWNED BY THE CALLER (copies out of any WASM
 *      heap). This avoids aliasing bugs where the next call invalidates a view.
 *      A zero-copy "borrow" fast path can be added later behind a flag.
 *   4. Geometry is Float64 for precision (H, K, R, t); pixels/descriptors are
 *      Uint8; point lists are Float64 interleaved [x0,y0,x1,y1,...].
 */

/** Row-major 3x3 matrix, length 9. Used for H, K, R. */
export type Mat3 = Float64Array;

/** 3-vector, length 3. Used for t. */
export type Vec3 = Float64Array;

/** Interleaved 2D points [x0, y0, x1, y1, ...]; count = length / 2. */
export type PointArray = Float64Array;

/** Single-channel 8-bit image. RGBA -> gray is a preprocessing step above. */
export interface GrayImage {
    data: Uint8Array;
    width: number;
    height: number;
}

export interface Keypoint {
    x: number;
    y: number;
    score: number;
    angle: number; // radians; 0 if the detector is not oriented
    level: number; // pyramid octave the point was found at
}

/** Binary descriptors packed row-major: `count` rows of `bytesPerDescriptor`. */
export interface Descriptors {
    data: Uint8Array;
    count: number;
    bytesPerDescriptor: number; // 32 for ORB
}

/** One correspondence, equivalent to OpenCV's cv::DMatch. */
export interface Match {
    queryIdx: number;
    trainIdx: number;
    distance: number;
}

export interface HomographyResult {
    H: Mat3;
    inliers: Uint8Array; // mask over the input correspondences (1 = inlier)
    numInliers: number;
    ok: boolean; // false if estimation failed / too few inliers
}

export interface Pose {
    R: Mat3; // rotation, OpenCV camera frame (camera looks down +Z)
    t: Vec3; // translation, in the units of the model-plane coordinates
    good: boolean; // false if H/K were degenerate
}

export interface DetectOptions {
    maxKeypoints?: number;
    threshold?: number; // detector response threshold
    levels?: number; // pyramid levels to search
}

export interface MatchOptions {
    /** If set, run k=2 + Lowe ratio test internally and keep good matches. */
    ratio?: number;
    /** Mutually-best filtering; ignored when `ratio` is set. */
    crossCheck?: boolean;
    /** Drop matches whose Hamming distance exceeds this. */
    maxDistance?: number;
}

export interface RansacOptions {
    threshold?: number; // reprojection threshold in pixels
    maxIterations?: number;
    confidence?: number; // 0..1
}

/**
 * The stateless CV surface. An implementation holds no per-frame state; all
 * inputs are passed in explicitly and all outputs are owned by the caller.
 */
export interface CvBackend {
    /** Detect keypoints in a grayscale image. */
    detect(image: GrayImage, options?: DetectOptions): Keypoint[];

    /** Compute binary descriptors for the given keypoints (OpenCV: `compute`). */
    describe(image: GrayImage, keypoints: Keypoint[]): Descriptors;

    /** Convenience: detect + describe in one pass (may reuse the pyramid). */
    detectAndCompute?(
        image: GrayImage,
        options?: DetectOptions
    ): { keypoints: Keypoint[]; descriptors: Descriptors };

    /** Brute-force Hamming match of query descriptors against train. */
    match(query: Descriptors, train: Descriptors, options?: MatchOptions): Match[];

    /**
     * Estimate the planar homography mapping `src` points to `dst` points
     * (RANSAC). `src[i]` corresponds to `dst[i]`.
     */
    estimateHomography(
        src: PointArray,
        dst: PointArray,
        options?: RansacOptions
    ): HomographyResult;

    /**
     * Decompose a planar homography into a camera pose given intrinsics `K`.
     * Pure geometry — returns R, t in the camera frame; converting to a
     * renderer matrix (GL modelview/projection) is a high-level adapter, not
     * part of this contract.
     */
    poseFromHomography(H: Mat3, K: Mat3): Pose;
}

/**
 * Async factory: WASM instantiation is async, but the returned backend's
 * methods are all synchronous. A pure-TS backend resolves immediately.
 *
 *   const cv = await createJsfeatBackend();   // or createPureCvBackend()
 *   const kps = cv.detect(frame);
 */
export type CreateCvBackend = () => Promise<CvBackend>;

Design decisions (from the discussion)

• match is a single method with a ratio option (runs k=2 + Lowe ratio internally) rather than exposing knnMatch separately — smaller surface. Raw knnMatch can be added if the tracker needs it.
• detectAndCompute is optional (?) — convenient because it can reuse the pyramid, but a backend may omit it and the high-level layer falls back to detect + describe.
• poseFromHomography stops at (R, t) in the camera frame; GL modelview/projection is a high-level renderer adapter (keeps the contract renderer-agnostic).
• Pre-allocated output buffers for the 30 fps hot path (e.g. match(..., outMatches)) are deliberately left out of the first draft; add as overloads if the loop needs zero-GC.

Acceptance criteria

• CvBackend interface (+ neutral types) is agreed and documented — authored in webarkit/webarkit as @webarkit/cv-backend-spec (packages/cv-backend-spec/src/cv_backend.ts, branch feature-webarkit-cv-backend). TS is the source of truth, and the feat(cv_backend): descriptor selection & capability declaration #128/feat(cv_backend): optional filterMatches step (GMS seam) #129 amendments are folded in: DescriptorKind/DescriptorNorm/DescribeOptions, Descriptors.kind/norm, BackendCapabilities, and the optional filterMatches seam with matchFilters. 13 conformance tests pin the negotiation rules, which are prose and invisible to the type system.
• jsfeatNext side (this repo): the primitives are adapter-ready. The audit found four of five already fine; the gap was keypoint orientation — ic_angle lived only in the example pages, duplicated, so nothing in src/ ever set keypoint_t.angle and the library could not run detect → describe on its own. Fixed in feat(orb): expose ic_angle, the keypoint orientation step (#96) #182 (orb.ic_angle), shipped in 0.15.0.
• Adapter (in webarkit/webarkit, not here): @webarkit/cv-backend-jsfeatnext implements CvBackend, depending on @webarkit/cv-backend-spec + @webarkit/jsfeat-next (>= 0.15.0), mapping neutral types ↔ matrix_t/keypoint_t. 23 tests. jsfeatNext depends on nothing contract-related, as required.
• Signatures stay aligned with OpenCV / PureCV naming so backends are interchangeable.
• A trivial round-trip demo composes the primitives (detect → describe → match → [filterMatches] → estimateHomography → poseFromHomography) via the interface, backend chosen at construction. Two views of a scene differing by an exact translation: 6 keypoints per view, 6 matches, 6 inliers, translation recovered to ~1e-5 px. The match count is asserted at >= 5 deliberately — with exactly 4 correspondences a homography fits them by construction and RANSAC would report success however wrong the matching was.

Note

Update 2026-09-03 — all criteria met; deliberately left open.
Everything above is done and pushed to webarkit/webarkit PR #4 (branch feature-webarkit-cv-backend), which is not merged yet. Closing waits on that.

Two release-blocking bugs surfaced during review and were fixed before 0.15.0 shipped, both on the surface of the very modules this issue depends on:

• bfmatcher.match/knnMatch derived one row stride from query and used it to address both matrices. A mismatched train set was read at the wrong offsets and, past the end, an out-of-range Int32Array index yields undefined which XOR coerces to 0 — so nothing threw and the matcher returned confident, meaningless Hamming distances. Now rejected up front, comparing effective row bytes (cols × channel × sizeof(type)), not cols.
• match_t/IMatch_T and pose_t/IPose_T were not re-exported from the package root, and the exports map blocks deep imports — so there was no way for a consumer to name the types of two of the modules 0.15.0 exists to publish. The adapter would have hit this on its first import.

Both in #184, released in 0.15.0.

Note

Update 2026-09-03 — closing. PR #4 merged into dev (webarkit/webarkit@232b3b7).

Before merging, a two-axis review (Standards / Spec, against this issue plus #128 and #129) ran against the full PR diff and turned up six findings, all fixed and verified before merge — none dismissed:

• detect() set the shared fast_corners threshold with no restore, unlike match()'s carefully-restored cross_check beside it. Not observable by calling detect() again through the adapter (every call resets the threshold regardless), only by another consumer reading the singleton directly between two adapter calls — which is exactly what the fix's regression test does.
• Two self-contradicting docs (a stale filename in a license header, a README claiming an image "not committed yet" that the same commit committed) and one README example (detectors: ['fast', 'yape']) that no backend can honestly claim, since DetectOptions has no detector selector — contradicting the negotiation-honesty rule stated two lines above it in the same file.
• Two minor duplications (a dead second keypoint-drawing pass in the demo; an unreachable-scope test helper reimplemented instead of hoisted).

Also live now: examples/pinball-static-jsfeatnext-backend.html — a static two-image demo of the full pipeline (detect → describe → match → [filterMatches] → estimateHomography → poseFromHomography) via @webarkit/cv-backend-jsfeatnext, with the target's outline reprojected onto the scene as the visual proof every stage agreed. Comparing against the real jsartoolkitNFT/WebARKitLib engine (DoG detector + FREAK descriptor, numOctaves() applied symmetrically to both the reference and the live query frame) surfaced that our adapter's detect() originally searched scale on the target only — fixed in 109caaf/938aab0, documented as a known limitation to revisit when the webcam demo lands (tracked informally, not yet its own issue).

Closing. Follow-on work (webcam demo, GMS/filterMatches implementation, PureCV as a second backend) continues under the roadmap, not this issue.

Out of scope (here)

• Implementing the PureCV, WebARKitLib or jsartoolkitNFT adapters (separate efforts).
• Designing the coarser whole-pipeline TrackerBackend seam (noted as an option only).
• Any stateful/AR-specific orchestration — that's the high-level roadmap issue.
• Zero-copy borrow fast path and pre-allocated output overloads (future optimizations).

Related

• Prototyped in the "Libreria realtà aumentata con jsfeatNext" design discussion.
• CV primitives it delegates to: Add pose_estimator module to complete the natural-feature AR pipeline #83 (bfmatcher, pose_estimator).
• Consumed by the high-level WebAR features roadmap (companion issue).