[Akshay Sonawane (apsonawane)]

Summary

Add production-oriented non-generative inference for multi-component CLM ranking and KEV decision packages without routing them through the autoregressive Generator API.

The public surface is intentionally small: one package-selected session, configured through the existing OgaConfig, with a JSON request/response boundary.

Public API

The C ABI adds three exports:

OgaCreateNonGenerativeSession(
    const OgaConfig* config,
    OgaNonGenerativeSession** session);
OgaDestroyNonGenerativeSession(OgaNonGenerativeSession* session);
OgaNonGenerativeSessionRunJson(
    OgaNonGenerativeSession* session,
    const char* request,
    size_t request_length,
    const char** response);

The C++ API provides one move-only RAII wrapper:

NonGenerativeSession session{config};
std::string response = session.Run(request_json);

Python follows the same model:

config = onnxruntime_genai.Config(package_path)
session = onnxruntime_genai.NonGenerativeSession(config)
response = session.run(request)

OgaConfig remains the source of truth for the package path, providers, provider options, hardware selection, overlays, session options, tokenizer settings, pad token, and context length. The session copies the effective configuration during construction.

Supported workflows

CLM free ranking

Request:

{
  "state": {},
  "instructions": "Choose the best candidate",
  "temperature": 1.0,
  "candidates": {
    "candidate-a": {},
    "candidate-b": {}
  }
}

Result:

{
  "model": "clm",
  "ranked": [
    {"rank": 1, "key": "candidate-a", "probability": 0.75},
    {"rank": 2, "key": "candidate-b", "probability": 0.25}
  ]
}

KEV typed decisions

Request:

{
  "state": {},
  "questions": {
    "question-id": {
      "type": "noul",
      "instructions": "Should this action be taken?",
      "criteria": {}
    }
  }
}

Result:

{
  "model": "kev",
  "answers": {
    "question-id": {
      "type": "noul",
      "noul": 0.75,
      "confidence": 0.5
    }
  }
}

The package contents select the private ranking or decision implementation. No public component-session, recursive-value, cache-control, prefix-control, benchmark, or model-specific handle families are introduced.

Runtime architecture

• Loads and validates a component manifest once into an immutable package record.
• Supports Mobius and Model Builder CLM/KEV component layouts.
• Reuses existing ORT GenAI Config, Tokenizer, Tensor, NamedTensors, provider, and session-option infrastructure.
• Runs one private ORT session per named component.
• Applies normal thread defaults, provider options, path resolution, and custom-op registration to every component session.
• Keeps CLM projection caches and KEV token/prefix caches private, bounded, and session-local.
• Serializes execution and cache mutation per session.
• Disables CUDA graph capture for component sessions until cross-component capture is qualified.

Reliability and portability

• Enforces configured context limits without silently truncating requests.
• Prevents fused CLM buckets from exceeding model capacity.
• Uses stable double-precision softmax for extreme valid temperatures.
• Preserves insertion order and integer-versus-floating-point JSON semantics.
• Rejects duplicate keys, excessive nesting, malformed Unicode, embedded NULs, and invalid schemas.
• Supports escaped Unicode manifest filenames and UTF-8 paths on Windows.
• Confines component files and package-relative configuration paths.
• Reuses the existing custom-op library allow-list and registration logic.
• Requires ONNX Runtime 1.30 or newer for KEV packages using com.microsoft::LinearAttentionGate.

Validation

• Full native suite: 328 tests passed
• Focused Python suite: 8 passed, 2 opt-in integration tests skipped
• Ruff and diff checks passed
• Real CLM and KEV package E2E passed with ONNX Runtime 1.31
• Shared-library inspection confirms only the three intended non-generative C exports
• All current review threads addressed and resolved

Notes

• Packages must contain genai_config.json.
• component_manifest.json is authoritative when present; exact released CLM/KEV layouts remain supported for compatibility.
• Managed-language wrappers are delivered by dependent binding changes.
• No model weights or generated packages are included.

[Copilot]

Copilot review overview

🟡 Changes recommended

The legacy component loader can follow symlinks outside the package root, bypassing package path containment.

Review effort: Balanced
Findings: 1

Open (1)

• Enforce canonical containment for symlinked legacy package components · New

Resolved since last review (2)

• Preserve KEV hidden-state rank and row-local indices
• Build KEV position IDs using component metadata and MRoPE axes

[Copilot]

Copilot review overview

🔵 Needs a closer look

UTF-8 package and manifest paths are not converted safely on Windows.

Review effort: Balanced
Findings: None

Resolved since last review (1)

• Enforce canonical containment for symlinked legacy package components

Previously missed (2)

In code that hasn't changed since last review

Use UTF-8 conversion for manifest component filenames

src/​models/​component_session.cpp:90

Manifest filenames are JSON UTF-8 strings, but this narrow std::filesystem::path construction uses the platform-native narrow encoding on Windows. A valid non-ASCII component filename can therefore fail validation even though the later fs::path(entry.filename) conversion is UTF-8-aware; use an explicit UTF-8 filesystem conversion for this safety check.

Convert UTF-8 package paths before filesystem operations

src/​models/​non_generative_session.cpp:36

package_path is the UTF-8 path accepted by the public API, but constructing std::filesystem::path directly from its narrow string is not Unicode-safe on Windows and can make session creation fail for non-ASCII package directories. The repository’s established conversion uses the platform-correct representation (src/models/model_package.cpp:19-23); convert from UTF-8 explicitly before canonicalizing.

[Baiju Meswani (baijumeswani)]

Thanks for simplifying this. I reviewed this revision with two independent reviewers. The API is much smaller now, and the main layering concern is addressed: the new Python binding calls the public C++ wrapper, which calls the C API. It does not call private ranking, decision, or component classes. The older private ORT header include in python.cpp was already present before this PR.

Reusing Config, Tokenizer, Tensor, NamedTensors, and the existing error and string ownership rules is the right direction. We did not find a new ownership problem at the binding boundary. I do not think another broad redesign is needed.

I suggest fixing two things before merging: component sessions should honor the existing configuration rules, and CLM should honor its configured context limit. The inline comments explain both and suggest small fixes. The other comments cover API clarity, documentation/tests, and a smaller internal cleanup; those are not equivalent runtime blockers.

Header, adapter, and public-only binding-fragment compilation checks passed across the reviews. We did not run fresh native model inference for this revision, and CI was still pending. This review does not establish model accuracy, CUDA correctness, or performance.

[Copilot]

🟡 Changes recommended

The missing tokenizer fixture causes the newly registered Python tests and C++ tokenizer tests to fail on a clean checkout.

1 open issue

• Add missing tokenizer fixture at shared test-model location · New

14 resolved since last review

• Limit JSON nesting depth to prevent stack exhaustion
• Silently truncates CLM tokens at 2048
• Keep logits in double precision for stable softmax
• Prevent cache buckets from exceeding context length
• Use a hash table for duplicate JSON key detection
• Use locale-independent conversion for JSON numbers
• Register custom operator libraries for component sessions
• Resolve component ORT paths against the package
• Apply the default limit of 16 intra-op threads
• Support Unicode escapes in the shared JSON parser
• New runtime tests are omitted from CI test driver
• Notation selection differs from Python at magnitude boundaries
• Locale-dependent parsing rejects formatted floating-point values
• Uses incorrect UTF-8 path conversion on Windows

_{Review effort: Balanced}