[Akshay Sonawane (apsonawane)]

Summary

Optimize private CLM/KEV request tokenization without expanding the public non-generative API.

Changes

• Add an internal Tokenizer::EncodeBatchRows adapter over ORT Extensions batch tokenization.
• Preserve ragged token rows so CLM masks, positions, and context-limit validation use each original sequence length.
• Reuse the same batch primitive in existing padded EncodeBatch overloads.
• Batch CLM structured text tokenization instead of invoking the tokenizer once per row.
• Preserve literal-tokenizer behavior and reject null batch entries explicitly.

API surface

No new public C, C++, or Python APIs are added. The runtime continues to expose:

• OgaCreateNonGenerativeSession
• OgaDestroyNonGenerativeSession
• OgaNonGenerativeSessionRunJson
• C++ and Python NonGenerativeSession

Graph capture, cache controls, action sidecars, component tensor handles, and public tokenizer APIs remain outside this change.

Validation

• ORT GenAI native library, Python extension, and unit-test targets build.
• Focused native non-generative tests pass.
• Python non-generative tests pass.
• Ruff and diff checks pass.
• Real CLM and KEV package E2E passes with ONNX Runtime 1.31.

Dependency

Stacked on #2649.

[Copilot]

🟡 Changes recommended

C++17 build failures and several cache, concurrency, portability, and validation defects remain.

15 open findings

• Double rounding allows unrepresentable size_t values · New
• C++20 contains call breaks C++17 builds · New
• C++20 unordered_map::contains breaks C++17 builds · New
• C++17 Android builds fail due to C++20 jthread APIs · New
• Ineligible fallback signatures unnecessarily serialize on package mutex · New
• Null empty text range causes undefined pointer arithmetic · New
• Hard-coded cache limits reject valid oversized sidecars · New
• Nonstandard safe_encoder paths never receive idle reaping · New
• Fused projection cache omits finite-value validation · New
• Zero-length tokenizer output can perform null pointer arithmetic · New
• Zero-length token sequence performs null pointer arithmetic · New
• Validation error names an environment variable instead of the sidecar · New
• BF16 fallback error message omits the accepted input type · New
• Developer-specific tokenizer path disables CI batch API coverage · New
• Absolute test path disables CI coverage for pinned-cache behavior · New

What changed in this PR

Optimizes typed ranking and decision sessions through shared package resources, bounded CUDA graph capture, cache improvements, and batched tokenization.

Changes:

• Adds package-level runtime policies and multi-shape CUDA graph admission.
• Introduces pinned/cost-aware caches, shared fallback encoders, and zero-copy feed views.
• Adds batched directory tokenization and validation coverage.

File	Description
test/​python/​test_non_generative_runtime.py	Tests pinned cache entries.
test/​python/​test_component_session_native.py	Tests runtime policies and batch tokenization.
src/​python/​python.cpp	Exposes batch tokenization to Python.
src/​ort_genai.h	Adds the C++ batch-tokenization wrapper.
src/​ort_genai_c.h	Declares batch-tokenization C APIs.
src/​models/​preprocessing/​genai_tokenizer.h	Declares row-wise batch encoding.
src/​models/​preprocessing/​genai_tokenizer.cpp	Implements batched tokenizer execution.
src/​models/​non_generative_session.cpp	Implements cache, fallback, validation, and feed optimizations.
src/​models/​component_session.h	Retains shared package resources.
src/​models/​component_session.cpp	Implements shared resources and bounded graph capture.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Akshay Sonawane (apsonawane)]

Addressed the review findings in f3aa5662:

• reject unrepresentable size_t JSON values using the exclusive 2^N bound
• replace C++20 contains, jthread, stop_token, and stop-aware waits with C++17-compatible code
• hold package readiness only while a new CUDA graph signature remains capture-eligible
• auto-expand initial cache capacity to hold immutable package-pinned entries
• start the fallback reaper lazily after any manifest-declared safe_encoder is acquired, independent of its filename
• validate fused state projections before caching
• handle zero-length tokenizer rows without null pointer arithmetic
• report clm_fallback_idle_ms.txt in parser errors
• include BF16 in the selector type diagnostic
• use the opt-in package root for the pinned-cache integration test

The public directory-tokenizer batch C/C++/Python API was removed in 542e859f, so the null-range/public-wrapper and developer-specific tokenizer-test findings are obsolete. Batch tokenization remains internal and uses the existing ORT Extensions OrtxTokenize primitive.

Validation: full Release build passed; 24 focused Python tests passed with 3 opt-in/provider skips; 9 focused native tests passed with 2 opt-in real-package skips.

[Copilot]

🟡 Changes recommended

Graph fallback, cache API compatibility, validation atomicity, and test-target linkage issues remain unresolved.

3 open findings

• Restore generic session after oversized capture rejection · New
• Preserve zero-capacity cache behavior with pinned projections · New
• Avoid linking internal tokenizer from public API unit tests · New

15 resolved since last review

• C++17 Android builds fail due to C++20 jthread APIs
• C++20 unordered_map::contains breaks C++17 builds
• C++20 contains call breaks C++17 builds
• Double rounding allows unrepresentable size_t values
• Zero-length token sequence performs null pointer arithmetic
• Zero-length tokenizer output can perform null pointer arithmetic
• Fused projection cache omits finite-value validation
• Nonstandard safe_encoder paths never receive idle reaping
• Hard-coded cache limits reject valid oversized sidecars
• Null empty text range causes undefined pointer arithmetic
• Ineligible fallback signatures unnecessarily serialize on package mutex
• Absolute test path disables CI coverage for pinned-cache behavior
• Developer-specific tokenizer path disables CI batch API coverage
• BF16 fallback error message omits the accepted input type
• Validation error names an environment variable instead of the sidecar

Previously missed (2)

In code that hasn't changed since last review

Validate KEV probabilities for finite and range constraints

src/​models/​non_generative_session.cpp:839

The new finite/range validation is only called by the ranking path. NativeDecisionSession::Decide still accepts NaN, infinity, or out-of-range values from either KEV probability output and returns them through Answer, so the stated finite-value hardening is incomplete. Apply this validation to the assembled KEV probability vector after its count check as well.

Validate sidecar completely before pinning cache records

src/​models/​non_generative_session.cpp:1085

Records are pinned before the complete sidecar is validated. During InvalidateCache, a malformed later record or trailing byte throws after earlier records have already repopulated the cache, leaving the live session with projections from a file that was rejected. Parse and validate all records into temporary storage first, then pin them only after the trailing-data check succeeds.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🟡 Changes recommended

Runtime policy defaults, sidecar atomicity, shared encoder release, and KEV value validation have unresolved correctness issues.

2 open findings

• Absent CUDA graph settings disable capture and override defaults · New
• Invalid sidecar leaves previously pinned records accessible · New

3 resolved since last review

• Restore generic session after oversized capture rejection
• Avoid linking internal tokenizer from public API unit tests
• Preserve zero-capacity cache behavior with pinned projections

Previously missed (2)

In code that hasn't changed since last review

Session timeout fails to release shared encoder reference

src/​models/​non_generative_session.cpp:978

With two ranking sessions sharing this encoder, each session holds one strong reference, so both reapers observe use_count() == 2 and neither ever releases its reference after the idle timeout. Reset this session's reference when its own timeout expires; any in-flight operation or other session already owns a separate shared_ptr and will keep the encoder alive.

Decision outputs lack finite and range validation

src/​models/​non_generative_session.cpp:2025

The decision path still returns pointer outputs without the finite/range validation added for ranking. A KEV head that emits NaN, infinity, or an out-of-range value reaches Answer, where comparisons and confidence calculations can produce an invalid typed result. Validate probabilities after either head layout has been flattened.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🟡 Changes recommended

Fused ranking bypasses safety fallback, and decision probabilities still permit non-finite or invalid values.

1 open finding

• Add CPU coverage for malformed sidecar package handling · New

2 resolved since last review

• Invalid sidecar leaves previously pinned records accessible
• Absent CUDA graph settings disable capture and override defaults

Previously missed (2)

In code that hasn't changed since last review

Validate KEV probabilities before constructing answers

src/​models/​non_generative_session.cpp:824

The new probability validation is only used by ranking. Both KEV head branches still pass model-produced values directly into Answer, so NaN, infinity, or values outside [0, 1] can propagate into the typed decision result and corrupt choice/confidence calculations. Apply RequireProbabilities to the extracted KEV probabilities before constructing answers as well.

Fall back to safe encoder on non-finite fused output

src/​models/​non_generative_session.cpp:1282

The fused fast path bypasses the new safety-encoder fallback. If the fused BF16 graph returns non-finite state projections (or probabilities), this check throws and encode_state, which is the only path that calls EnsureSafeEncoder(), is never reached. This is especially relevant when precomputed actions enable the fused path and the primary encoder was intentionally omitted at construction. Treat non-finite fused output as a failed fast path and continue through the non-fused safe-encoder path instead of returning an error.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Copilot]

🔵 Needs a closer look

Shared fallback lifetime, package identity, and output/sidecar validation still have correctness issues.

0 open findings

1 resolved since last review

• Add CPU coverage for malformed sidecar package handling

Previously missed (3)

In code that hasn't changed since last review

Fingerprint manifest-resolved component files

src/​models/​non_generative_session.cpp:57

This identity still fingerprints only hard-coded component paths, while component_manifest.json supports arbitrary filenames. The new process-wide fallback registry is keyed by this value, so replacing a manifest-mapped safe_encoder such as safe.onnx leaves the key unchanged; another live ranking session can keep returning the old encoder even after invalidation. Derive the fingerprint from the manifest's resolved component files (including external data) rather than this fixed list.

This issue also appears on line 1000 of the same file.

Idle reaper cannot release shared encoder references

src/​models/​non_generative_session.cpp:978

When two ranking sessions share this encoder, each session holds a shared_ptr, so use_count() remains at least 2 even after both are idle. Neither reaper can then release its reference, and the process retains the large fallback session until a ranking session is destroyed. The local pointer returned by EnsureSafeEncoder() already keeps an in-flight call alive, so reset this session's reference after its own idle timeout without requiring global uniqueness.

This issue also appears on line 1058 of the same file.

Validate KEV pointer outputs for finite [0,1] range

src/​models/​non_generative_session.cpp:2029

The new finite/range validation is applied to CLM outputs but not to either KEV pointer-head path. A pointer model returning NaN, infinity, or a value outside [0, 1] therefore reaches Round4 and is exposed in the typed decision result. Validate the assembled KEV vector before constructing answers.

🧠 Review effort: Balanced

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[Baiju Meswani (baijumeswani)]

This looks good for approval. The change is small, reuses the existing tokenizer, and does not add public APIs. Keeping the original row lengths before padding preserves the masks and context checks. Input and output ownership also looks sound.

The two inline comments are non-blocking follow-ups: make sure the tokenizer parity tests actually run, and measure the performance gain without assuming batch tokenization is parallel.

All 67 checks passed. CPU CI reported 3,110 Python passes, including all six native component tests. The literal-tokenizer tests and real-package integration tests were skipped. I reviewed the pinned ORT Extensions implementation but did not run fresh model inference locally.

[Baiju Meswani (baijumeswani)]

Approved. The internal batch-tokenization refactor is focused, preserves ragged row handling and existing tokenizer modes, and adds no public API. The testing and performance suggestions in my COMMENT review are non-blocking follow-ups.