Scope finalization for JEV integration

[onlyarnav]

I want to lock down scope before starting, since it changes what actually gets
built:

1. Scope: should the first PR land as a general tools/jev/ adapter
   usable by any skill, or scoped narrowly to pr-management-triage first,
   with wider adoption gated on eval results (see below)? I'd lean toward
   the narrow pilot so we're not maintaining a general-purpose tool before
   a single skill has proven it out.

2. RFC: does this need a new RFC given it touches vendor neutrality,
   or is a tool adapter + per-skill PR enough, treated like any other
   opt-in tools/<system>/ adapter? My inclination is the latter, provided
   the adapter is fully optional and every skill degrades to its current
   behavior with it disabled — but I'd like a maintainer/PMC read on this
   before investing in code.

3. Bar for adoption: assuming (1) is the pilot route, what would
   count as a clear yes/no signal? I'm planning to run a held-out sample of
   already-labeled PRs from pr-management-triage's history through Jev
   and report agreement rate, p50/p95 latency, and cost per 100 calls back
   here. Flag if there's a different metric or sample size you'd want to
   see before considering wider rollout.

In the meantime I'll start on the adapter itself (dependency-free HTTP
client, pinned model version, fail-open on any error/timeout/low
confidence) since that part is needed regardless of how (1)/(2) land — I'll
hold off on wiring it into any skill until there's agreement here.

[potiuk]

1. Both.
2. No need
3. Good idea. Let's see it initially and compare.

[potiuk]

I also think that pr-managemen-triage is a very good first point, because vast majority of tasks there are basically classification of certain "text" (PR) to predefined set of "what kind of PR it is".

The next one - and I might do it as a follow-up - is security family - it also has this clear classification: Valid/Invalid/Defense-in-Depth - but to test this one you need to have a big body of security issues that you can run it against.

Which I actually have in Airflow :). But it's private.

[potiuk]

Also - I think we should make it a generic contract - for those three types of output that Jev provides: Choice, Score and Noul.

Undoubtedly there will be a flood of similar services in coming weeks - because the promise of having faster, cheaper and more reliable classification than LLMs for specialized tasks is far too attractive to pass it on (including frontier models). We will likely have to have support for things like calibration of confidence and similar - and those should be generic-enough to apply it to other solutions.

This very nicely fits in our two principles:

• affordability
• vendor neutrality