euphemia

An RL environment for the decision a voice agent actually makes.

In a cascaded STT → LLM → TTS stack, the language model's action space is not "what text". It is whether to speak, when, and what — and today's models are trained on turn-complete chat transcripts, so they assume the turn is always theirs. That assumption is behind every "the AI talked over me" and every awkward second of dead air.

euphemia makes that decision explicit, gives it physics, and scores it.

python3 -m euphemia.cli --n 60 baselines     # no API key, no GPU, ~1s
python3 -m euphemia.cli --n 60 sweep         # the frontier a fixed threshold is stuck on
python3 -m euphemia.cli --n 60 ab-prosody    # what the acoustic channel is worth
python3 -m euphemia.cli --n 12 tapes         # render through real TTS + Whisper

---

Status, data & honest findings

• Runs out of the box in synthetic mode — no data, no API key, no GPU.
• The real results need real data. The most important finding is that a policy trained in the synthetic simulator does not transfer to real speech. The defensible numbers (silence-timer vs word-reading endpointer, AUROC 0.44 → 0.77, cross-validated; a GRPO policy that beats a fixed VAD on the held-out latency/interruption frontier) are all trained on real telephone calls, not in the simulator. See BENCHMARK.md.
• The simulator is a mechanism illustrator and a latency-budget model, not a substitute for real data. Treat it accordingly.

Data (bring your own)

The empirical pause distributions are derived from Switchboard (LDC-licensed), so they are not shipped with this repo. The environment detects their absence and falls back to hand-specified ("synthetic") distributions automatically. To use the real distributions, regenerate them from your own Switchboard copy:

python scripts/build_switchboard_gaps.py --swb /path/to/swb_ms98_transcriptions

See euphemia/data/README.md. Do not commit derived data to a public repository.

License

Apache-2.0.

---

The central mechanic

The caller is scripted as timed segments separated by gaps. Each gap is secretly either a HESITATION or an END_OF_TURN. The agent never sees which.

If the agent takes the floor during a hesitation, the segments the caller had not yet delivered are lost — unless that persona is patient enough to repeat itself. Those segments carry the slots the call exists to collect.

So a turn-taking failure becomes a task failure, not a style deduction:

✂️  CUT-IN    "Got it, and then?"
💥 LOST       day, time, note

Silence is not a safe harbour either. Wait too long and the caller hangs up, taking everything they had left to say with them. Both directions destroy information, which is what keeps the optimum in the middle.

---

Results

90 procedurally generated scenarios (--n 90 --seed 7), 8 caller personas, 4 domains.

A silence threshold cannot tell you anything

policy	core	EOT AUROC	latency	floor	p50 gap	cut-ins	slots lost
oracle (cheats)	0.979	1.000	1.000	1.000	200ms	0.00	0.00
vad:1200	0.662	0.500	0.237	0.860	1200ms	0.50	0.18
vad:250	0.649	0.500	0.703	0.583	250ms	1.77	0.83
vad:400	0.642	0.500	0.568	0.694	400ms	1.33	0.83
vad:700	0.619	0.500	0.419	0.716	700ms	1.17	0.69
patient	0.620	0.519	0.118	1.000	1800ms	0.00	1.17
eager	0.399	0.500	0.000	0.534	0ms	2.08	0.83

Every fixed threshold sits at chance. That is structural, not a tuning failure: its belief is a function of elapsed silence, and elapsed silence carries no information about whether the caller meant to continue. Tuning moves you along the frontier; it never lifts you off it.

The oracle — handed ground truth through an explicit cheat channel — scores 0.977. The gap between 0.66 and 0.98 is the headroom.

Prosody breaks the frontier

Same policy, same thresholds. The only change is whether the observation carries acoustic cues a recogniser normally throws away.

base	prosody	core	AUROC	latency	cut-ins
700ms	off	0.647	0.500	0.428	1.08
700ms	on	0.788	0.812	0.714	0.71
1000ms	off	0.644	0.500	0.292	0.79
1000ms	on	0.769	0.806	0.565	0.42

Latency and cut-ins improve together — the trade a fixed threshold cannot make. A cascaded stack that discards audio at the ASR boundary is throwing away the evidence that would let it stop trading.

But you cannot synthesise that signal

Prosody extracted from TTS-rendered audio does not discriminate:

HESITATION    (n=13)             END_OF_TURN   (n=11)
  7/13  falling + fading           5/11  falling + fading
  5/13  falling + lengthened       6/11  falling + lengthened

Near-identical distributions. A TTS engine reads every segment as a self-contained utterance and gives it a declarative terminal fall, because it has no idea whether the speaker intended to continue. The one cue that separates "thinking" from "finished" is exactly the cue synthetic speech cannot carry.

This is asserted as a test (test_tts_prosody_does_not_distinguish_...) so it cannot be quietly forgotten. The A/B above therefore runs on synthetic tags calibrated to be right ~75% of the time: it measures what the channel is worth if you can extract it, not a claim that TTS gets you there.

Getting there needs recordings of people who genuinely intended to continue or stop. That is a statement about data collection, not about modelling.

Word-count timing is wrong by 39%

duration model	mean abs. error vs real synthesis
400 ms/word (from the 150 wpm literature figure)	39% (range 0–64%)
304 ms/word (fitted to real audio)	10%
syllable-count model	14% (worse — rejected)

Measured across 20 utterances from the taskset. The constant is now empirical, and tapes remove the estimate entirely.

---

Two design choices that make it trainable

1. Commitment instead of polling

A WAIT carries a plan, not just a refusal:

{"action": "WAIT",
 "p_end_of_turn": 0.2,
 "commit_at_silence_ms": 900,
 "draft": "Sure, what day were you thinking?"}

The policy sets its own, semantically conditioned endpointing threshold — the thing a fixed VAD cannot express — and drafts the reply it will speak when that threshold is crossed. The environment then fast-forwards instead of re-polling.

This mirrors how low-latency production stacks actually work (speculative generation while the caller trails off). Measured against an identical poll-every-100ms policy: 54.8 → 12.2 model calls per episode, a 78% cut, with byte-identical behaviour (same core score, same p50 gap, same cut-in count). If the caller resumes, the draft is discarded and counted as waste.

2. Dense reward from a proper scoring rule

Every decision point records (predicted p_end_of_turn, ground truth), scored with the Brier score. Episode-level rewards are one scalar for a behaviour chosen hundreds of times per call — a thin gradient. Calibration yields a labelled example at every tick, against truth the policy could not have seen.

Omitting the probability is not an escape hatch. Every action implies a prediction: speaking asserts the turn ended, waiting asserts it did not. A policy that declines to state a belief is claiming certainty and eats the Brier cost of being wrong.

Reported alongside: AUROC (discrimination) and ECE (calibration), which come apart, and which make the numbers comparable to the VAP turn-shift literature.

---

Reward rubric

component	weight	what it resists
task_success	0.26	slot F1 — the thing cut-ins destroy
calibration	0.16	Brier on end-of-turn belief (dense)
latency	0.16	two-sided: dead air and pre-commitment
floor_respect	0.16	cut-ins and raw overlap
barge_in	0.10	half yield, half whether the reply engaged
brevity	0.08	voice replies must fit working memory
backchannel	0.04	placement vs. a human timing prior (JSD)
speakability	0.04	markdown, $1,234.50, 19:30 — rule-based

Design constraints, enforced by tests:

• No degenerate optimum. eager and patient are one-dimensional maximisers. If either ever tops the board, the rubric is broken, not the policy.
• Latency is two-sided. A 0ms gap is not ideal; it is evidence you committed before the caller stopped. Without that ramp, eager outscored every sane threshold.
• Non-applicable components are dropped, not defaulted, then weights renormalise. Free 1.0s let easy scenarios inflate the mean.

---

Curriculum

Tiers are ordered by measured degradation, not intuition — a test asserts it. (The first draft labelled rapid_fire "hard"; its 200ms pauses turn out to be the easiest in the set.)

tier	personas	why
easy	plain, rapid_fire, interrupter	pauses too short to trip any threshold
medium	rambler, elderly, mumbler	pauses fool a threshold, caller is patient
hard	busy_thinker, corrector	ambiguous pauses and an unforgiving caller

Monotonicity is asserted only for responsive policies. One that waits 1.2s never interrupts anyone, so the impatience defining the hard tier never bites and the gradient legitimately flattens.

---

Real speech

Optional, CPU-only, no GPU:

role	model	size
TTS	Piper / Kokoro-82M / macOS say	~60MB / 82M / 0
STT	faster-whisper tiny.en (int8)	~75MB
VAD	Silero	~2MB
turn detection	Pipecat smart-turn v3	8MB, ~12ms CPU

uv pip install 'euphemia[audio]'
python3 -m euphemia.cli --n 12 tapes
python3 -m euphemia.cli --n 12 eval --model ... --tapes tapes --prosody

Tapes solve the speed problem. Nothing about the caller depends on the policy, so synthesis, recognition and prosody extraction are measured once and cached. Later runs replay real durations and real ASR errors in 0.1ms per episode — the same amortisation offline RL uses for logged trajectories.

Real Whisper partials revise and degrade in ways no hand-written confusion table reproduces:

 400ms  'Hi, yes.'
 800ms  "Hi, yeah, I'd like..."
1600ms  "Hi, yet I'd like to book a table."       ← "yeah" → "yet"
2341ms  "I yet I'd like to book a table for six." ← "Hi" dropped

An agent that commits to an early partial commits to text that changes.

---

Training

The environment is policy-agnostic — anything implementing Policy works.

Groq cannot do gradient RL on this. Verified against their API reference: logprobs, top_logprobs, n > 1 and frequency/presence penalties are all unsupported, and LoRA is inference-only, enterprise-tier, llama-3.1-8b only, bring-your-own-adapter. Groq is excellent as the policy under evaluation, the user simulator, and the judge — not as a trainer.

For gradients, use a backend that returns logprobs (vLLM via prime-rl, or TRL GRPO). rewards.score() returns the scalar; trace.predictions gives the dense per-step signal.

---

Layout

euphemia/
  types.py       Action, Observation, EpisodeTrace  (frozen dataclasses)
  timing.py      constants, each marked measured or ASSUMPTION
  scenario.py    procedural taskset generator
  persona.py     8 caller personas
  simulator.py   the discrete-event duplex channel
  rewards.py     the rubric, Brier / AUROC / ECE
  policies.py    Policy protocol + reference baselines
  llm.py         OpenAI-compatible policy (Groq, Ollama, vLLM)
  speakable.py   TTS-hostility linter
  audio/         prosody, backends, tapes   (optional)
tests/           41 tests pinning the claims above

Honest limitations

• The prosody A/B uses synthetic tags. The real extractor's discriminative power on human speech is unmeasured — TTS provably does not provide it.
• The backchannel timing prior is hand-specified, not fitted to Switchboard. set_backchannel_prior() takes empirical counts; do that before quoting JSD as comparable to published work.
• Baseline task_success is information survival, not comprehension. The scripted baselines are given perfect listening through a cheat channel so that information loss shows up in their numbers at all; it is not comparable to a real policy's slot extraction.
• smart-turn v3 is described as the incumbent but not yet benchmarked here. Beating a fixed threshold proves little in 2026; that is the real bar.
• Callers are scripted, so the environment tests turn-taking under a fixed conversational plan, not open-ended dialogue.