feat(p07): agentship-voice — the seam and both framework adapters - #42
Draft
harshuljain13 wants to merge 5 commits into
Draft
feat(p07): agentship-voice — the seam and both framework adapters#42harshuljain13 wants to merge 5 commits into
harshuljain13 wants to merge 5 commits into
Conversation
…ctually want First slice of the voice package. No framework is a hard dependency, so the seam, the config surface and their tests run keyless in CI -- the same shape as agentship-observability, where the vendor SDK never becomes a dependency of the contract. The spec's shared component does not survive contact with the real 1.8 APIs. It assumed one `AgentNodeProcessor` that both frameworks could hold. They cannot, because they disagree on where the agent sits: Pipecat wants FrameProcessor.process_frame(frame, direction) -- a node in a frame graph LiveKit wants llm.LLM.chat(chat_ctx) -> LLMStream[ChatChunk] -- the LLM slot of a session Checked against pipecat-ai 1.8.1 and livekit-agents 1.8.0 installed and introspected, not against the versions the spec named in July (1.6 and 1.6.7 -- both since superseded). So the shared piece is smaller than specced: given what the human said, stream back what the agent says. That is VoiceTurn, and it is the whole vendor-free surface. Each adapter wraps it in its own idiom, which keeps the adapters thin and the agent identical to the one the REST service runs. Writing both adapters at once is what proved this -- an ABC drawn against either framework alone would have encoded that framework's assumption about where an agent belongs, and this is the one place the "wait for the second implementation" rule cannot be followed cheaply, because the disagreement IS the design input. Two decisions in VoiceTurn are about being heard rather than about plumbing. Only content is yielded: a pipeline speaks whatever it is handed, so `reasoning` (the model's scratchpad, emitted separately precisely so a client can tell it apart) and tool bookkeeping would have the agent read its own notes aloud. And chunks are yielded as they arrive rather than joined, because time-to-first-audio is what a listener experiences as latency -- TTS can start on the first clause while the model is still producing the rest. The session id threads the whole conversation, not one utterance, so a voice caller gets the same short-term memory the REST path does. A test asserts two utterances share one id, because minting per utterance yields a fluent agent that forgets the previous sentence. agentship-voice is deliberately NOT added to the six released-in-lockstep packages yet; it ships nothing until the adapters land, and `versions.py --check` still reports six. 788 passed (+6), 14 skipped, 3 xfailed. Lint clean. Tests need no framework, no audio device and no provider key -- if that ever stops being true, the seam has leaked.
Barge-in turns out to be a memory-correctness problem, not only a cancellation one, and the phase spec did not say so. The /voice playbook does: on interrupt the `[cancelled by user]` marker must be appended where AUDIO stopped, not where generation stopped. Otherwise the model believes it said sentences the human never heard, and the next turn opens with "as I mentioned..." about words that were cut off mid-phrase. That makes it a seam concern rather than an adapter detail, because only the seam holds the turn. So VoiceTurn now separates two things that are easy to conflate: generated -- everything the agent produced spoken -- only what an adapter CONFIRMED reached the speaker Confirmation has to come from the adapter: TTS buffers, so text handed over is not yet text heard, and only the framework knows when sound actually left the speaker. `transcript()` records `generated` normally and `spoken` + the marker when interrupted. Writing the test corrected my own model of it. I first assumed generation runs ahead of playback on its own; it does not, because an async generator is pull-based and produces only what the consumer asks for. The gap that matters appears when the adapter pulls chunks into the TTS buffer faster than they play -- so the test now models the buffer, which is the real shape, rather than a race that cannot happen at this seam. Latency also moves earlier than the spec had it. The playbook is explicit that latency is a first-class return value from task one and retrofitting it is a rewrite, so LatencyTrace lands with the seam rather than in 10c. `llm_ttft_ms` is stamped on the first chunk and kept separate from `llm_total_ms`: the delta between them is the entire value of streaming, and a single number would hide whether the pipeline is overlapped or merely fast. Unrun stages report None, not 0 -- zero is a measurement, and claiming one we never took is the same class of dishonesty as an engine over-declaring a capability. Spec updated with all four corrections and the completed tasks marked. NOTE, not fixed here: abandoning a stream mid-iteration makes OpenTelemetry log "Failed to detach context" -- the span was attached in a different context from the one GeneratorExit tears down in. Harmless today (logged, not raised) but it would fire on EVERY barge-in, which is the normal case for voice rather than an edge one. Filed for the adapter work. 794 passed (+12), 14 skipped, 3 xfailed. Lint clean. Still no framework, no key, no mic.
…ot from us Hosts the agent as a Pipecat frame processor. Two processors rather than one, because the two facts we need sit on opposite sides of the TTS service. AgentNodeProcessor sits BEFORE tts: finalised transcript in, reply pushed downstream chunk by chunk as it arrives, so synthesis starts on the opening clause instead of after the last one. SpokenWitness sits AFTER tts. Pipecat's TTS services emit TTSTextFrame as they synthesise, which is the only in-band signal for "this text is actually being spoken" -- and it travels downstream, so a processor before TTS can never see it. Without the second processor the clip point on a barge-in would be "what we handed to TTS", which is ahead of what the human heard by the entire audio buffer. Approximating there would defeat the generated/spoken split the previous commit exists for, so the witness reads the real signal instead. Three behaviours from the /voice playbook, each with a test: Only FINAL transcripts run the agent. Partials exist so a UI can show words appearing; acting on them runs the agent several times per sentence and answers a question that is not finished. An empty transcript costs nothing -- no agent call, no model spend. A VAD that trips on a door slam must not produce a reply to a sound. A new utterance supersedes a reply still running, and cancellation is AWAITED. Returning while the old task is still alive lets a superseded reply push text after the new turn has started, which is how two answers end up interleaved in one ear. Two test bugs of my own, both found by running rather than reasoning: TranscriptionFrame SUBCLASSES TextFrame in Pipecat, so `isinstance(f, TextFrame)` also matched the transcript passing through and made a "no reply was produced" assertion pass for the wrong reason -- it now compares exact type. And TTSTextFrame requires `aggregated_by`, so the frames I was constructing could not have been built at all. Pipecat is imported lazily inside a function. A package that cannot be imported without an optional extra is one that cannot be introspected, documented or doctored, and missing_dependency() names the install rather than surfacing an ImportError from three frames inside the framework. `run()` is explicitly NotImplementedError until the transport/VAD task; host() is usable now, which is what the demo needs. 801 passed (+7), 14 skipped, 3 xfailed. Lint clean. The adapter tests skip cleanly with no [pipecat] extra; the seam's own tests still need no framework at all.
LiveKit has no pipeline node to drop an agent into. It has an AgentSession with slots (stt=, vad=, llm=, tts=) and the agent goes in the llm one, so what we implement is llm.LLM: given a chat context, return a stream of ChatChunk. That is a genuinely different shape from Pipecat's FrameProcessor, and having now written both, the seam being text-in/text-out rather than a pipeline node is settled rather than argued. The asymmetry worth recording is who reports what was actually spoken. On Pipecat we add a processor after TTS because nothing else reports it. LiveKit already does the work: it emits `conversation_item_added` with a ChatMessage carrying `interrupted` and content ALREADY truncated to the words that played. So here the adapter subscribes instead of measuring. A second measurement of ours would not be more accurate than the framework's own -- it would just be a different number to reconcile when they disagree. Only the newest user message runs the agent, though LiveKit hands over the whole context. AgentShip keeps its own conversation through the session id -- the same short-term memory the REST path uses -- so replaying LiveKit's transcript as well would give the agent the conversation twice and have it answer as though the user had repeated themselves. Tested, because it is invisible until the second turn. Tools passed by LiveKit are accepted and ignored: tools belong to the agent's spec and run inside the engine, and injecting a second set would give the model two competing tool lists for one turn. Also adds `get_adapter(name)`, so an unknown `voice.framework` fails at selection naming the real choices rather than deep inside a session. Found by running, not reasoning: my own test helper `_hosted()` was shadowed by the local `turn, _hosted = _hosted()`, which raised UnboundLocalError in two tests -- a name I would have kept had the suite not caught it. 811 passed (+10), 14 skipped, 3 xfailed. Lint clean. Both adapters skip cleanly without their extras; the seam still needs no framework at all.
Adds pipeline assembly and drives the cascade end to end with no key, no network and no microphone: audio arrives, STT transcribes, the agent answers, TTS speaks, and the witness records what was spoken -- through Pipecat's real Pipeline. What is faked is the PROVIDER, never our own code. The stand-ins subclass Pipecat's real STTService/TTSService (whose only abstract methods are run_stt/run_tts), so the pipeline, the frame ordering, the agent node and the witness are all genuine. Stubbing our processors would have proven only that the stubs agree with each other. Assembly takes the services as arguments rather than building them from config, which is what makes that possible; `run()` will be the thin layer that resolves real providers and calls it. Order lives in one readable list, because the order IS the design -- particularly that the witness sits AFTER tts, since that is the only place the frame saying what is actually being spoken can be seen. Two things the integration corrected, neither of which I would have found by reading: The witness was listening for TTSTextFrame alone. Pipecat's own source says `push_text_frames: Whether to push TextFrames` -- TTSTextFrame is the WORD-TIMESTAMP path, and a service without word timestamps pushes a plain TextFrame. Listening for one of the two would have silently recorded nothing for most services, and "nothing was spoken" is precisely the failure this witness exists to prevent. Then, having widened it, I had to check the opposite risk: that the witness was now catching the agent's own TextFrames passing through and so echoing generation rather than observing speech. Measured it -- the agent generated two chunks, TTS was called with two sentences, and the witness recorded ONE. Speech is a strict prefix of generation, which is the invariant that makes an interrupted transcript honest, so the test now asserts exactly that rather than the weaker "something was recorded" it asserted while I was wrong about the frame type. A passing assert is not evidence until you know WHY it passes: `assert turn.spoken` was green both before and after the frame-type fix, for different reasons. NOTE for the transport task: PipelineTask and PipelineRunner are deprecated since Pipecat 1.3 in favour of the worker API. They still work and the tests use them; `run()` should be written against the current one rather than inheriting a deprecation. 816 passed (+3), 14 skipped, 3 xfailed. Lint clean.
harshuljain13
force-pushed
the
feat/p07-voice-package
branch
from
September 11, 2026 00:10
c71b2f9 to
916bf6b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — the pieces below are done and green;
run()(transport/VAD/STT/TTS assembly) is not.Picking this up on another machine? Everything you need is in this description, because
.spec-dev/lives at the workspace root and is not in any git repo — the phase-speccorrections below exist only on the machine they were written on.
Decisions taken (agreed before coding)
that voice is transport in front of the identical agent, so memory/tools/tracing are inherited
Four spec corrections, verified against the real dependencies
The phase was designed in July against Pipecat 1.6 / LiveKit Agents 1.6.7. Both are superseded
(pipecat-ai 1.8.1, livekit-agents 1.8.0), and installing and introspecting them
invalidates a structural assumption.
1. There is no component both frameworks can hold. C3 assumed one shared
AgentNodeProcessor. They disagree about where the agent sits:No shared base class. What they do share is smaller — given what the human said, stream back
what the agent says — which is
VoiceTurn, the whole vendor-free surface.2. The ABC's method is
host(turn), notas_pipeline_node(agent)— the old name encodesPipecat's model and is wrong for LiveKit.
3. Latency is a task-one return value, not a 10c concern.
llm_ttft_msis kept separatefrom
llm_total_ms; that delta is the entire value of streaming. Unrun stages reportNone,not
0— zero is a measurement.4. Barge-in is a memory-correctness problem.
[cancelled by user]must land where audiostopped, not where generation stopped, or the model believes it said things nobody heard.
The asymmetry that matters
Who reports what was actually spoken differs by framework:
SpokenWitnessprocessor, placed after TTS —TTSTextFrametravels downstream, so a processor before TTS can never see itconversation_item_added— LiveKit already truncates the message to the words that played, so we subscribe rather than take a worse second measurementWhat is done
811 passed, 14 skipped, 3 xfailed. Lint clean. The seam's tests need no framework, no key and
no microphone; each adapter's tests skip cleanly without its extra.
What is next
run()on both adapters — transport / VAD / STT / TTS assemblyagentship voice serveCLI +doctorpre-flightagentship-voiceto the six lockstep packages — deliberately excluded until it shipsKnown, not fixed
Abandoning a stream mid-iteration makes OpenTelemetry log
Failed to detach context— the spanwas attached in a different context from the one
GeneratorExittears down in. Harmless today(logged, not raised) but it fires on every barge-in, which for voice is the normal case.