diff --git a/docs.json b/docs.json index b91ee717..c95f94bf 100644 --- a/docs.json +++ b/docs.json @@ -72,6 +72,7 @@ "group": "Fundamentals", "pages": [ "pipecat/fundamentals/service-settings", + "pipecat/fundamentals/interruptions", "pipecat/fundamentals/user-input-muting", "pipecat/fundamentals/detecting-user-idle", "pipecat/fundamentals/stt-latency-tuning", diff --git a/overview/introduction.mdx b/overview/introduction.mdx index 6639ffd3..d5b272fe 100644 --- a/overview/introduction.mdx +++ b/overview/introduction.mdx @@ -3,7 +3,7 @@ title: "Introduction" description: "Build voice and multimodal AI agents with the Pipecat ecosystem." --- -Pipecat is an open source ecosystem for building voice and multimodal AI agents. It provides everything you need to create, deploy, and scale real-time AI applications that can see, hear, and speak. +Pipecat is an open source ecosystem for building voice and multimodal AI agents. It provides everything you need to create, deploy, and scale real-time AI applications that can see, hear, and speak. The framework is free to use under the BSD-2 license, and works with any AI provider and any hosting environment — you pay only for the services you choose. ## The Pipecat Ecosystem diff --git a/pipecat-cloud/introduction.mdx b/pipecat-cloud/introduction.mdx index cb2df907..e84f886b 100644 --- a/pipecat-cloud/introduction.mdx +++ b/pipecat-cloud/introduction.mdx @@ -49,6 +49,10 @@ description: "Deploy your AI agents to production at scale" +## Pricing + +Pipecat Cloud uses usage-based pricing. See the [pricing page](https://www.daily.co/pricing/pipecat-cloud) for current rates and a calculator to estimate your total cost per minute, including third-party AI services. + ## Next Steps diff --git a/pipecat/fundamentals/interruptions.mdx b/pipecat/fundamentals/interruptions.mdx new file mode 100644 index 00000000..1b5f43da --- /dev/null +++ b/pipecat/fundamentals/interruptions.mdx @@ -0,0 +1,141 @@ +--- +title: "Interruptions" +description: "How Pipecat stops the bot when the user speaks, what happens to in-flight LLM and TTS output, and how to control it." +--- + +Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready for the new user input. + +Interruptions are **enabled by default**. This page explains how they work under the hood, what ends up in the conversation context, and how to configure or trigger them yourself. + + + Looking for how Pipecat decides *when* a user turn starts and ends? See + [Speech Input & Turn Detection](/pipecat/learn/speech-input) and [User Turn + Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies). + This page covers what happens *after* an interruption is triggered. + + +## What happens when the user interrupts + +When a [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies) triggers with `enable_interruptions=True` (the default), the user aggregator broadcasts an `InterruptionFrame` both upstream and downstream through the pipeline. From there: + + + + `InterruptionFrame` is a + [SystemFrame](/api-reference/server/frames/system-frames), so each processor + handles it immediately instead of waiting behind queued frames. Each + processor cancels its processing task and discards queued `DataFrame`s and + `ControlFrame`s. Frames marked uninterruptible (like + `FunctionCallResultFrame` and `EndFrame`) are preserved and still processed. + + + The in-flight LLM completion is cancelled mid-stream. Any registered + function calls with `cancel_on_interruption=True` are cancelled and emit a + `FunctionCallCancelFrame`. See [Function + Calling](/pipecat/learn/function-calling) for details. + + + The TTS service stops synthesizing, clears its text aggregation and word + timestamps, and drops pending output. + + + The output transport drains its audio queue, discarding audio that was + generated but not yet played. If a background audio mixer is active, the + transport drains only the bot's speech so the background audio keeps playing + without a gap. + + + +The result: the bot goes silent within roughly one audio write, and the pipeline is clean and ready for the user's new turn. + +## What ends up in the context + +A common question: if the bot is cut off mid-sentence, what does the LLM context contain? + +**Only the words that were actually spoken.** As the bot speaks, the output transport pushes `TTSTextFrame`s downstream in sync with audio playback. Text that never played never reaches the assistant context aggregator. On interruption, the aggregator commits the partial, spoken-so-far text to the context as the assistant message. + +This means the LLM's next completion sees an accurate transcript of the conversation: the bot's message ends where the user cut it off, not where the LLM's generation ended. + +You can observe this with the `on_assistant_turn_stopped` event, which reports the committed text and whether the turn was interrupted: + +```python +@assistant_aggregator.event_handler("on_assistant_turn_stopped") +async def on_assistant_turn_stopped(aggregator, message): + if message.interrupted: + print(f"Bot was cut off after saying: {message.content}") +``` + +## Controlling interruptions + +**Let the bot finish speaking** with [user input muting](/pipecat/fundamentals/user-input-muting). Mute strategies block user audio, transcriptions, and interruption signals while active, so speech over the bot is discarded rather than answered later. Use `AlwaysUserMuteStrategy` to mute whenever the bot is speaking, or `FirstSpeechUserMuteStrategy` to protect just the introduction: + +```python +from pipecat.processors.aggregators.llm_response_universal import ( + LLMContextAggregatorPair, + LLMUserAggregatorParams, +) +from pipecat.turns.user_mute import AlwaysUserMuteStrategy + +user_aggregator, assistant_aggregator = LLMContextAggregatorPair( + context, + user_params=LLMUserAggregatorParams( + user_mute_strategies=[AlwaysUserMuteStrategy()], + ), +) +``` + +**Require a minimum number of words** so short utterances like "okay" or "yeah" don't interrupt the bot. This is configured on the [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies): + +```python +from pipecat.turns.user_start import MinWordsUserTurnStartStrategy + +start_strategy = MinWordsUserTurnStartStrategy(min_words=3) +``` + +**Filter out backchannels with a model.** The [Krisp VIVA Interruption Prediction strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#krispvivaipuserturnstartstrategy) distinguishes genuine interruptions from acknowledgments like "uh-huh". + +**Disable interruptions** so in-flight work is never cancelled: + +```python +from pipecat.turns.user_start import VADUserTurnStartStrategy + +start_strategy = VADUserTurnStartStrategy(enable_interruptions=False) +``` + + + Disabling interruptions does not ignore the user. Speech over the bot is + still transcribed and processed as a normal user turn — the bot's reply is + queued and plays as soon as the current speech finishes. If you want the bot + to finish speaking *and* discard what the user said over it, use mute + strategies instead. + + +## Triggering an interruption yourself + +Sometimes the bot should stop itself: a timeout fires, an external event arrives, or your own logic decides the current response is no longer relevant. + +From inside a custom `FrameProcessor`, broadcast the interruption directly: + +```python +await self.broadcast_interruption() +``` + +From code that has a reference to a processor or the worker, you can also push an `InterruptionWorkerFrame`. The pipeline worker converts it into an `InterruptionFrame` and sends it through the whole pipeline: + +```python +from pipecat.frames.frames import InterruptionWorkerFrame + +await worker.queue_frame(InterruptionWorkerFrame()) +``` + +Both approaches run the same interruption flow described above: the bot stops speaking, in-flight work is cancelled, and the spoken-so-far text is committed to context. + +## Speech-to-speech services + +Realtime speech-to-speech services (like Gemini Live and OpenAI Realtime) handle interruption detection on the provider side. In these pipelines, the service emits the speaking events and Pipecat's aggregators follow along using [external turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies#externaluserturnstartstrategy). The provider decides when the user barged in; Pipecat still flushes local audio output so the bot goes silent right away. + +## Related + +- [Speech Input & Turn Detection](/pipecat/learn/speech-input) - how user turns are detected +- [User Turn Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) - full strategy reference +- [User Input Muting](/pipecat/fundamentals/user-input-muting) - suppress user input while the bot speaks +- [System Frames](/api-reference/server/frames/system-frames) - `InterruptionFrame` reference diff --git a/pipecat/get-started/introduction.mdx b/pipecat/get-started/introduction.mdx index cfd6aa23..d67e9201 100644 --- a/pipecat/get-started/introduction.mdx +++ b/pipecat/get-started/introduction.mdx @@ -3,7 +3,7 @@ title: "Pipecat" description: "Open source Python framework for building voice and multimodal AI agents." --- -Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. +Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. The framework is free to use under the BSD-2 license, and you choose your own stack: any speech-to-text, LLM, and text-to-speech provider, hosted on your own infrastructure or [Pipecat Cloud](https://www.daily.co/pricing/pipecat-cloud) — you pay only for the providers you choose. Want to dive right in? Build and run your first Pipecat application diff --git a/pipecat/get-started/quickstart.mdx b/pipecat/get-started/quickstart.mdx index 4447521a..6c67f17e 100644 --- a/pipecat/get-started/quickstart.mdx +++ b/pipecat/get-started/quickstart.mdx @@ -225,11 +225,11 @@ tts = CartesiaTTSService( voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"), ), ) -llm = OpenAIRealtimeLLMService( +llm = OpenAIResponsesLLMService( api_key=os.getenv("OPENAI_API_KEY"), - settings=OpenAIRealtimeLLMService.Settings( + settings=OpenAIResponsesLLMService.Settings( model=os.getenv("OPENAI_MODEL", "gpt-4.1"), - system_instruction="You are a helpful assistant.", + system_instruction="You are a helpful assistant in a voice conversation. Your responses will be spoken aloud, so avoid emojis, bullet points, or other formatting that can't be spoken. Respond to what the user said in a creative, helpful, and brief way.", ), ) ``` @@ -297,25 +297,27 @@ The worker handles pipeline execution, collects metrics, and manages RTVI events Event handlers manage the bot's lifecycle and user interactions: ```python +# Kick off the conversation when the client signals it's ready +@worker.rtvi.event_handler("on_client_ready") +async def on_client_ready(rtvi): + context.add_message({"role": "developer", "content": "Start by concisely introducing yourself."}) + await worker.queue_frames([LLMRunFrame()]) + # Event handler for when a client connects @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): - logger.info(f"Client connected") - # Add a greeting message to the context - context.add_message("developer", "Say hello and briefly introduce yourself.") - # Prompt the bot to start talking when the client connects - await worker.queue_frames([LLMRunFrame()]) + logger.info("Client connected") # Event handler for when a client disconnects @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): - logger.info(f"Client disconnected") + logger.info("Client disconnected") # Cancel the worker when the client disconnects # This stops the pipeline and all processors, cleaning up resources await worker.cancel() ``` -When a client connects, the bot adds a greeting instruction and queues a context frame to initiate the conversation. When disconnecting, it properly cancels the worker to clean up resources. +When the client signals it's ready over the RTVI protocol, the bot adds a greeting instruction and queues an `LLMRunFrame` to start the conversation. When the client disconnects, the worker is cancelled to clean up resources. ### Running the Pipeline @@ -325,9 +327,10 @@ Finally, the pipeline is executed by a WorkerRunner: # Create a WorkerRunner to run the worker runner = WorkerRunner(handle_sigint=False) -# Finally, run the worker using the runner +# Finally, add the worker and run it # This will start the pipeline and begin processing frames -await runner.run(worker) +await runner.add_workers(worker) +await runner.run() ``` The runner manages the pipeline's execution lifecycle. Note that `handle_sigint=False` because the main runner handles system signals. diff --git a/pipecat/learn/context-management.mdx b/pipecat/learn/context-management.mdx index 04bf3e84..9bd80a08 100644 --- a/pipecat/learn/context-management.mdx +++ b/pipecat/learn/context-management.mdx @@ -215,7 +215,9 @@ Place the assistant context aggregator **after `transport.output()`**. This posi Always place the assistant context aggregator **after** `transport.output()` - to ensure proper word-level context updates during interruptions. + to ensure proper word-level context updates during interruptions. See + [Interruptions](/pipecat/fundamentals/interruptions#what-ends-up-in-the-context) + for what the context contains when the bot is cut off mid-sentence. ## Manual Context Control diff --git a/pipecat/learn/speech-input.mdx b/pipecat/learn/speech-input.mdx index eee04be6..196bbe83 100644 --- a/pipecat/learn/speech-input.mdx +++ b/pipecat/learn/speech-input.mdx @@ -156,26 +156,9 @@ stop_strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6) ### Interruptions -Interruptions stop the bot when the user starts speaking. This is controlled by the `enable_interruptions` parameter on start strategies (enabled by default). +When a user turn starts with interruptions enabled (the default, via the `enable_interruptions` parameter on start strategies), the bot immediately stops speaking, in-flight work is cancelled, and the pipeline is ready for the new user input. -When a user turn starts with interruptions enabled: - -1. Bot immediately stops speaking -2. Pending audio and text is cleared -3. Pipeline ready for new user input - -To disable interruptions: - -```python -from pipecat.turns.user_start import VADUserTurnStartStrategy - -start_strategy = VADUserTurnStartStrategy(enable_interruptions=False) -``` - - - Keep interruptions enabled (default) for natural conversations. This enables - users to interrupt the bot mid-sentence, just like human conversations. - +Keep interruptions enabled for natural conversations. For how the cancellation works, what ends up in the LLM context, and how to tune, disable, or trigger interruptions yourself, see [Interruptions](/pipecat/fundamentals/interruptions). ## Best Practices