docs: add full-duplex voice AI assistant guide by maciejkucmierz-swm · Pull Request #1344 · software-mansion/react-native-audio-api

maciejkucmierz-swm · 2026-10-14T14:29:08Z

Based on the APIs introduced in:

⚠️ Breaking changes ⚠️

None. Documentation-only change.

Introduced changes

Add “Build a full-duplex voice AI assistant” guide for React Native Audio API 1.0.
Demonstrate communication-session configuration, voice-processing capture, Android voice output classification, and OpenAI Realtime audio streaming.
Cover queued playback, barge-in, late-decode invalidation, and conservative conversation truncation.
Distinguish essential audio functionality from optional UI features. Keep application-specific error handling and cleanup as comments/pseudocode.
Include a call-path diagram, native setup instructions, and a suggested local MVP token server.
Clarify that echo suppression depends on the device, OS, and audio route.

Only the new guide is included; existing documentation pages are unchanged.

Checklist

Updated relevant documentation
Added/Conducted relevant tests — docs build, formatting, link checks, and six mocked audio-path checks passed. Physical-device validation was not performed.
Performed self-review of the code

github-actions · 2026-10-14T14:43:12Z

WPT non-regression comparison

PASS — no regressions · 0 improved section(s) · overall 3427 → 3427 (0)

Unchanged sections (28)

Spec section	Base pass	Head pass
Processing model	1	1
Other	56	56
AnalyserNode	138	138
AudioBuffer	140	140
AudioBufferSourceNode	220	220
AudioContext	58	58
AudioNode	261	261
AudioParam	680	680
BiquadFilterNode	275	275
ChannelMergerNode	30	30
ChannelSplitterNode	7	7
ConstantSourceNode	64	64
ConvolverNode	178	178
DelayNode	127	127
DestinationNode	0	0
DynamicsCompressorNode	4	4
GainNode	15	15
IIRFilterNode	87	87
MediaElementAudioSourceNode	0	0
MediaStreamAudioDestinationNode	1	1
MediaStreamAudioSourceNode	0	0
OfflineAudioContext	35	35
OscillatorNode	104	104
PannerNode	727	727
PeriodicWave	33	33
ScriptProcessorNode	0	0
StereoPannerNode	104	104
WaveShaperNode	82	82

_{Baseline: eb17d846e2e16cf3c82e0807a35f30565f10653e · Candidate: 6ee10db6abdcd1dbf5ecc52e079cb8a519a43bc2}

_{Workflow run · this comment is updated on every push.}

mdydek

can it be written more begginer-friendly when it comes to using our library. I think we can assume the user's knowledge about tokens and other stuff, but focus more on the features used from our library

mdydek

I like the structure much better, but I feel like we could give some more code snippets rather than wall of text, prefer keeping it brief and giving some mini code snippets that can help the user to understand how to structure that

mdydek · 2026-10-20T17:14:52Z

+`}
+/>
+
+The examples below illustrate these interfaces, rather than a complete application. The communication settings described here require Android 12 (API 31)+ or iOS 14+.


Suggested change

The examples below illustrate these interfaces, rather than a complete application. The communication settings described here require Android 12 (API 31)+ or iOS 14+.

The examples below illustrate these interfaces, but bear in mind that the communication settings described here require Android 12 (API 31)+ or iOS 14+.

mdydek · 2026-10-20T17:15:19Z

+`}
+/>
+
+The examples below illustrate these interfaces, rather than a complete application. The communication settings described here require Android 12 (API 31)+ or iOS 14+.


and also we can wrap this in an info block

mdydek · 2026-10-20T17:16:39Z

+await AudioManager.setAudioSessionActivity(true);
+```
+
+The configuration must be in place before creating the audio context or starting capture. Microphone permission is still required; Android communication routing also needs `MODIFY_AUDIO_SETTINGS`. See [permissions](../other/non-expo-permissions.mdx) for the native declarations.


Suggested change

The configuration must be in place before creating the audio context or starting capture. Microphone permission is still required; Android communication routing also needs `MODIFY_AUDIO_SETTINGS`. See [permissions](../other/non-expo-permissions.mdx) for the native declarations.

The configuration must be in place before starting the playback or capture. Microphone permission is still required; Android communication routing also needs `MODIFY_AUDIO_SETTINGS`. See [permissions](../other/non-expo-permissions.mdx) for the native declarations.

mdydek · 2026-10-20T17:17:37Z

+});
+```
+
+The output profile is fixed when the context is created and is ignored on iOS. It does **not** acquire audio focus, activate communication mode, or select a speaker by itself. Changing that one option is not a substitute for configuring the session and recorder.


Suggested change

The output profile is fixed when the context is created and is ignored on iOS. It does **not** acquire audio focus, activate communication mode, or select a speaker by itself. Changing that one option is not a substitute for configuring the session and recorder.

mdydek · 2026-10-20T17:18:07Z

+
+The output profile is fixed when the context is created and is ignored on iOS. It does **not** acquire audio focus, activate communication mode, or select a speaker by itself. Changing that one option is not a substitute for configuring the session and recorder.
+
+On **Android**, the settings can also be combined differently: keep the `inCommunication` session, speaker request, and `voiceCommunication` microphone preset, but create the context with `androidOutputProfile: 'media'` (the default). This keeps media-classified playback while the communication session is active.


are you sure it will work with the media output profile?

mdydek · 2026-10-20T17:21:05Z

+await recorder.start();
+```
+
+Channel indices start at zero, so `getChannelData(0)` returns the mono channel as a `Float32Array`. Its samples are floating-point values, normally between `-1` and `1`; they are not already PCM16 bytes or base64 text.


Suggested change

Channel indices start at zero, so `getChannelData(0)` returns the mono channel as a `Float32Array`. Its samples are floating-point values, normally between `-1` and `1`; they are not already PCM16 bytes or base64 text.

`getChannelData()` returns the channel as a `Float32Array`. Its samples are floating-point values, normally between `-1` and `1`; they are not already PCM16 bytes or base64 text.

mdydek · 2026-10-20T17:21:45Z

+
+An AI reply often arrives as many small pieces rather than one complete recording. [`AudioBufferQueueSourceNode`](../sources/audio-buffer-queue-source-node.mdx) is a playback source that consumes a queue of decoded audio buffers in order. Unlike creating a separate source for every chunk, it gives the reply one continuous playback path.
+
+`connect()` attaches the source to the graph. `audioContext.destination` is the graph's output to the device:


also links to the documentation parts

mdydek · 2026-10-20T17:22:30Z

+
+Call `start()` only once for that queue source, not once per chunk or reply. More buffers can be enqueued while it is playing. An empty queue waits for more audio, so it can also be reused for later replies in the same conversation.
+
+**If pieces of a reply play in the wrong order**, check the decoding flow. The queue preserves enqueue order, but independent asynchronous decode calls may finish in a different order from arrival. The app needs to serialize decoding and enqueueing; the queue cannot restore the original order afterwards. Gaps can also mean the queue is waiting for data, rather than that another `start()` call is needed.


wrap it in the tip block

mdydek · 2026-10-20T17:23:25Z

+- `AudioManager` configures the communication session and route.
+- `AudioRecorder` supplies microphone samples through callbacks.
+- `AudioContext` and `AudioBufferQueueSourceNode` decode and play streamed replies.


same case with links to docs pages

mdydek · 2026-10-20T17:24:08Z

+- `AudioContext` and `AudioBufferQueueSourceNode` decode and play streamed replies.
+- The app coordinates audio formats, decode ordering, and interruption decisions.
+
+Understanding those boundaries helps distinguish an audio-data problem from a routing or conversation-state problem. See the [AudioManager](../system/audio-manager.mdx), [AudioRecorder](../inputs/audio-recorder.mdx), and [queue source](../sources/audio-buffer-queue-source-node.mdx) references for the individual APIs.


Suggested change

Understanding those boundaries helps distinguish an audio-data problem from a routing or conversation-state problem. See the [AudioManager](../system/audio-manager.mdx), [AudioRecorder](../inputs/audio-recorder.mdx), and [queue source](../sources/audio-buffer-queue-source-node.mdx) references for the individual APIs.

See the [AudioManager](../system/audio-manager.mdx), [AudioRecorder](../inputs/audio-recorder.mdx), and [queue source](../sources/audio-buffer-queue-source-node.mdx) references for the individual APIs.

docs: add full-duplex voice AI assistant guide

b2ef94a

closetcaiman added the documentation Additions, corrections, or updates to documentation and technical specs label Oct 14, 2026

mdydek requested changes Oct 14, 2026

View reviewed changes

docs: focus full-duplex guide on audio API concepts

6ee10db

mdydek requested changes Oct 20, 2026

View reviewed changes

	The examples below illustrate these interfaces, rather than a complete application. The communication settings described here require Android 12 (API 31)+ or iOS 14+.
	The examples below illustrate these interfaces, but bear in mind that the communication settings described here require Android 12 (API 31)+ or iOS 14+.

	The configuration must be in place before creating the audio context or starting capture. Microphone permission is still required; Android communication routing also needs `MODIFY_AUDIO_SETTINGS`. See [permissions](../other/non-expo-permissions.mdx) for the native declarations.
	The configuration must be in place before starting the playback or capture. Microphone permission is still required; Android communication routing also needs `MODIFY_AUDIO_SETTINGS`. See [permissions](../other/non-expo-permissions.mdx) for the native declarations.


		The output profile is fixed when the context is created and is ignored on iOS. It does not acquire audio focus, activate communication mode, or select a speaker by itself. Changing that one option is not a substitute for configuring the session and recorder.

		On Android, the settings can also be combined differently: keep the `inCommunication` session, speaker request, and `voiceCommunication` microphone preset, but create the context with `androidOutputProfile: 'media'` (the default). This keeps media-classified playback while the communication session is active.

	Channel indices start at zero, so `getChannelData(0)` returns the mono channel as a `Float32Array`. Its samples are floating-point values, normally between `-1` and `1`; they are not already PCM16 bytes or base64 text.
	`getChannelData()` returns the channel as a `Float32Array`. Its samples are floating-point values, normally between `-1` and `1`; they are not already PCM16 bytes or base64 text.


		An AI reply often arrives as many small pieces rather than one complete recording. [`AudioBufferQueueSourceNode`](../sources/audio-buffer-queue-source-node.mdx) is a playback source that consumes a queue of decoded audio buffers in order. Unlike creating a separate source for every chunk, it gives the reply one continuous playback path.

		`connect()` attaches the source to the graph. `audioContext.destination` is the graph's output to the device:


		Call `start()` only once for that queue source, not once per chunk or reply. More buffers can be enqueued while it is playing. An empty queue waits for more audio, so it can also be reused for later replies in the same conversation.

		If pieces of a reply play in the wrong order, check the decoding flow. The queue preserves enqueue order, but independent asynchronous decode calls may finish in a different order from arrival. The app needs to serialize decoding and enqueueing; the queue cannot restore the original order afterwards. Gaps can also mean the queue is waiting for data, rather than that another `start()` call is needed.

	Understanding those boundaries helps distinguish an audio-data problem from a routing or conversation-state problem. See the [AudioManager](../system/audio-manager.mdx), [AudioRecorder](../inputs/audio-recorder.mdx), and [queue source](../sources/audio-buffer-queue-source-node.mdx) references for the individual APIs.
	See the [AudioManager](../system/audio-manager.mdx), [AudioRecorder](../inputs/audio-recorder.mdx), and [queue source](../sources/audio-buffer-queue-source-node.mdx) references for the individual APIs.

Conversation

maciejkucmierz-swm commented Oct 14, 2026

⚠️ Breaking changes ⚠️

Introduced changes

Checklist

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WPT non-regression comparison

Uh oh!

mdydek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdydek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Oct 14, 2026 •

edited

Loading