Flutter voice assistant: fast speech recognition, interruption, and a large knowledge base

[jshannon2002-source]

🏷️ Discussion Type

Question

Body

I'm a respiratory-care professional developing a Flutter Android education app with coding assistance. I'm seeking volunteer technical guidance on making its voice assistant responsive while supporting a large domain of knowledge. I am using Grok, ChatGPT, Muse, and similar apps as examples of the conversational experience I want: quick responses, strong speech recognition, and the ability to interrupt while the assistant is talking.

I'm interested in documented architecture patterns and properly licensed code examples or reference implementations. I understand the full implementation of a commercial assistant may be private. I'm keeping my full app source private as well; this is a request for free technical advice, not paid services.

What I need help understanding:

1. For low latency, when should a Flutter app use native speech-to-speech versus a streaming speech-to-text -> LLM -> text-to-speech pipeline? How do you measure and reduce time to first audible response?
2. How do you implement reliable interruption (barge-in) while narration or an AI answer is playing? What should happen to microphone capture, VAD/turn detection, response cancellation, queued playback, and conversation context when the user interrupts?
3. How can the assistant use a large domain knowledge base without sending all its content in every prompt or making every answer slow? How would you separate model knowledge from app-specific content, and use retrieval, pre-indexing, caching, or bounded context? What is the latency/accuracy tradeoff?
4. Are there Flutter/Dart or Android examples, reusable components, or open-source projects you recommend for these patterns? Please include the relevant license and any practical limits.

There is also a concrete issue in my current implementation:

• A first spoken command opens the requested study-guide chapter and starts narration.
• In Android emulator tests, later injected commands such as "Stop," a question, or "Home" during narration do not produce the expected input/speech events.
• The capture track and RTP counters remain active with nonzero reported energy. That alone does not prove intelligible speech is reaching the recognizer.
• I have not established that the same behavior occurs on a physical device.

Current setup: Flutter/Dart Android app; flutter_webrtc pinned to 1.6.2+hotfix.3; WebRTC speech session using OpenAI Realtime; semantic VAD with application-controlled responses and speech interruption configured. Client/server configuration does not intentionally disable turn detection during narration.

What diagnostics would distinguish Android/WebRTC capture or audio-processing suppression from a provider-side VAD/event problem? Which audio source/mode, audio-focus, echo-cancellation, or emulator-input checks would you try first?

I can prepare a small sanitized reproduction and redacted diagnostics based on what would be most useful. Public replies with suggestions or relevant experience would be appreciated. Thank you.

Guidelines

• I have read and understood this category's guidelines before making this post.

[malkamanuranga]

This sounds like a fantastic project. Building real-time voice apps in Flutter is tricky but incredibly rewarding right now. I can share some insights, especially regarding the WebRTC architecture and the specific bug you are hitting.

1. The Emulator Bug (Start Here)
   To save your sanity: stop testing voice barge-in on the Android emulator. Emulators are notoriously terrible at handling audio routing, hardware audio focus, and Acoustic Echo Cancellation (AEC).

What is likely happening is either the emulator's OS is suppressing the virtual microphone while the speaker is active, or the microphone is picking up the narration audio (loopback), turning it into a muddy signal that the OpenAI VAD (Voice Activity Detection) rejects as background noise rather than human speech.

Immediate Diagnostics:

The Headphone Test: Build the app to a physical Android device and test it while wearing wired/Bluetooth headphones. If barge-in works perfectly with headphones but fails when using the phone's external speaker, your issue is AEC. The mic is hearing the app's own narration.

WebRTC Constraints: Make sure your getUserMedia constraints explicitly enable hardware echo cancellation: {"audio": {"echoCancellation": true, "noiseSuppression": true}}.

Audio Focus: Check your Android audio_session or just_audio configuration (if you are using those). Ensure the audio routing is set to "communications" mode (like a phone call), not "media" mode (like Spotify). Communications mode activates the phone's hardware echo cancellers.

2. S2S vs. STT -> LLM -> TTS Pipeline
   Since you want that "ChatGPT app" feel (quick responses, fluid interruptions), sticking with Native Speech-to-Speech (like the OpenAI Realtime WebRTC API you are currently using) is exactly the right path.

Traditional pipelines (STT -> LLM -> TTS) are much easier to debug because you can see the text at every step, but they naturally stack latency (often 1.5 to 2 seconds). WebRTC S2S gets you down to ~300-500ms (Time to First Audio).

3. Handling Interruption (Barge-in)
   For true barge-in, the system needs to react instantly when the VAD detects new speech. Since you are using OpenAI Realtime, rely on their Server VAD.
   When the Server VAD detects a user talking over the assistant:

OpenAI will automatically fire a speech_started event over the WebSocket/WebRTC data channel.

Your Flutter app must listen for this event and instantly command your local audio player to stop() and flush the remaining audio queue.

OpenAI handles the context automatically—they figure out exactly which word the assistant was cut off at, truncate the transcript, and append the user's new interrupted thought.

4. Handling Large Knowledge Domains (RAG)
   Do not pass the whole study guide into the prompt—it will kill your latency and cost a fortune. You want Retrieval-Augmented Generation (RAG).

Pre-process your respiratory care study guides into small chunks (paragraphs).

Create vector embeddings for these chunks (OpenAI has a cheap embeddings API for this).

Store them in a vector database (like Supabase, Pinecone, or even locally on the device using SQLite VSS).

The Flow: User asks a question -> You quickly query the vector database for the top 3 most relevant paragraphs (takes ~100ms) -> You use OpenAI Realtime's "function calling/tools" feature to seamlessly inject those 3 paragraphs into the context -> The AI answers out loud.

5. Flutter References
   I highly recommend looking at the open-source examples by LiveKit. Even if you don't use their paid backend servers, their open-source Flutter SDK (livekit_client on pub.dev) has massive, well-documented examples of handling WebRTC, audio focus across iOS/Android, and AI voice agents. Looking at their source code for how they manage the microphone tracks during AI playback will likely solve a lot of your headaches.

Good luck with the app! Respiratory care education is a great use case for this tech. Let us know what happens when you test it on a physical device. good luck.....

[ahbaqdadi]

I would start with the missing speech events before changing the overall architecture. Active RTP counters/energy are useful, but do not establish that the injected words survive the capture/processing path.

1. Separate turn detection from response generation. For your application-controlled setup, check the effective session.created/session.updated configuration, not just the request you intended to send. With VAD enabled, create_response: false lets the app decide when to call response.create; it does not itself disable speech detection. OpenAI explicitly documents retaining VAD while disabling automatic responses. Semantic VAD's eagerness concerns when an utterance ends, so tuning it is not my first explanation for no speech-start event. See VAD and manual response control.

For one reproduction, log incoming data-channel event types and timestamps before your UI/state-machine filters: session.updated, error, input_audio_buffer.speech_started, input_audio_buffer.speech_stopped, input_audio_buffer.committed, response.created, response.done. Include playback start/stop and outbound session changes on the same timeline. If an event arrives there but never reaches your command handler, investigate the client handler first. If speech-start arrives but no response follows, check your app's explicit response creation separately.

2. Run a small comparison, changing one thing at a time. Use the same harmless phrase (e.g. “stop reading now”) before narration, during narration, and immediately after stopping playback, in the same session. Repeat on a physical device with wired headphones, then speaker output. A headphone/speaker difference narrows routing/echo/processing suspects; it does not prove AEC is the cause.

On the emulator, document how you inject speech. Host-speaker playback, host-microphone capture, and an injected audio track are different paths. Android documents that the virtual microphone uses host audio only when its host-input option is enabled. For a capture diagnosis, a short synthetic sample from the actual outgoing capture path, if your instrumentation supports it, is stronger evidence than a second recorder using a different source. Avoid adding a competing microphone recorder to the test.

Record route, audio-focus changes and native audio source/mode around playback startup. VOICE_COMMUNICATION uses echo cancellation/AGC if available; neither that setting nor a requested WebRTC constraint proves hardware AEC is active. I haven't verified how your exact flutter_webrtc build maps those settings.

3. Identify who owns the narration. Is the chapter read through the Realtime remote audio track, or through a separate Flutter/Android TTS/player? OpenAI documents automatic unplayed-audio truncation for its WebRTC/SIP output on interruption. My implication for your app: a separate chapter player still needs its own stop/queue-clear logic, and any chapter-position context must be managed by your app. For a manual stop of Realtime output, the client event reference specifies response.cancel before output_audio_buffer.clear; guard against there being no active response. These events do not stop an unrelated local narrator.

A useful sanitized reproduction would contain the acknowledged turn-detection settings, that event timeline, playback owner, injection method, and the before/during/after results. That can distinguish client event filtering from a capture-path or service-side question without publishing your app source.

[jshannon2002-source]

Update October 10 — thanks for the diagnostic suggestions. We found and fixed a reproducible client-side narration bug, but have not yet verified the complete live-microphone interaction.

Reproduced failure: a speech-start/VAD event speculatively paused the guide reader. When the final transcript matched our own recently narrated text, the echo filter rejected it; the reader nevertheless remained paused. This is distinct from a genuine missing speech-start event.

Applied repair:

• Capture the exact playback owner, item, position and operation/generation before the speculative pause.
• Resume once only after confirmed rejected echo and silence, if the same reader still owns that pause. No paragraph replay or new AI answer.
• Real Stop, a partial user question, explicit/manual/system Pause, navigation, capture loss or session failure cancel that recovery.
• Reject echo at receipt before delayed transcript queuing; priority Stop stays ahead of the echo filter.

Results: the original regression failed before the repair. Afterward, 99 affected voice/host tests and 82 surrounding reader/native-control/progress tests passed; static analysis was clean. These fixtures exercise reader ownership but use synthetic transport, so they do not prove recognizer accuracy, audible interruption or echo cancellation. The fresh debug build is installed on the retained emulator. Live acceptance is still pending. A broader regression also had Windows packaging/inventory errors, so we are not claiming a release-ready app.

The separate chat-answer path remains a concern: it can stop output on raw speech-start before the final echo decision, and it does not share the guide reader's recovery ownership. Earlier live tests were mixed: narration was audible, one Stop worked, another was misrecognized, and unsolicited self-interruptions occurred. Those are historical observations, not a successful test of this new build.

What would you test next to separate residual self-echo from genuine user barge-in? Is an ownership-fenced speculative pause/recovery appropriate for a separately owned narrator, and what pattern would you use for streamed chat output that cannot simply be resumed? We'd appreciate a small licensed Flutter/WebRTC reference or a precise event-timeline checklist. We can provide a sanitized minimal reproduction; no credentials, recordings or private guide content are included here.

[ahbaqdadi]

Your update narrows this to a real client-state regression. For a separately owned reader, your ownership check is a reasonable design: recovery must consume a one-use pause token for that exact reader/item/generation. Every explicit Stop/Pause, navigation, new playback owner or session failure must invalidate it before awaiting anything. A duplicate or late echo callback must then be harmless. That is design advice based on your description; I have not inspected or run your app.

The streamed-chat path needs an additional distinction: a provisional local pause is reversible only if the stream and buffered audio actually remain available; cancellation/queue clearing is not a pause. I would check these points next:

1. Who cancels first? Log the acknowledged session.audio.input.turn_detection configuration, including both create_response and interrupt_response. Setting only create_response: false does not prevent the server cancelling a response on VAD start. For an experiment where your app owns that decision, the current client reference documents both flags false while retaining VAD events. This transfers interruption responsibility to your app; it is not an echo fix. Compare the same playback-only trial under both policies.
2. Choose an explicit chat policy. Either cancel immediately and accept that a false trigger leaves the answer interrupted, or wait for sufficiently reliable user evidence before committing cancellation. Waiting for a final transcript adds delay; measure it. If you experiment with temporarily muting/gating local WebRTC output, do not assume this preserves a replayable buffer or pauses the server's playback clock. Test that behavior separately. Do not resume an old cancelled response or automatically generate a replacement after rejecting echo.
3. On a confirmed interruption, invalidate that chat generation, cancel its still-active response, then clear WebRTC output; also stop any independent local TTS queue. Guard the response ID/state and serialize this transition so a late callback cannot clear a newer response's audio. OpenAI's client event reference specifies cancel before clear. For WebRTC, the server handles unplayed-output truncation; for a WebSocket/local PCM player, the client must track the played duration and send conversation.item.truncate. See interruption and truncation. None of that resumes a separate chapter narrator.

For a small live acceptance matrix, use a harmless synthetic sentence and run reader and streamed chat separately, on the new build:

Trial	Expected observation
Playback only; user silent	No committed Stop/new answer; any reader recovery occurs once, without replay.
User says Stop during playback	Output becomes audibly silent; no later recovery or stale audio.
User asks an unrelated short question	Genuine speech reaches the handler and produces at most one intended answer.
User deliberately repeats words just narrated	Still treated as possible user speech; textual overlap alone cannot establish echo.
Manual Pause/navigation while an echo decision is pending	Late rejection never restarts either output owner.

Repeat first with wired headphones, then speaker output at a recorded volume; retain the emulator result as a separate capture-path case. A route difference is evidence to investigate, not proof of AEC failure.

For each trial, log one monotonic timeline: playback owner/item/generation and position; effective session settings; raw speech_started/speech_stopped and committed input item ID; transcript arrival and classifier decision; pause/cancel/clear commands with target response ID; response.done status; output-buffer cleared/stopped events; and recovery or token invalidation. Record audible stop time too: sending a command is not the same as hearing silence.

Finally, dropping a transcript in your UI does not remove its already-committed audio from the server conversation. Inspect that item before the next app-controlled response. The API supports deleting a specifically identified conversation item, but only treat it as echo when you have adequate evidence; an uncertain match should not silently discard a real question. Input transcription is asynchronous guidance, not an exact record of what the speech model heard (reference).

AI-assisted with Codex (GPT-6). API details checked against current official documentation; the suggested matrix and ownership policy have not been live-tested in your Flutter build.

[NavidZamanKhan]

Hi! Since you're already using OpenAI Realtime with Flutter, I'd focus on debugging the audio input before changing your architecture.

A few suggestions:

1. Test on a physical Android device first. Emulator microphone behavior can be unreliable. Also check that microphone input is enabled in the emulator settings.

2. Log the input_audio_buffer.speech_started event. If it never appears, check whether actual speech is reaching the server, not just an active audio track.

3. Check Android microphone permissions, audio focus, MODE_IN_COMMUNICATION, and echo cancellation. Try using headphones to rule out playback interference.

4. For interruptions, keep the microphone active and use response.cancel and output_audio_buffer.clear when manually stopping a Realtime response.

5. For large knowledge bases, use RAG to retrieve only relevant information instead of sending everything to the LLM.

For latency, measure the time between the user's last spoken word and the first audible response. Native speech-to-speech is a good starting point, while STT → LLM → TTS offers more control.

Useful references:

OpenAI Realtime documentation:
https://developers.openai.com/api/docs/guides/realtime-conversations

Flutter WebRTC (MIT License):
https://github.com/flutter-webrtc/flutter-webrtc

I'd start by comparing VAD event logs from the emulator and a physical device. That should help narrow down where the issue originates.

Hope this helps!

[jshannon2002-source]

Thank you for the concrete diagnostic checklists, especially the distinction between playback ownership and cancellation. I am following up with a request for public reference code so this can be evaluated against a known baseline.

Does anyone have a maintained, openly licensed Flutter/Android example that demonstrates all of the following?

• An opt-in foreground voice session without a wake word, with the microphone remaining available during assistant speech.
• Genuine Stop/Pause or question interruption during both conversational output and long-form narration.
• A timestamped trace separating capture, VAD, transcript completion, command dispatch, cancellation/clear and audible silence.
• A test for the user deliberately repeating narrated words, so textual overlap is not mistaken for proof of echo.
• Read-along driven by actual playback/word boundaries, with only the currently spoken word highlighted, plus a visualizer driven by output activity.

A link to specific sample files or a small reproducible harness would be more useful than another broad architecture change. I am seeking free advice/reference examples; this is not a request to disclose private app source or a claim that a particular SDK is defective. Thank you.