Skip to main content

​
Voice Interaction

Communicate with your agent using voice on supported channels.

​
Supported Engines

​
Text-to-Speech (TTS)

5 TTS engines available for agent voice output. Two TTS paths (by design):
  • Settings > Voice — channel replies, WebRTC, and IM voice bubbles (5 providers + free Edge fallback, streaming, smart summary)
  • Agent config TTS — chat command “read this out loud” produces an Audio artifact via the tts_generate tool (OpenAI/ElevenLabs + Gateway billing)
Both paths are production-tested: 632+ channel voice tests + 23 harness llms/tts unit tests (98% coverage).

​
Speech-to-Text (STT)

5 STT engines for voice input recognition: Local Whisper (free, privacy-first), OpenAI Whisper, Groq, Deepgram, and xAI Grok STT. Custom OpenAI-compatible endpoints are also supported via the Base URL setting.

​
Video Audio Transcription

Video messages (including Telegram circular video notes) are automatically transcribed using the same STT pipeline. The agent understands what is said in video messages without any extra configuration.

​
Audio File Upload

Upload audio files directly in the WebUI chat input. 8 formats supported: mp3, wav, ogg, flac, m4a, aac, wma, opus. Three upload methods:
  • Click the attach button to select files
  • Drag and drop audio files into the chat area
  • Paste (some formats)
Audio file size limit is 25MB. Uploaded files are automatically transcribed via the STT pipeline and injected as context for the agent. The Tauri desktop client also supports audio upload with a native file dialog covering 30+ file formats.

​
Voice Modes

Myrm Agent supports 4 voice interaction modes, selectable in Settings > Voice:

​
Native Function Calling (Realtime & Gemini Live)

In Realtime and Gemini Live modes, tool declarations are dynamically injected from the selected Agent Profile. The AI can call tools (web search, memory, file ops, code execution, browser, kanban, background tasks) directly during the voice conversation — without server-side Agent relay. This enables sub-300ms tool-augmented voice interactions.

​
Voice Background Tasks

During a voice conversation, you can say things like “research XXX for me” and the AI will automatically dispatch it as a background task:
  • Instant dispatch: A single voice command creates a background research task without interrupting the conversation
  • Panel visibility: Dispatched tasks appear in the Activity Panel in real time, with one-click cancel or steer
  • Durable persistence: Tasks are persisted via the Kanban system with restart recovery and zombie detection
  • Completion notification: Results are automatically pushed to the originating chat when the task finishes

​
Background Task Completion Announcement

When a voice-dispatched background task finishes, your agent proactively speaks up during your next voice session — across all four voice modes symmetrically (Audio Only / Agent Bridge / OpenAI Realtime / Gemini Live):
  • Speak, walk away, hear the result: Dispatch a long task by voice, leave the computer, and the agent announces the result summary when you’re back in a voice session. No need to poll the panel or ask “is it done yet?”
  • Never interrupts current speech: If the agent is mid-sentence when the task completes, the announcement queues safely in the TTS pipeline and plays the moment it finishes speaking — queue-safe insertion, no abrupt cut-offs
  • Announcement is part of the conversation: The announcement is spoken as a natural continuation of the chat, and the chat timeline refreshes in real time so you can immediately follow up with details and the agent answers with full context
  • No duplicate announcements across windows: With multiple windows or devices open, only the active voice session announces — session-scoped deduplication, exactly once

​
Full-Duplex & Barge-in Awareness

Myrm Agent supports full-duplex voice conversations — you can interrupt the Agent at any time while it’s speaking (Barge-in). When you do:
  1. TTS playback stops immediately
  2. The AI model is automatically notified that its previous reply was interrupted — it won’t repeat what it already said, and will naturally address your new input
This makes voice conversations feel more like talking to a real person: interruptions don’t cause the Agent to lose track or repeat itself.

​
Desktop Pet Companion & Voice Orb Integration

In the Tauri desktop client, voice interaction is seamlessly unified with the floating Desktop Pet Companion:
  • Floating Pet with Halo Ripple: The pet companion features a dynamic acoustic halo wave that pulses with speaking volume, replacing heavy full-screen voice overlays.
  • Independent Alpha Mouse Click-Through: The transparent pet window stays on top with pixel-level alpha sampling, ensuring zero mouse interception on underlying IDEs or browsers.
  • Global PTT with Screen Vision Context: Holding the global push-to-talk hotkey captures the current active window screenshot, OCR text, and title, streaming multimodal context directly into the voice session without switching windows.
  • Quick Replay (F13 DoD): Double-clicking the pet companion instantly replays the last spoken agent response or interrupts active playback.
  • 100ms Smart Throttling: Audio volume levels and state transitions are throttled to 10fps over Tauri IPC, reducing CPU overhead by 83% for all-day background copilot presence.

​
Setup

  1. Navigate to Settings > Voice
  2. Select your preferred TTS and STT engines
  3. Configure language and voice preferences
  4. For custom STT endpoints, set the Base URL field (available for OpenAI and xAI providers)
  5. For Realtime mode, configure an OpenAI API key in Settings > Providers
  6. For Gemini Live mode, configure a Google API key in Settings > Providers

​
Security

All voice data is processed with end-to-end encryption (E2EE). Audio is not stored unless explicitly configured.