Voice Devices

A voice device is a physical smart speaker that is bound to exactly one Clapilot user account on exactly one Clapilot instance. After pairing, the device authenticates with its own bearer token and talks to the existing personal chat (push-to-talk / walkie-talkie) and live-voice relay endpoints on behalf of its owner. The first hardware target is the M5Stack Atom VoiceS3R (ESP32-S3).

The server side (device registry, pairing flow, device authentication) and the firmware for the Atom VoiceS3R ship together. The firmware lives in firmware/clapilot-voice/ (ESP-IDF, see its README for build, flash and setup) and supports both conversation modes, push-to-talk and live voice, plus an optional on-device wake word on boards that can listen locally. The API is designed so that any device that can perform HTTPS requests can pair and talk to Clapilot.

UI name: Sprachgeräte (DE), Voice devices (EN), Dispositivi vocali (IT). The settings surface exists in the web app (Settings -> Sprachgeräte, route /settings/voice-devices) and in the iOS/macOS apps under the profile settings section.

Supported hardware

The registry identifies hardware by the device_kind string the firmware sends in the pairing request. Known kinds are defined once in src/lib/voice-devices/kinds.ts (VOICE_DEVICE_KINDS, with per-kind capability metadata) and each has a translated label (voiceDevices.kind.<device_kind>) that the web and Apple settings lists show instead of the raw value. An unknown or missing device_kind (older firmware sends none) pairs normally and is stored as the default kind m5stack_atom_voices3r; it never rejects the pairing.

Meta Ray-Ban Display web client

The sibling ClapilotRayban client pairs using device_kind: meta_rayban_display. The web, iOS and macOS settings lists show Meta Ray-Ban Display. Pairing, revocation, owner identity, target routing and chat storage use the existing device contracts.

The glasses web client uses Meta's system text composer (focus a text field, then tap to dictate or write) and sends reviewed text turns to POST /api/chat. It resumes the device chat on reopen, sends visible-page heartbeats, renders streamed replies and fetches reply audio with bearer authentication for explicit playback where supported. It polls reply-audio for asynchronous specialist replies. Main/team history remains available in Clapilot; the client shows replies to its new turns. Stop is supported for the exposed device/specialist session, not main/team targets whose session IDs are not exposed by heartbeat.

This client does not capture raw microphone audio or implement continuous live relay sessions. Its settings explain that continuous live audio requires a native companion integration. The system composer is the voice-entry surface even if the owner has selected live mode. The browser stores a revocable device credential locally; forgetting the browser is not server-side unpairing. No new management or business-data tools are exposed.

Capability flags per kind (hasDisplay, hasKeyboard, hasPsram, fullDuplexAudio) live next to the kind list so server and UI code can branch on hardware features without string comparisons.

Related, but separate: Home Assistant assist satellites and media-player speakers are output-only announcement targets managed by the Home Assistant module (Settings -> Module -> Home Assistant, tool home_assistant_announce). They receive text that Home Assistant renders with its own TTS; they never pair with Clapilot, hold no device token, do not record speech, and do not appear in this registry. Everything below applies only to paired Clapilot voice devices.

Binding semantics

• One device belongs to one user (voice_devices.user_id). A device row is the binding, not a visibility filter: voice devices are a personal-first module, like personal chat sessions and push-device registrations. Owner-scoped queries here are intentional and are not the workspace-global visibility bug described in the repository guidelines.
• One device belongs to one instance. The pairing response includes the instance_url the device must use for all later calls; a device that should move to another instance or another user is unpaired and paired again.
• Each device gets its own personal chat session (voice_devices.chat_session_id, titled after the device). By default the device talks in that session, so conversations from the device show up in the owner's normal chat list; the owner can read and continue them in the web or Apple chat UI. The owner can redirect a device's spoken turns elsewhere with its turn target.
• Unpairing deletes the device row and revokes its token immediately. The device's chat session stays in the owner's chat list.

Pairing flow

1. The owner opens Settings -> Sprachgeräte and presses Gerät koppeln. The app calls POST /api/voice-devices/pairing-codes and shows the returned code as XXXX-XXXX in a monospace block, with a countdown until expiry and a copy button. Generating a new code invalidates the user's previous unconsumed codes.
2. The owner enters the code in the device's setup assistant (captive-portal or companion-app flow, defined by the firmware step). The device sends the code to POST /api/voice-devices/pair together with its hardware_id (for example its MAC address), device_kind, and firmware_version.
3. The server validates the code, creates the voice_devices row bound to the code's user, creates the device chat session, and returns the device token once plus everything the device needs to operate: instance_url, chat_session_id, the owner's display name, and the relative paths of the endpoints it may call.
4. While a code is active the settings panel polls GET /api/voice-devices every three seconds; as soon as the new device appears in the list the panel shows a success line and stops polling.
5. The device stores the token in its own secure storage and starts sending heartbeats.

Pairing codes are eight characters from the alphabet ABCDEFGHJKLMNPQRSTUVWXYZ23456789 (no 0, O, 1, I), displayed as XXXX-XXXX, valid for ten minutes, and single-use. Input is normalised before lookup: uppercase, then strip separators and whitespace, so the dash and lowercase input are accepted; any character outside the alphabet makes the code invalid instead of being remapped.

Device token and allowlisted endpoints

The device token has the format clpd_<base64url 32 bytes>. It is returned once in the pairing response and stored as a SHA-256 hash (voice_devices.token_hash); only the first twelve characters (token_prefix, for example clpd_AbCdEf1) are kept in clear text so the owner can recognise a device in the settings list.

Devices send Authorization: Bearer clpd_.... Device bearer auth is accepted only on this allowlist:

Every other route returns 401 for a device token, including all user-facing /api/voice-devices/* management routes, /api/auth/*, and the rest of the business APIs. A device token never grants admin rights, never accesses other users' data, and cannot pair or unpair devices.

Heartbeat

The device calls POST /api/voice-devices/heartbeat periodically (the Atom firmware sends one every 60 s; the settings UI treats a device as online when last_seen_at is less than two minutes old). The body carries an optional firmware_version and a free-form status object (for example rssi, ip, uptime_s). The server updates last_seen_at, last_seen_ip, firmware_version, and last_status, and answers with the device identity, its chat_session_id, the owner's current device settings (voice_mode, button_actions, live_voice, wake_word_enabled, turn_target_kind), and server_time so the device can align its clock. button_actions is a top-level object { "press", "double_press", "hold", "wake_word" } (see Conversation modes); voice_mode stays for older firmware. The heartbeat is how settings changes reach the device: the firmware applies the returned button_actions (older firmware: voice_mode) and live_voice to its next button press and starts or stops on-device wake word listening according to wake_word_enabled, so a change made in Clapilot takes effect within about a minute. GET /api/voice-devices/me returns the same shape without writing anything.

Push-to-talk (walkie talkie) through POST /api/chat

For a push-to-talk turn the device sends one utterance as a normal personal-chat turn to POST /api/chat with a type: "audio" attachment (base64 data, mimeType, name, optional durationMs), exactly like the Apple chat composer's press-and-hold mic. The firmware streams the microphone while the button is held: a capture task fills a ~0.5 s ring buffer, the request goes out with chunked transfer encoding as raw audio/pcm (PCM16 mono 24 kHz, little-endian) in 32 ms base64 chunks, and the server wraps the bytes into a WAV and computes durationMs itself. So no board has to hold the whole utterance in memory (the Cardputer's free heap would allow only a second or two), there is no length cap, and the upload is finished a few hundred milliseconds after release. A tap shorter than 300 ms opens no request. The heartbeat and the voice turns share one kept-alive HTTPS connection: a TLS handshake costs about 3 s on a PSRAM-less board (longer than a short utterance) and Cloudflare closes the connection after every streamed reply, so the heartbeat keeps it warm, a heartbeat issued right after each reply rebuilds it in the background, and a button press only writes the request head (measured: 12 ms instead of 2.8 s). The spoken reply is streamed as well: a reader task fills a 32 KB ring from /api/chat/audio/[id]?format=pcm16 while the codec drains it, so replies of any length play on every board. If the chat stream ends before a slow agent has answered, the device keeps asking GET /api/voice-devices/reply-audio while the server reports the reply as pending, for up to five minutes, and shows the elapsed time on a display. The backend stores the audio as a user-owned chat asset, runs speech-to-text, injects the transcript into the agent turn, and marks the turn for an assistant audio reply. For device-authenticated turns the TTS reply is synthesized before the stream ends and announced in-stream as a clapilot.audio SSE frame carrying the attachment (id, url, mimeType, durationMs, transcript) right before [DONE]; the device then fetches /api/chat/audio/[id]?format=pcm16&sample_rate=<rate> (the Atom firmware asks for 24000, its codec rate; the route resamples to any supported rate) and streams the raw PCM16 mono bytes into its I2S output. If the frame is missing (older server, asynchronous specialist reply), the device falls back to GET /api/voice-devices/reply-audio?since=<ISO>, which returns the newest spoken reply for the device's configured target regardless of where the turn was delivered (see Turn targets), or, on servers without that route, to polling the session history for the newest assistant audio attachment.

A device-authenticated POST /api/chat without a sessionId uses the device's own chat session, so the conversation appears in the owner's chat list under the device name, unless the device's turn target says otherwise. The device may pass its chat_session_id explicitly; it cannot address sessions that do not belong to its owner. Routing fields in the device request body (roomId, group flags) are ignored: the stored turn target alone decides where a device turn goes.

Turn targets

Each paired device has a turn target (voice_devices.turn_target, JSON) that decides where its spoken turns are delivered. The owner chooses it per device; the device itself has no say and cannot change it. Four kinds exist:

Spoken replies exist for all four kinds. For the personal targets and for the main agent in a room the TTS reply is synthesized before the chat stream ends and announced with the clapilot.audio frame as described above; room replies are stored as audio assets on the team-chat message. Specialist replies run asynchronously: their audio is synthesized when the specialist turn completes, and the device picks it up through GET /api/voice-devices/reply-audio?since=<ISO>, which reports the newest reply for the target (audio: null, pending: true while the specialist is still working).

The owner changes the target under Settings -> Sprachgeräte in the device row (Ziel / Target / Destinazione), or in the same voice-device settings screen of the iOS/macOS app. The list of selectable targets comes from GET /api/voice-devices/options (targets: device session, main chat, the owner's team rooms, enabled specialists; labels are server-localized; the older GET /api/voice-devices/targets still returns the same list on its own), and the choice is saved with PATCH /api/voice-devices/{id} and { turn_target }. A target the owner may not use (room they are not a member of, disabled specialist, agent DM room) is rejected with 400. A malformed stored value (for example a team_chat target without roomId) is read as device_session.

The turn target governs walkie-talkie turns. A live conversation talks through the relay, which is a personal-session concept: live conversations are not posted into team rooms and are not pinned to a specialist. A room or specialist target therefore only takes effect for walkie-talkie turns (turn, talk_while_held, turn_after_release).

Conversation modes

A device can hold two kinds of conversation; which one starts depends on how the owner maps the device's single button and the wake word:

The mapping is stored per device as voice_devices.button_actions (jsonb), set by the owner, read by the device from the heartbeat, and never changed by the device itself:

The firmware recognizes a hold after about 350 ms and a double tap when the second tap follows within about 300 ms; while double_press is not none, a single press therefore starts about 0.3 s late.

Compatibility with voice_mode. voice_devices.voice_mode (push_to_talk | live) and the API field remain for older firmware (press-to-start-live vs hold-to-talk) and older clients. It is derived: every save of button_actions sets voice_mode = 'live' when press is live, otherwise 'push_to_talk'. A PATCH that only sends { voice_mode } replaces the gestures with the matching preset (push_to_talk → default, live → the live preset above). Migration 358 converted existing live devices to the live preset; a row whose JSON is missing or malformed is read with the preset of its voice_mode.

Live conversations additionally use a voice (voice_devices.live_voice): one of the GPT-Live relay voices (marin, cedar, quartz, ripple, vesper, willow, stone, gleam, meridian, bossa, tempo, beacon, delta, cinder), or null for the server default (marin). The device passes it as voice when it creates the relay session.

Live conversations are billed per minute of open relay session by the provider (the relay keeps the provider Realtime socket open for the whole conversation, including pauses), whereas walkie-talkie turns are billed per turn (speech-to-text, chat completion, text-to-speech). The firmware ends idle live sessions on its own, see below.

The owner sets the gestures and the voice under Settings -> Sprachgeräte in the device row, or in the same voice-device settings screen of the iOS/macOS app: Drücken / Press / Premere, Doppeltippen / Double tap / Doppio tocco, Halten / Hold / Tenere premuto (only kinds with a button, hasButton in src/lib/voice-devices/kinds.ts; meta_rayban_display has none), and Aktivierungswort startet / Wake word starts / La parola di attivazione avvia right after the wake word switch (only kinds with a wake word). Each row is a compact dropdown (Walkie-Talkie, Live-Gespräch, Nichts, Sprechen solange gehalten, Walkie-Talkie nach Loslassen); what each action does is explained in the row's (i) info popover. Stimme / Voice / Voce appears only while a visible gesture or the wake word action is live, with Standard for the server default. The voice list comes from GET /api/voice-devices/options (its modes list is still returned for older app builds but no longer used). Changes are saved with PATCH /api/voice-devices/{id} and { button_actions } (full object or a partial one such as { "hold": "live" }, merged onto the stored map) / { live_voice } (null resets the voice). Unknown gestures, unknown actions, an empty object or a non-object return 400 (api.voiceDevices.error.invalidButtonActions) and nothing is written. The device picks the change up with its next heartbeat (about a minute); a live conversation that is already running keeps its voice until it ends.

Wake word

Devices whose hardware can listen locally offer an optional wake word (voice_devices.wake_word_enabled, boolean, default false). home_assistant_voice_pe and m5stack_atom_voices3r support it (supportsWakeWord in src/lib/voice-devices/kinds.ts); the Cardputers (no PSRAM) and meta_rayban_display never show the toggle. On the Atom the ES8311 microphone runs at 24 kHz and is resampled to 16 kHz for the detector (audio_hal_read_16k, 1-2-1 low-pass + 3:2); without echo cancellation the listener only runs between turns, as on every board.

• The owner turns it on per device under Settings -> Sprachgeräte (Aktivierungswort / Wake word / Parola di attivazione, an on/off toggle; the privacy explanation is in its (i) info popover) or in the voice-device settings of the iOS/macOS app. It is saved with PATCH /api/voice-devices/{id} and { wake_word_enabled: true | false }. A non-boolean value, or true for a kind without wake word support, is rejected with 400; turning it off is always accepted.
• The device reads the flag from every heartbeat (and GET /api/voice-devices/me) as the top-level wake_word_enabled. The server sends the effective value: true only when the owner enabled it and the stored kind supports it.
• When it is on, the device listens for the wake word ("Hey Jarvis" for now) and then starts what button_actions.wake_word says (see Conversation modes): a live conversation for live, otherwise (turn) a walkie-talkie recording that ends on silence. The turn goes to the device's turn target exactly like a button press.
• Privacy: wake word detection runs entirely on the device. No audio is sent to Clapilot or a provider before the wake word fires, and the hardware mute switch disables listening. The setting is off by default and is never changed by the device or by an agent.

Firmware side

• Detector (main/wake_word.cpp): the microWakeWord v2 "Hey Jarvis" streaming model (Apache-2.0, ESPHome model collection, vendored in main/wake_models/ with its LICENSE/NOTICE) on TensorFlow Lite Micro (espressif/esp-tflite-micro) with the TensorFlow microfrontend (esphome/esp-micro-speech-features): 40 log-mel features per 10 ms from 16 kHz microphone audio (the XMOS's native output on the Voice PE, resampled from 24 kHz on the Atom), int8 quantized, probability cutoff 0.97 averaged over 5 inferences, ~1 s warm-up after every (re)start. The firmware implementation is Clapilot's own; only the Apache-2.0 libraries and model are reused.
• Custom wake word (personal builds): menuconfig Clapilot Voice → Wake word model → Custom local model embeds main/wake_models/local/custom_wake_word.tflite (gitignored) with its own name, cutoff (0–255, compared with the mean of the last window probabilities) and window. A "Hey Clapilot" model was trained with the scripts in firmware/clapilot-voice/tools/wake_word_training/ (microWakeWord, Piper English + German voices, German MLS speech negatives, look-alike phrases such as "Hey Copilot"); because the standard microWakeWord negatives are CC-BY-NC, such models stay local and out of distributed firmware. Personal build: put the model there and add CONFIG_CLAPILOT_WAKE_WORD_CUSTOM=y (plus name/cutoff) in a gitignored sdkconfig.defaults.local_wakeword appended to the Voice PE defaults.
• Listener (main/wake_listener.c): one task reads the microphone only while the wake word is enabled, the mute switch is off and no turn is running. Every turn (button or wake word) pauses it first and resumes it afterwards, so the microphone always has one reader.
• After the wake word: the device beeps and the ring turns blue. For turn the first 450 ms (the wake word's tail and the start beep) are ignored, the recording starts at the speech onset (with ~270 ms pre-roll) and ends after 1.2 s of silence once at least 0.7 s were spoken, at the 20 s cap, or without any upload when nobody speaks within 6 s (energy detector with a tracked noise floor, constants in app_main.c). For live it opens a live session that ends 20 s after the user last spoke (LIVE_WAKE_IDLE_TIMEOUT_MS), or with the button.
• Live idle timeout: activity is the user's speech as reported by the relay's server VAD (input_audio_buffer.speech_started / speech_stopped), assistant audio and running tool calls. The continuously streamed microphone audio no longer counts, which had kept every live session open until the 10-minute cap.

Announcements

Clapilot can speak on a device: announcements from the owner's agent, automations, and questions the owner can answer by voice. Every device kind has a speaker, so the setting is offered for all kinds. It is off by default (voice_devices.announcements_enabled, boolean, default false) and is only changed by the owner, never by the device or an agent.

• Toggle: Settings -> Sprachgeräte per device, Durchsagen / Announcements / Annunci, an on/off switch whose explanation is in its (i) info popover; the same toggle exists in the voice-device settings of the iOS/macOS app. Saved with PATCH /api/voice-devices/{id} and { announcements_enabled: true | false }; a non-boolean value returns 400.
• Quiet hours (optional, shown while announcements are on): Ruhezeit / Quiet hours / Ore di silenzio. On the web two compact time fields (from / until) with a clear button; the window is saved once both times are set and differ. On iOS/macOS a toggle sets a default window of 22:00–07:00 and two hour/minute pickers adjust it. Saved with PATCH and { quiet_hours: { start: "HH:MM", end: "HH:MM", timezone? } }; both clients send the device/browser time zone (Intl.DateTimeFormat().resolvedOptions().timeZone / TimeZone.current), an omitted or unknown zone becomes Europe/Berlin. { quiet_hours: null } clears it. Times outside 00:00–23:59, a missing start or end, or start == end return 400, and nothing is written. The end is exclusive and the window may wrap past midnight (22:00–07:00). Announcements are not played inside the window.
• Test announcement: while announcements are on, the row shows the push-channel state as a dot (Verbunden / Nicht verbunden on hover) and a circular megaphone button (Testdurchsage / Test announcement / Annuncio di prova). It sends POST /api/voice-devices/{id}/announce with a localized test sentence and shows the result inline below the row: success (state playing/done, or queued) or the mapped 409 codes offline (device not connected), disabled (announcements off) and quiet_hours (quiet hours active). The button is disabled while the device has no push connection.
• Layout: each device is one block: status dot, name, kind and last-seen time, an (i) with the technical details (firmware, token prefix, hardware id, chat session, pairing date), rename/unpair buttons; below it compact rows (label + (i) on the left, control on the right; two columns on wide screens): target, press, double tap, hold (kinds with a button), voice (when something starts a live conversation), wake word and wake word action (if supported), announcements, quiet hours. Explanations live in the (i) popovers; only saved marks, errors and the test result appear inline.
• Summary fields: every owner-facing device (GET /api/voice-devices, PATCH responses) carries announcements_enabled, quiet_hours ({ start, end, timezone } or null) and push_connected (whether the device currently holds its push-channel WebSocket on this server; computed live, not stored). The settings list refreshes every 30 s, so the indicator follows connects and disconnects with that delay.

Push channel

Clapilot reaches a device without waiting for its next heartbeat through one WebSocket the device keeps open for its whole runtime:

• Endpoint: GET /api/voice-devices/ws (WebSocket upgrade) with Authorization: Bearer clpd_…. Upgrades never reach Next.js: scripts/standalone-ws-server.mjs (the web container's entry) hands them to the in-process hub on globalThis.__clapilotVoiceDeviceHub (src/lib/voice-devices/push-hub.ts), which src/instrumentation.ts installs at server start with the device-token check (resolveVoiceDeviceByToken). Unknown tokens get 401, a server that has not finished booting 503. One connection per device: a reconnect replaces the older socket. The server pings every 30 s and drops a connection that stayed silent for 90 s; the firmware pings every 20 s and reconnects after 5 s. next dev does not load the wrapper, so the channel only exists in the Docker/production server.
• Server → device (JSON text frames): {"type":"hello","device_id"} after connecting; {"type":"announce","id","audio_url","chime":true} to speak a message; {"type":"ask","id","audio_url","chime":true,"answer":"turn"|"live"} to speak a question and then listen. audio_url is a chat audio asset (/api/chat/audio/{id}, synthesized with the instance's chat TTS and owned by the device's user), fetched by the firmware as PCM16 24 kHz like a spoken reply.
• Device → server: {"type":"status","id","state"} with playing, done, failed, listening, answered or no_answer, stored in voice_device_pushes.status.
• Delivery (sendVoiceDevicePush, src/lib/voice-devices/announce.ts): checks that announcements are on for the device, that it is outside its quiet hours and connected, synthesizes the text, writes a voice_device_pushes row (kind, text, answer_mode, source agent/settings_test/automation, status), pushes it, and waits up to 6 s for playing. A device in the middle of a turn queues the message (the result is then queued) and plays it right after; it never interrupts the user. Errors use the codes no_devices, not_found, ambiguous, disabled, quiet_hours, offline, invalid_text and tts_failed.
• Chat record: every push is stored as an assistant message in the device's chat session first, and its speech is attached to that message (message_meta.assistantAudio, the same as a spoken reply), so the device chat shows what was said. An announcement is stored quietly; a question is delivered like an automation result (agent history and a phone notification).
• Questions: the question is part of the device chat and the agent's history. After playing it the device listens hands-free (answer: "turn", same end-of-speech rules as a wake-word turn) or opens a live session (answer: "live"). The answer's request carries pageContext.askId; POST /api/chat and POST /api/chat/live/relay/session resolve it (same device and user, at most 30 minutes old) to the question text, which the agent receives as context ("this voice message answers your question …"). A client-supplied answer context is stripped. With the mute switch on, the device reports no_answer without listening.
• Firmware (main/device_link.c): the WebSocket client, a four-message queue, and status reports, on boards with PSRAM (Voice PE, Atom VoiceS3R; the Cardputers cannot hold a second TLS connection, so they stay offline for announcements); the main loop plays pushed messages between turns (the wake word listener is paused while it does), with a two-note chime before the speech and the ring pulsing blue.
• Sources: the agent tools voice_device_announce / voice_device_ask / voice_device_list (chat, scheduled tasks, self-acting agents with a user, Home Assistant webhook automations whose prompt asks for an announcement), and the settings test button (POST /api/voice-devices/{id}/announce). See Agent tool contracts.

Live voice through the relay

For a live conversation the device uses the same server-held relay that the Apple Watch and other native clients use. The flow implemented by the Atom firmware (firmware/clapilot-voice/main/live_session.c):

1. On the starting button press the device calls POST /api/chat/live/relay/session with its device token, voice (when the owner set one) and clientContext: { routePath: "/voice-device", pageContext: { scope: "voice-device", platform: "esp32", deviceKind } }. The server creates the provider Realtime session under the owner's identity and returns transport: "clapilot_relay", relay_session_id, the model and voice in use, and PCM 24 kHz audio metadata.
2. The response also carries ws_path (/api/chat/live/relay/ws) and a one-time ws_ticket. The device opens wss://<instance>/api/chat/live/relay/ws?session=<relay_session_id>&ticket=<ws_ticket>; that single socket carries the conversation in both directions. Next.js route handlers cannot answer an HTTP upgrade, so the handler lives in scripts/standalone-ws-server.mjs, the container entry that attaches an upgrade listener to the standalone server and then runs it unchanged. It authenticates with the ticket (constant-time compare) of a relay session that an authenticated request created, so no auth logic is duplicated, and it forwards frames verbatim between device and relay. When the server offers no ticket (older instance) or the socket cannot be opened, the device falls back to the POST/SSE transport described below.
3. The device streams its microphone as mono 24 kHz PCM16 in 320 ms blocks. Over the WebSocket it sends them as binary frames and the bridge wraps each one into an input_audio_buffer.append event with base64 on the server; that removes a third of the uplink bytes and the encoding cost from the device, which is what lets it keep up with real time. Text frames stay available and are forwarded verbatim, and the fallback path base64-encodes on the device and posts to POST /api/chat/live/relay/[sessionId]/input. Capture and transport run in separate tasks with a bounded queue, so a slow network never stalls recording; the oldest block is dropped when the queue is full. Turn detection runs on the server (provider VAD); the device sends no commit events.
4. It reads relay events from the same WebSocket (fallback: the SSE stream GET /api/chat/live/relay/[sessionId]/events) and plays response.output_audio.delta frames (base64 PCM16 at 24 kHz) through a playback buffer that primes with 300 ms before a burst so short gaps between deltas do not click. input_audio_buffer.speech_started clears whatever is still queued so a reply does not keep playing over a new question; transcript events are only logged. GPT-Live keeps its output channel open and streams silence between replies, so silent deltas (peak below about -36 dBFS) are dropped unless a reply is still queued; otherwise the idle silence would count as "assistant speaking" and keep the microphone muted.
5. Function calls (response.function_call_arguments.done or a function_call item in response.output_item.done) are executed by the device through POST /api/chat/live/tools with toolName, arguments, the device's chat_session_id as sessionId, and the same clientContext as above. The tool output is answered back into the relay as a conversation.item.create / function_call_output event followed by response.create, exactly as the Apple clients do; the existing live tool contract applies unchanged, see Agent Tool Contracts.
6. The conversation ends on the next button press, after 60 s without audio in either direction (only counted while the assistant is not speaking), after a hard cap of 10 minutes, or when the relay closes the session (*.closed event, or 404/409 on the input route). The device then calls DELETE /api/chat/live/relay/[sessionId] and returns to idle.

Relay sessions created by a device token use a stricter server VAD on the GA path (threshold: 0.65, silence_duration_ms: 800) because a room microphone without echo cancellation otherwise starts a turn on breathing, and each such turn mutes the microphone while the model answers it. Browser and Apple clients keep the default sensitivity. The GPT-Live path (gpt-live-1) does its own turn detection and takes no threshold.

A reconnecting WebSocket client deliberately does not get buffered audio replayed (the SSE transport still does): a device that drops and reconnects mid-conversation would otherwise play seconds of stale speech over the live reply, and keep its own microphone muted while doing so. Non-audio events are replayed so session and tool state stay consistent.

The bridge pings a connected device every 20 s but treats any inbound frame as a sign of life (audio, text, the client's own pings, pongs); only a connection that stays completely silent for three intervals is dropped. A device whose pong arrives late is therefore not disconnected mid-reply. The web container log ([live-ws]) records one line per conversational milestone for device sessions (speech started, user: …, assistant: …, response completed, upstream errors) and, on disconnect, the close code plus how many frames and seconds of audio went up, how many pings/pongs the device sent, and a type×count summary of the events that went down. On the GPT-Live path the transcript lines come from clapilot.live.transcript.segment and tool delegations from response.function_call_arguments.done; that path has no speech-started/response-done events. This is enough to tell "the provider never heard speech" from "the reply was generated but not played" without a serial console.

Loudness on the Atom is set in firmware/clapilot-voice/main/app_config.h (AUDIO_OUTPUT_VOLUME_PERCENT, AUDIO_PLAYBACK_GAIN through a soft limiter). The level is bounded by the USB power budget, not by the codec: raising the codec to 0 dB with a 1.4x-2.5x software gain made the NS4150B pull the supply rail down, beeps and replies stuttered, the button read spurious presses and the chip reset. The codec therefore stays at the 85 % that runs stable from a hub port, and loudness comes from a compressor in audio_hal_play: samples are multiplied by AUDIO_PLAYBACK_GAIN (2.0) and everything above AUDIO_COMPRESSOR_KNEE (8000) is squeezed 4:1, so speech comes out about 6 dB louder while a full-scale sample still ends below the old peak level and the supply sees no higher current spikes. The boot banner and the ready: line log the last reset reason, so a BROWNOUT is visible in the console even though the USB console re-enumerates on a reset.

The ES8311 codec path on the Atom has no acoustic echo cancellation, so the firmware keeps the microphone muted while the speaker plays and for 250 ms afterwards; otherwise the server VAD would hear the reply and answer itself. Barge-in by voice is therefore not possible yet: the user waits for the reply to finish (or presses the button to end the conversation). This is a device-side limitation, not a relay one; devices with echo cancellation can leave the microphone open.

See GPT-Live voice transport for the relay routes and event types.

Firmware builds per board

The firmware in firmware/clapilot-voice/ is one code base with a board layer (main/board.h, main/boards/*.h) selected through idf.py menuconfig → Clapilot Voice → Target board, or through the per-board sdkconfig.defaults.* overlay. Each board has its own build directory and generated sdkconfig, so the three targets can be built side by side:

Flash with idf.py -B <build dir> -p <port> flash. Pairing data in NVS survives re-flashing.

Setup after a failure. When Wi-Fi does not connect (wifi_failed), pairing cannot reach the instance (pair_failed), the code is rejected (invalid_code) or the token is revoked (unauthorized), the device restarts into setup (portal or on-device form) and shows the reason. If the saved configuration is still complete (Wi-Fi, instance, token or pairing code), setup is a timed retry: when no phone joins the setup hotspot (or no key is pressed on keyboard boards) for 5 minutes (SETUP_RETRY_IDLE_MS), the device restarts and tries the saved settings again, so it recovers by itself after a router or instance outage.

Cardputer behaviour

• Display (LVGL on the ST7789): header with device name, mode (talk/live) and Wi-Fi signal; a scrolling conversation area with You: and Agent: lines (push-to-talk turns show (voice message) for the user side, live sessions show both transcripts as the relay delivers them); the typed line above the status; a status line (Listening…, Thinking…, Speaking…, Ready).
• Keys: hold Space to talk (push-to-talk) or press it to start/stop a live conversation, exactly like the Atom's button; G0 on the side does the same. Any other printable key starts a typed message; Enter sends it as a text turn to the same target (POST /api/chat with pageContext.input = "keyboard"), the reply is shown and spoken; Backspace edits, Esc (Fn+`) or Del (Fn+Backspace) clears.
• Setup: with nothing configured (or after wifi_failed, invalid_code, unauthorized), the device shows a five-step form instead of opening the portal: Wi-Fi SSID, password, instance URL (prefilled with https://app.clapilot.com), pairing code, device name. Enter moves on, Esc goes back, the last Enter saves and restarts into the normal pairing flow. The captive portal is not started on keyboard boards.
• Memory: neither Cardputer has PSRAM (the Adv is a StampS3A with an ESP32-S3FN8 too). Push-to-talk streams the microphone as it records (see above), the live playback ring holds ~1.3 s and the WebSocket frame buffer is 8 KB; both modes run within the internal RAM.
• Loudness: AUDIO_PLAYBACK_GAIN is 4.0 on the Cardputer (the NS4168 path is quiet at unity) and the PDM microphone is scaled by AUDIO_MIC_SOFT_GAIN (8); both live in main/boards/cardputer.h and are the first thing to tune on hardware.
• Audio on the Adv: the ES8311's left I2S slot carries the microphone (the right slot showed a constant ±15480 waveform on this unit, so the slot is fixed rather than picked by level), the microphone gain is 18 dB (30 dB clipped at 0 dBFS), and the DAC volume register is raised by +6 dB on top of the 100 % codec level because the speaker path is far quieter than the Atom's; a boot-time acoustic self-test logs the microphone level with the speaker silent versus during a 1 kHz tone.
• Telling the variants apart: both look alike; the Adv has an I2C bus on GPIO 8/9 with the ES8311 (0x18), the TCA8418 (0x34) and an IMU (0x69), the classic drives its matrix through those pins. If keys do nothing after flashing, the other build is the right one. Verified on hardware 2026-09-21: Cardputer-Adv (display, keyboard, ES8311).
• Display strings are English only for now (device UI, not covered by the app localization system).

Home Assistant Voice PE behaviour

The board layer reuses the XMOS and AIC3204 bring-up of the Apache-2.0 Muse Gadget SDK (esp32/main/voice_board.c), which follows the vendor ESPHome configuration.

• Flashing: this replaces ESPHome, so the device leaves Home Assistant Assist until ESPHome is installed again (from the ESPHome project page above). The XMOS keeps its own firmware in its own flash and is only reset, never written. For a clean start run idf.py -B build-ha-voice-pe -p <port> erase-flash flash; if the port does not answer, hold the centre button while plugging in USB-C (download mode).
• Centre button: the talk button (hold for push-to-talk, press to start/stop live). It is also the boot strapping pin, so the factory reset (hold for 6 s) starts after the ring shows the white boot spinner, not while plugging in.
• LED ring (main/status_light.c): white spinner = starting, lavender breathing = setup hotspot open, blue spinner = connecting, steady blue = listening, two lavender comets = waiting for the agent, blue pulse = speaking, two red blinks = error, dim red = mute switch on, otherwise dark.
• Dial: volume in 5 % steps (AIC3204 digital volume), shown on the ring and stored in NVS.
• Mute switch: while it is on, a button press only plays the error tone; nothing is recorded.
• Audio: the left XMOS output slot (full chain up to AGC; the boot log prints the stages of both slots) is resampled 16 → 24 kHz; replies are resampled 24 → 48 kHz stereo. The amplifier switches off after 3 s without audio. Because the XMOS removes the echo, live mode does not mute the microphone while the agent speaks; input_audio_buffer.speech_started clears the playback buffer so the user can interrupt.

Unpairing

DELETE /api/voice-devices/{id} (owner, cookie auth) hard-deletes the device row. The device receives 401 on its next request and must be paired again to be used. Renaming (PATCH /api/voice-devices/{id} with { name }, 1–80 characters) only changes the display name and the title shown in the owner's chat list from then on; the token and binding are unchanged. The same route accepts { turn_target }, { button_actions }, { voice_mode } (legacy), { live_voice }, { wake_word_enabled }, { announcements_enabled } and { quiet_hours } (any combination, optionally together with name) to change where the device's turns go, how it converses and whether Clapilot may speak on it, see Turn targets, Conversation modes, Wake word and Announcements.

Security notes

• The device token is shown once at pairing and stored only as a SHA-256 hash. Clapilot cannot display it again; a lost token means unpair and pair again.
• Pairing codes are hashed at rest (code_hash), expire after ten minutes, are single-use, and are invalidated when the user generates a new code.
• POST /api/voice-devices/pair is public (no cookie) and rate limited; throttled calls receive 429 with Retry-After. Unknown, expired, and already-used codes all return the same 400 with code: "invalid_code" so nothing about code state is leaked.
• Device tokens are rejected outside the allowlist above. The management routes (GET /api/voice-devices, pairing-code creation, options/target list, rename, target/button actions/mode/voice/wake word/announcements/quiet hours change, unpair) require the owner's normal clapilot_session cookie.
• Turn targets are validated against the owner's own access (room membership, enabled specialists) when set and are never taken from the device request; a device token cannot post into rooms its owner cannot see.
• last_seen_ip and last_status are visible only to the owning user in their device list.
• There is deliberately no chat or live-voice agent tool for pairing or unpairing devices; issuing a device credential is a settings-only workflow (same policy as API keys).

Data model

Migration db/migrations/319_voice_devices.sql adds:

• voice_device_pairing_codes: id, user_id (FK users), code_hash (unique), expires_at, consumed_at, device_id (nullable), created_at
• voice_devices: id, user_id (FK users, owner/binding), name, device_kind (default m5stack_atom_voices3r; see Supported hardware), hardware_id, firmware_version, token_hash (unique), token_prefix, chat_session_id (FK chat_sessions, nullable), last_seen_at, last_seen_ip, last_status (jsonb), created_at, updated_at

Migration db/migrations/320_voice_device_turn_targets.sql adds:

• voice_devices.turn_target (jsonb, not null, default {"kind":"device_session"}), see Turn targets
• chat_audio_assets.group_message_id (FK chat_group_messages, nullable, cascade delete) so spoken replies for room turns can belong to a team-chat message; chat_message_id becomes nullable and a check constraint requires at least one of the two references to be set

Migration db/migrations/321_voice_device_voice_mode.sql adds:

• voice_devices.voice_mode (text, not null, default 'push_to_talk', check push_to_talk | live)
• voice_devices.live_voice (text, nullable; null = server default voice), see Conversation modes

Migration db/migrations/358_voice_device_button_actions.sql adds:

• voice_devices.button_actions (jsonb, not null, default {"press":"turn","double_press":"live","hold":"talk_while_held","wake_word":"turn"}) and converts rows with voice_mode = 'live' to {"press":"live","double_press":"turn","hold":"talk_while_held","wake_word":"live"}, see Conversation modes

Migration db/migrations/356_voice_device_wake_word.sql adds:

• voice_devices.wake_word_enabled (boolean, not null, default false), see Wake word

Migration db/migrations/357_voice_device_announcements.sql adds:

• voice_devices.announcements_enabled (boolean, not null, default false) and voice_devices.quiet_hours_start, quiet_hours_end, quiet_hours_timezone (text, nullable; "HH:MM" and an IANA zone), see Announcements
• voice_device_pushes: one row per pushed announcement or question (device_id, user_id, kind, text, answer_mode, source, status, audio_asset_id, chat_session_id, timestamps)

API

The full request/response contract for all user-facing and device-facing routes is in the API Reference.

When Live Voice uses a connected Codex/ChatGPT subscription provider, device options offer arbor, breeze, cove, ember, juniper, maple, sol, spruce, and vale. The same PCM relay runs gpt-live-1-codex through server-side WebRTC; the device keeps its existing owner-scoped token and tool permissions. Stored voices from another provider normalize to the selected model default. Subscription login failure does not switch to API billing.