Voice Devices

A voice device is a physical smart speaker that is bound to exactly one Clapilot user account on exactly one Clapilot instance. After pairing, the device authenticates with its own bearer token and talks to the existing personal chat (push-to-talk / walkie-talkie) and live-voice relay endpoints on behalf of its owner. The first hardware target is the M5Stack Atom VoiceS3R (ESP32-S3).

The server side (device registry, pairing flow, device authentication) and the firmware for the Atom VoiceS3R ship together. The firmware lives in firmware/clapilot-voice/ (ESP-IDF, see its README for build, flash and setup) and supports both conversation modes, push-to-talk and live voice; a wake word is a later step. The API is designed so that any device that can perform HTTPS requests can pair and talk to Clapilot.

UI name: Sprachgeräte (DE), Voice devices (EN), Dispositivi vocali (IT). The settings surface exists in the web app (Settings -> Sprachgeräte, route /settings/voice-devices) and in the iOS/macOS apps under the profile settings section.

Related, but separate: Home Assistant assist satellites and media-player speakers are output-only announcement targets managed by the Home Assistant module (Settings -> Module -> Home Assistant, tool home_assistant_announce). They receive text that Home Assistant renders with its own TTS; they never pair with Clapilot, hold no device token, do not record speech, and do not appear in this registry. Everything below applies only to paired Clapilot voice devices.

Binding semantics

• One device belongs to one user (voice_devices.user_id). A device row is the binding, not a visibility filter: voice devices are a personal-first module, like personal chat sessions and push-device registrations. Owner-scoped queries here are intentional and are not the workspace-global visibility bug described in the repository guidelines.
• One device belongs to one instance. The pairing response includes the instance_url the device must use for all later calls; a device that should move to another instance or another user is unpaired and paired again.
• Each device gets its own personal chat session (voice_devices.chat_session_id, titled after the device). By default the device talks in that session, so conversations from the device show up in the owner's normal chat list; the owner can read and continue them in the web or Apple chat UI. The owner can redirect a device's spoken turns elsewhere with its turn target.
• Unpairing deletes the device row and revokes its token immediately. The device's chat session stays in the owner's chat list.

Pairing flow

1. The owner opens Settings -> Sprachgeräte and presses Gerät koppeln. The app calls POST /api/voice-devices/pairing-codes and shows the returned code as XXXX-XXXX in a monospace block, with a countdown until expiry and a copy button. Generating a new code invalidates the user's previous unconsumed codes.
2. The owner enters the code in the device's setup assistant (captive-portal or companion-app flow, defined by the firmware step). The device sends the code to POST /api/voice-devices/pair together with its hardware_id (for example its MAC address), device_kind, and firmware_version.
3. The server validates the code, creates the voice_devices row bound to the code's user, creates the device chat session, and returns the device token once plus everything the device needs to operate: instance_url, chat_session_id, the owner's display name, and the relative paths of the endpoints it may call.
4. While a code is active the settings panel polls GET /api/voice-devices every three seconds; as soon as the new device appears in the list the panel shows a success line and stops polling.
5. The device stores the token in its own secure storage and starts sending heartbeats.

Pairing codes are eight characters from the alphabet ABCDEFGHJKLMNPQRSTUVWXYZ23456789 (no 0, O, 1, I), displayed as XXXX-XXXX, valid for ten minutes, and single-use. Input is normalised before lookup: uppercase, then strip separators and whitespace, so the dash and lowercase input are accepted; any character outside the alphabet makes the code invalid instead of being remapped.

Device token and allowlisted endpoints

The device token has the format clpd_<base64url 32 bytes>. It is returned once in the pairing response and stored as a SHA-256 hash (voice_devices.token_hash); only the first twelve characters (token_prefix, for example clpd_AbCdEf1) are kept in clear text so the owner can recognise a device in the settings list.

Devices send Authorization: Bearer clpd_.... Device bearer auth is accepted only on this allowlist:

Every other route returns 401 for a device token, including all user-facing /api/voice-devices/* management routes, /api/auth/*, and the rest of the business APIs. A device token never grants admin rights, never accesses other users' data, and cannot pair or unpair devices.

Heartbeat

The device calls POST /api/voice-devices/heartbeat periodically (the Atom firmware sends one every 60 s; the settings UI treats a device as online when last_seen_at is less than two minutes old). The body carries an optional firmware_version and a free-form status object (for example rssi, ip, uptime_s). The server updates last_seen_at, last_seen_ip, firmware_version, and last_status, and answers with the device identity, its chat_session_id, the owner's current device settings (voice_mode, live_voice, turn_target_kind), and server_time so the device can align its clock. The heartbeat is how settings changes reach the device: the firmware applies the returned voice_mode and live_voice to its next button press, so a mode switch made in Clapilot takes effect within about a minute. GET /api/voice-devices/me returns the same shape without writing anything.

Push-to-talk (walkie talkie) through POST /api/chat

For a push-to-talk turn the device records one utterance and sends it as a normal personal-chat turn to POST /api/chat with a type: "audio" attachment (base64 data, mimeType, name, optional durationMs), exactly like the Apple chat composer's press-and-hold mic. The backend stores the audio as a user-owned chat asset, runs speech-to-text, injects the transcript into the agent turn, and marks the turn for an assistant audio reply. For device-authenticated turns the TTS reply is synthesized before the stream ends and announced in-stream as a clapilot.audio SSE frame carrying the attachment (id, url, mimeType, durationMs, transcript) right before [DONE]; the device then fetches /api/chat/audio/[id]?format=pcm16&sample_rate=<rate> (the Atom firmware asks for 24000, its codec rate; the route resamples to any supported rate) and streams the raw PCM16 mono bytes into its I2S output. If the frame is missing (older server, asynchronous specialist reply), the device falls back to GET /api/voice-devices/reply-audio?since=<ISO>, which returns the newest spoken reply for the device's configured target regardless of where the turn was delivered (see Turn targets), or, on servers without that route, to polling the session history for the newest assistant audio attachment.

A device-authenticated POST /api/chat without a sessionId uses the device's own chat session, so the conversation appears in the owner's chat list under the device name, unless the device's turn target says otherwise. The device may pass its chat_session_id explicitly; it cannot address sessions that do not belong to its owner. Routing fields in the device request body (roomId, group flags) are ignored: the stored turn target alone decides where a device turn goes.

Turn targets

Each paired device has a turn target (voice_devices.turn_target, JSON) that decides where its spoken turns are delivered. The owner chooses it per device; the device itself has no say and cannot change it. Four kinds exist:

Spoken replies exist for all four kinds. For the personal targets and for the main agent in a room the TTS reply is synthesized before the chat stream ends and announced with the clapilot.audio frame as described above; room replies are stored as audio assets on the team-chat message. Specialist replies run asynchronously: their audio is synthesized when the specialist turn completes, and the device picks it up through GET /api/voice-devices/reply-audio?since=<ISO>, which reports the newest reply for the target (audio: null, pending: true while the specialist is still working).

The owner changes the target under Settings -> Sprachgeräte in the device row (Ziel / Target / Destinazione), or in the same voice-device settings screen of the iOS/macOS app. The list of selectable targets comes from GET /api/voice-devices/options (targets: device session, main chat, the owner's team rooms, enabled specialists; labels are server-localized; the older GET /api/voice-devices/targets still returns the same list on its own), and the choice is saved with PATCH /api/voice-devices/{id} and { turn_target }. A target the owner may not use (room they are not a member of, disabled specialist, agent DM room) is rejected with 400. A malformed stored value (for example a team_chat target without roomId) is read as device_session.

The turn target governs push-to-talk turns. A device in live mode talks through the relay, which is a personal-session concept: live conversations are not posted into team rooms and are not pinned to a specialist. A room or specialist target therefore only takes effect while the device is in push-to-talk mode.

Conversation modes

Each device has a conversation mode (voice_devices.voice_mode) that the owner sets per device; the device reads it from the heartbeat response and cannot change it itself.

Live mode additionally has a voice (voice_devices.live_voice): one of the GPT-Live relay voices (marin, cedar, quartz, ripple, vesper, willow, stone, gleam, meridian, bossa, tempo, beacon, delta, cinder), or null for the server default (marin). The device passes it as voice when it creates the relay session.

Live mode is billed per minute of open relay session by the provider (the relay keeps the provider Realtime socket open for the whole conversation, including pauses), whereas push-to-talk is billed per turn (speech-to-text, chat completion, text-to-speech). The firmware ends idle live sessions on its own, see below.

The owner sets mode and voice under Settings -> Sprachgeräte in the device row (Modus / Mode / Modalità next to the target; Stimme / Voice / Voce appears only while the mode is live, with Standard for the server default), or in the same voice-device settings screen of the iOS/macOS app. Mode labels, descriptions and the voice list come from GET /api/voice-devices/options, and the choice is saved with PATCH /api/voice-devices/{id} and { voice_mode } / { live_voice } (null resets the voice). Invalid values are rejected with 400. The device picks the change up with its next heartbeat (about a minute); a live conversation that is already running keeps its voice until it ends.

Live voice through the relay

For a live conversation the device uses the same server-held relay that the Apple Watch and other native clients use. The flow implemented by the Atom firmware (firmware/clapilot-voice/main/live_session.c):

1. On the starting button press the device calls POST /api/chat/live/relay/session with its device token, voice (when the owner set one) and clientContext: { routePath: "/voice-device", pageContext: { scope: "voice-device", platform: "esp32", deviceKind } }. The server creates the provider Realtime session under the owner's identity and returns transport: "clapilot_relay", relay_session_id, the model and voice in use, and PCM 24 kHz audio metadata.
2. The response also carries ws_path (/api/chat/live/relay/ws) and a one-time ws_ticket. The device opens wss://<instance>/api/chat/live/relay/ws?session=<relay_session_id>&ticket=<ws_ticket>; that single socket carries the conversation in both directions. Next.js route handlers cannot answer an HTTP upgrade, so the handler lives in scripts/standalone-ws-server.mjs, the container entry that attaches an upgrade listener to the standalone server and then runs it unchanged. It authenticates with the ticket (constant-time compare) of a relay session that an authenticated request created, so no auth logic is duplicated, and it forwards frames verbatim between device and relay. When the server offers no ticket (older instance) or the socket cannot be opened, the device falls back to the POST/SSE transport described below.
3. The device streams its microphone as mono 24 kHz PCM16 in 320 ms blocks. Over the WebSocket it sends them as binary frames and the bridge wraps each one into an input_audio_buffer.append event with base64 on the server; that removes a third of the uplink bytes and the encoding cost from the device, which is what lets it keep up with real time. Text frames stay available and are forwarded verbatim, and the fallback path base64-encodes on the device and posts to POST /api/chat/live/relay/[sessionId]/input. Capture and transport run in separate tasks with a bounded queue, so a slow network never stalls recording; the oldest block is dropped when the queue is full. Turn detection runs on the server (provider VAD); the device sends no commit events.
4. It reads relay events from the same WebSocket (fallback: the SSE stream GET /api/chat/live/relay/[sessionId]/events) and plays response.output_audio.delta frames (base64 PCM16 at 24 kHz) through a playback buffer that primes with 300 ms before a burst so short gaps between deltas do not click. input_audio_buffer.speech_started clears whatever is still queued so a reply does not keep playing over a new question; transcript events are only logged.
5. Function calls (response.function_call_arguments.done or a function_call item in response.output_item.done) are executed by the device through POST /api/chat/live/tools with toolName, arguments, the device's chat_session_id as sessionId, and the same clientContext as above. The tool output is answered back into the relay as a conversation.item.create / function_call_output event followed by response.create, exactly as the Apple clients do; the existing live tool contract applies unchanged, see Agent Tool Contracts.
6. The conversation ends on the next button press, after 60 s without audio in either direction (only counted while the assistant is not speaking), after a hard cap of 10 minutes, or when the relay closes the session (*.closed event, or 404/409 on the input route). The device then calls DELETE /api/chat/live/relay/[sessionId] and returns to idle.

Relay sessions created by a device token use a stricter server VAD on the GA path (threshold: 0.65, silence_duration_ms: 800) because a room microphone without echo cancellation otherwise starts a turn on breathing, and each such turn mutes the microphone while the model answers it. Browser and Apple clients keep the default sensitivity. The GPT-Live path (gpt-live-1) does its own turn detection and takes no threshold.

A reconnecting WebSocket client deliberately does not get buffered audio replayed (the SSE transport still does): a device that drops and reconnects mid-conversation would otherwise play seconds of stale speech over the live reply, and keep its own microphone muted while doing so. Non-audio events are replayed so session and tool state stay consistent.

The bridge pings a connected device every 20 s but treats any inbound frame as a sign of life (audio, text, the client's own pings, pongs); only a connection that stays completely silent for three intervals is dropped. A device whose pong arrives late is therefore not disconnected mid-reply. The web container log ([live-ws]) records one line per conversational milestone for device sessions (speech started, user: …, assistant: …, response completed, upstream errors) and, on disconnect, the close code plus how many frames and seconds of audio went up, how many pings/pongs the device sent, and a type×count summary of the events that went down. On the GPT-Live path the transcript lines come from clapilot.live.transcript.segment and tool delegations from response.function_call_arguments.done; that path has no speech-started/response-done events. This is enough to tell "the provider never heard speech" from "the reply was generated but not played" without a serial console.

The ES8311 codec path on the Atom has no acoustic echo cancellation, so the firmware keeps the microphone muted while the speaker plays and for 250 ms afterwards; otherwise the server VAD would hear the reply and answer itself. Barge-in by voice is therefore not possible yet: the user waits for the reply to finish (or presses the button to end the conversation). This is a device-side limitation, not a relay one; devices with echo cancellation can leave the microphone open.

See GPT-Live voice transport for the relay routes and event types.

Unpairing

DELETE /api/voice-devices/{id} (owner, cookie auth) hard-deletes the device row. The device receives 401 on its next request and must be paired again to be used. Renaming (PATCH /api/voice-devices/{id} with { name }, 1–80 characters) only changes the display name and the title shown in the owner's chat list from then on; the token and binding are unchanged. The same route accepts { turn_target }, { voice_mode } and { live_voice } (any combination, optionally together with name) to change where the device's turns go and how it converses, see Turn targets and Conversation modes.

Security notes

• The device token is shown once at pairing and stored only as a SHA-256 hash. Clapilot cannot display it again; a lost token means unpair and pair again.
• Pairing codes are hashed at rest (code_hash), expire after ten minutes, are single-use, and are invalidated when the user generates a new code.
• POST /api/voice-devices/pair is public (no cookie) and rate limited; throttled calls receive 429 with Retry-After. Unknown, expired, and already-used codes all return the same 400 with code: "invalid_code" so nothing about code state is leaked.
• Device tokens are rejected outside the allowlist above. The management routes (GET /api/voice-devices, pairing-code creation, options/target list, rename, target/mode/voice change, unpair) require the owner's normal clapilot_session cookie.
• Turn targets are validated against the owner's own access (room membership, enabled specialists) when set and are never taken from the device request; a device token cannot post into rooms its owner cannot see.
• last_seen_ip and last_status are visible only to the owning user in their device list.
• There is deliberately no chat or live-voice agent tool for pairing or unpairing devices; issuing a device credential is a settings-only workflow (same policy as API keys).

Data model

Migration db/migrations/319_voice_devices.sql adds:

• voice_device_pairing_codes: id, user_id (FK users), code_hash (unique), expires_at, consumed_at, device_id (nullable), created_at
• voice_devices: id, user_id (FK users, owner/binding), name, device_kind (default m5stack_atom_voices3r), hardware_id, firmware_version, token_hash (unique), token_prefix, chat_session_id (FK chat_sessions, nullable), last_seen_at, last_seen_ip, last_status (jsonb), created_at, updated_at

Migration db/migrations/320_voice_device_turn_targets.sql adds:

• voice_devices.turn_target (jsonb, not null, default {"kind":"device_session"}), see Turn targets
• chat_audio_assets.group_message_id (FK chat_group_messages, nullable, cascade delete) so spoken replies for room turns can belong to a team-chat message; chat_message_id becomes nullable and a check constraint requires at least one of the two references to be set

Migration db/migrations/321_voice_device_voice_mode.sql adds:

• voice_devices.voice_mode (text, not null, default 'push_to_talk', check push_to_talk | live)
• voice_devices.live_voice (text, nullable; null = server default voice), see Conversation modes

API

The full request/response contract for all user-facing and device-facing routes is in the API Reference.