VAD and Handling Interruptions

This guide covers two closely related topics: using VAD (voice activity detection) and handling chat break events.

Two Operating Modes​

Device-side voice interaction supports two VAD modes, with completely different SDK call contracts:

• Server VAD (continuous conversation mode): The device continuously sends uplink audio. The cloud detects when the user stops speaking, notifies the device, and determines conversation turn boundaries.
• Device-side VAD / manual button mode: The device determines the start and end of each utterance locally (or through a button). Each utterance opens and closes its own uplink audio stream.

Server VAD vs Device-Side VAD: SDK Calls​

Server VAD​

How It Works​

The Tuya AI platform provides server VAD: the device continuously sends audio throughout the Session, and the cloud detects turn boundaries and notifies the device when the user stops speaking:

Session starts
    |
    v
tai_send_audio_start() (only once for the entire Session)
    |
    v
Continuous tai_send_audio_chunk() ====================> Cloud ASR + VAD
    |                                                       |
    |          Turn end detected (silence exceeds threshold) |
    |  Receive TAI_EVT_CHAT_BREAK <--------------------------+
    v
Update local state only (clear this turn's TTS playback and end the
"listening" indication) -- keep the uplink audio stream open
    |
    v
Wait for on_text / on_audio callbacks (play the AI response)
    |
    v
User speaks again -> reuse the same open uplink stream for the next turn

Definition of a turn: The period from when the user starts speaking until the AI starts responding. Pauses shorter than the silence threshold do not split an utterance into separate turns.

Handle the Turn-End Event​

RTC TCP Client:

void on_event(tai_ctx_t *ctx, const tai_event_msg_t *msg, void *ud)
{
    if (msg->event_type == TAI_EVT_CHAT_BREAK) {
        // Turn boundary (the cloud no longer sends TAI_EVT_SERVER_VAD; see warning above)
        // 1. Stop/clear this turn's TTS playback (the user may have interrupted,
        //    invalidating the current response)
        // 2. Update local state: end the "listening" indication, switch UI state, etc.
        stop_playback_and_flush();
        stop_listening_indication();
        // Do not call tai_send_audio_end() -- the uplink stream must remain open
        // Do not call tai_send_audio_start() either -- no restart is needed with server VAD
    }
}

When to Use tai_send_audio_end​

tai_send_audio_end() is not an interface for server-VAD mode. It belongs only to device-side VAD / manual button mode:

Enable and Configure Server VAD​

Pass chatAttributes through event_user_data_json. This string is sent to the cloud with every EventStart packet. Server VAD is enabled by default in the SDK (leaving this unset uses the built-in defaults). Override it if you need custom settings:

tai_config_t cfg = {
    // ...
    .event_user_data_json =
        "{\"sys.workflow\":\"asr-llm-tts\","
        "\"asr.enableVad\":true,"
        "\"processing.interrupt\":true}",
};

Is Device-Side VAD Necessary?​

Handle Chat Breaks​

When the user speaks again while the AI is responding, the current response needs to be interrupted.

Two Directions of Interruption​

RTC TCP Client​

Receive a server-initiated chat break (TAI_EVT_CHAT_BREAK, type=4):

TAI_EVT_CHAT_BREAK has two roles: it signals an interruption when the user speaks during the AI response; in server-VAD mode, it is also the turn-end signal (sent when the cloud detects that the user has stopped speaking; the current cloud no longer sends TAI_EVT_SERVER_VAD). Device-side handling is the same in both cases:

void on_event(tai_ctx_t *ctx, const tai_event_msg_t *msg, void *ud)
{
    if (msg->event_type == TAI_EVT_CHAT_BREAK) {
        // 1. Stop TTS playback
        audio_player_stop();
        // 2. Read the server interruption time from attr 111 and advance the cutoff
        uint64_t cutoff = parse_break_time(msg->user_data, msg->user_data_len);
        if (cutoff) audio_cutoff_ms = cutoff > audio_cutoff_ms ? cutoff : audio_cutoff_ms;
        // 3. Clear the playback buffer; only streams newer than the cutoff requeue
        audio_buffer_flush();
        // Do not stop microphone capture or call tai_send_audio_end().
        // Do not call tai_send_audio_start() to reopen the uplink stream either:
        // in server-VAD mode, the uplink Event remains open throughout.
        // If attr 111 is missing or its time is invalid, fail closed: treat the
        // in-flight stream's START as the cutoff instead of letting it play on.
    }
}

on_audio latches START's server time and filters on it:

void on_audio(tai_ctx_t *ctx, const tai_audio_msg_t *msg, void *ud)
{
    if (msg->stream_flag == TAI_STREAM_START ||
        msg->stream_flag == TAI_STREAM_ONE_SHOT)
        stream_start_ms = msg->timestamp_ms;   // server time, not local time
    // MIDDLE/END carry no new time; they reuse this turn's latched start
    if (msg->len && stream_start_ms > audio_cutoff_ms)
        audio_buffer_push(msg->data, msg->len);
}

Send a client-initiated chat break:

// The user presses a button to interrupt the AI response
tai_chat_break(ctx);   // Idempotent; safe to call multiple times

// Server-VAD mode: nothing more to do -- the uplink audio stream remains open.
// The user can simply continue speaking; no restart is needed.
// Only device-side VAD/manual mode needs to reopen the audio stream for the next utterance:
// tai_send_audio_start(ctx, TAI_AUDIO_PCM, 1, 16, 16000);

Typical Interaction Flows​

Continuous Conversation with Server VAD​

Session established (tai_connect succeeds)
    |
    v
tai_send_audio_start()  <-- Call only once for the entire Session
    |
    v
Continuous tai_send_audio_chunk() ================> Cloud ASR + VAD
    |                                                   |
    |    Receive TAI_EVT_CHAT_BREAK (once per turn) <-----+
    v
Clear this turn's TTS playback and update local state (leave uplink unchanged)
    |
    v
on_text / on_audio callbacks -> play the AI response
    |
    |   User speaks during playback (interruption, also receives CHAT_BREAK)
    v
Receive TAI_EVT_CHAT_BREAK -> stop playback, clear downlink buffer
    |                        (keep the uplink stream open)
    v
Reuse the same uplink stream for the next turn...
    |
    v
No interaction for a while (e.g. 1 minute) -> tai_disconnect()
                                           Enter standby/wake-word mode

Manual Button / Device-Side VAD Mode​

User presses the button / local VAD detects speech
    |
    v
tai_send_audio_start()  <-- Call once per utterance
    |
    v
Continuous tai_send_audio_chunk() while the button is held --> Cloud ASR
    |
    v
Button released / local end detected -> tai_send_audio_end()
    |                                  (tell the cloud to start processing)
    v
Wait for on_text / on_audio callbacks (AI response playing)
    |
    |   User presses the button again (interruption)
    v
tai_chat_break(ctx) -> stop playback -> tai_send_audio_start() to open a new stream

Considerations​

• In server-VAD mode, the turn-end signal is TAI_EVT_CHAT_BREAK: only clear this turn's downlink playback. Do not call tai_send_audio_end() at any point during the Session, and do not call tai_send_audio_start() to reopen the uplink stream.
• TAI_EVT_SERVER_VAD is the turn-end signal used by older servers. The current cloud no longer sends it; the protocol constant remains only for compatibility. Do not rely on it in new code.
• In manual button mode, when the user releases the button to stop recording, call tai_send_audio_end() directly. There is no need to wait for VAD.