RTC TCP Client SDK Reference

1. Overview​

RTC TCP Client (API prefix tai_*) is the TCP implementation of tRTC (Tuya's proprietary RTC protocol). It is provided as source code and supports cross-platform porting through the PAL (Platform Abstraction Layer), making it suitable for environments such as POSIX and FreeRTOS (ESP-IDF).

Header file: tuya_ai.h (single include)

Features​

• Simple API: Typed send functions (tai_send_text, tai_send_audio_*, and tai_send_image) eliminate the need to manually assemble data structures
• Background receive thread: Automatically starts a background thread after tai_connect to handle receiving and Keepalive
• Callback-driven: Receives data through on_audio, on_text, on_image, on_event, and on_disconnect
• No external dependencies: The user-provided PAL only needs to supply raw TCP and platform primitives; the SDK handles TLS and cryptography internally through bundled mbedTLS

2. Constant Definitions​

2.1 Packet Types​

2.2 Stream Flags​

2.3 Event Types​

2.4 Client Types​

2.5 Audio Codecs​

2.6 Image Formats​

2.7 Image Payload Types​

2.8 Sign Levels​

2.9 Data IDs​

2.10 Return Codes​

3. Configuration (tai_config_t)​

Populate this structure before calling tai_ctx_init. All pointer fields must remain valid for the lifetime of the context.

3.1 Server Configuration​

3.2 Identity Configuration​

3.3 Session Options​

3.4 Business Identifiers​

3.5 Security Configuration​

3.6 Keepalive Configuration​

3.7 Test Configuration​

3.8 Platform Adaptation​

3.9 Callbacks​

All callbacks are invoked on the background receive thread.

Callback signatures:

Each callback receives a single pointer to a const message structure plus user_data (instead of the former multi-argument form).

void (*on_audio)     (tai_ctx_t *ctx, const tai_audio_msg_t      *msg, void *user_data);
void (*on_text)      (tai_ctx_t *ctx, const tai_text_msg_t       *msg, void *user_data);
void (*on_image)     (tai_ctx_t *ctx, const tai_image_msg_t      *msg, void *user_data);
void (*on_event)     (tai_ctx_t *ctx, const tai_event_msg_t      *msg, void *user_data);
void (*on_disconnect)(tai_ctx_t *ctx, const tai_disconnect_msg_t *msg, void *user_data);
int  (*on_flow_control)(tai_ctx_t *ctx, void *user_data);

Receive Backpressure (on_flow_control)​

• The worker calls the hook before each read and between complete Frames. A conforming server never consults it during the connect handshake (the ack precedes media); only a server coalescing media ahead of the ack could. The hook must not block; the application synchronizes shared queue state.
• Returning 0 pauses both reads and parsing, stalling all inbound traffic including ChatBreak, ASR text, Pong, and EOF detection. Resume processes complete buffered Frames before reading again; partial input returns to bounded blocking receive.
• An intentional pause suspends receive-liveness timeout accounting, and resume grants a fresh ping_timeout_ms budget. Ping and stop requests remain active; Ping send failure still disconnects.
• Admission is also checked before each codec-frame callback within an Audio Packet. A mid-Packet pause retains that Packet's remaining bytes without copying; resume delivers the remaining frames before any later Packet. Applications discard obsolete audio inside on_audio by comparing the server timestamp (msg->timestamp_ms), not by mutating SDK receive state.

Backpressure is rechecked every AGENTIC_KIT_TAI_FLOW_CONTROL_POLL_MS (50 ms by default), and sustained receive traffic yields CPU for AGENTIC_KIT_TAI_WORKER_YIELD_MS (10 ms by default).

pal_t adds a mandatory sleep_ms callback. It must sleep without busy-waiting for at least the requested milliseconds, treat 0 as a no-op, and not depend on socket readability. Recompile all custom PALs and consumers against the updated header.

Received Message Structures​

tai_audio_msg_t (audio callback):

tai_text_msg_t (text callback):

tai_image_msg_t (image callback):

tai_event_msg_t (Event callback):

tai_disconnect_msg_t (disconnect callback):

Disconnect Reasons (reason)​

detail (when reason==TRANSPORT)​

detail (when reason==PROTOCOL)​

4. Lifecycle API​

tai_ctx_size​

size_t tai_ctx_size(void);

Returns the amount of memory required by the context. The caller must allocate at least this many bytes and pass the memory to tai_ctx_init.

tai_ctx_init​

tai_ctx_t *tai_ctx_init(void *mem, const tai_config_t *cfg);

Initializes the context. mem must be at least tai_ctx_size() bytes and remain valid for the lifetime of the context.

Parameters:

• mem - Preallocated memory block
• cfg - Configuration structure

Return value: Returns tai_ctx_t* (that is, mem cast to this type) on success, or NULL on failure.

tai_ctx_deinit​

void tai_ctx_deinit(tai_ctx_t *ctx);

Deinitializes the context and releases internal resources. mem can be freed after this call. It must be called after tai_disconnect.

tai_connect​

int tai_connect(tai_ctx_t *ctx);

Performs the TLS handshake, sends ClientHello, establishes a Session, and starts the background receive thread.

Return value: TAI_OK on success, or TAI_ERR_* on failure.

Behavior:

• Blocks until the Connection is established or fails
• After success, the background thread begins processing Ping/Pong and receiving data
• The tai_send_* functions can be called once the Connection succeeds

tai_disconnect​

void tai_disconnect(tai_ctx_t *ctx);

Stops and joins the background receive thread first, then sends a best-effort SessionClose, and finally closes the transport (TLS / TCP).

Behavior:

• Blocks until the background thread exits (join)
• Data cannot be sent after this call
• Call tai_ctx_deinit afterward
• Must not be called from a callback (it would join the worker thread that is executing the callback, causing a self-deadlock); to trigger disconnection from a callback, use tai_request_disconnect

tai_request_disconnect​

void tai_request_disconnect(tai_ctx_t *ctx);

Requests the background worker to stop, but does not join it or tear down the Connection. It can be called from any thread, including from within a receive callback (unlike tai_disconnect). The worker exits on its next loop iteration.

Purpose: Safely triggers disconnection from a callback. After this call, the thread that owns the context must still call tai_disconnect() to join the worker, send SessionClose, and release resources.

5. Send API​

tai_send_text​

int tai_send_text(tai_ctx_t *ctx, const char *text, size_t len);

Sends a text message. Internally, one call sends four application Packets in sequence: EventStart -> text (with the ONE_SHOT stream flag) -> EventPayloadsEnd -> EventEnd; the caller does not need to finish the sequence manually.

Parameters:

• text - UTF-8 text
• len - Text length in bytes

Return value: TAI_OK or TAI_ERR_*.

tai_send_audio_start​

int tai_send_audio_start(tai_ctx_t *ctx,
                         uint8_t codec, uint8_t channels,
                         uint8_t bit_depth, uint32_t sample_rate);

Starts an audio stream. It must be called before tai_send_audio_chunk.

Parameters:

• codec - Encoding format: TAI_AUDIO_PCM (101) or TAI_AUDIO_OPUS (111)
• channels - Number of channels (usually 1)
• bit_depth - Bit depth (usually 16)
• sample_rate - Sample rate (usually 16000)

tai_send_audio_chunk​

int tai_send_audio_chunk(tai_ctx_t *ctx, const uint8_t *pcm, size_t len);

Sends one chunk of audio data. It can be called repeatedly (for middle Packets in the stream).

Parameters:

• pcm - Audio data (a PCM or Opus frame, depending on the codec passed to audio_start)
• len - Data length in bytes

tai_send_audio_end​

int tai_send_audio_end(tai_ctx_t *ctx);

Ends the audio stream. Notifies the server that the current audio input is complete and processing can begin.

tai_send_image​

int tai_send_image(tai_ctx_t *ctx,
                   const uint8_t *data, size_t len,
                   uint8_t format, uint16_t width, uint16_t height);

Sends a single image.

Parameters:

• data - Image binary data
• len - Data length
• format - Image format: TAI_IMG_JPEG (1) or TAI_IMG_PNG (2)
• width / height - Image dimensions

tai_send_image_with_text​

int tai_send_image_with_text(tai_ctx_t *ctx,
                             const char *text, size_t text_len,
                             const uint8_t *img_data, size_t img_len,
                             uint8_t format,
                             uint16_t width, uint16_t height);

Sends text and an image together (for example, in an image-understanding scenario). Internally, the text Packet is sent before the image Packet to form one complete request.

tai_chat_break​

int tai_chat_break(tai_ctx_t *ctx);

Sends a Chat break Event. This is used when the user actively interrupts the AI response (for example, with a button). After receiving it, the server stops generating the current response.

tai_send_mcp_response​

int tai_send_mcp_response(tai_ctx_t *ctx, const char *json_rpc_response);

Sends the response to an MCP command. When on_event receives TAI_EVT_MCP_CMD, the device executes the command and returns the result through this function.

Parameters:

• json_rpc_response - Response string in JSON-RPC format

6. Logging​

The SDK's global log switch is the compile-time AGENTIC_KIT_LOG_LEVEL (gated once in common/log.h; it applies to the entire SDK, not only this module); this module can additionally be lowered alone with AGENTIC_KIT_TAI_LOG_LEVEL (the renamed former TAI_LOG_LEVEL — lower-only, defaulting to the SDK-wide value). Valid values:

Log lines above the ceiling are eliminated at compile time; below it they emit unconditionally — there is no runtime level (tai_set_log_level() was removed together with the runtime layer). Compile production builds with -DAGENTIC_KIT_LOG_LEVEL=2 to keep only error + warn, or lower only this module with -DAGENTIC_KIT_TAI_LOG_LEVEL=N (below 3, the tai_log_packet JSON formatter and its calls vanish together); changing where lines go (or dropping them by level) is a build decision as well: define AGENTIC_KIT_LOG and take over the dispatch with your own macro.

Media middle frames log at INFO with 1/N sampling by default (AGENTIC_KIT_TAI_LOG_MEDIA_SAMPLE_N, default 50; set it to 0 to disable sampling and enter flood mode). In flood mode middle frames emit at DEBUG and only when the effective ceiling is 4 — with a ceiling of 3, setting sampling to 0 leaves middle frames with no logs at all (fewer than the default sampling; do not mistake it for packet loss).

7. Typical Usage Flow​

#include "tuya_ai.h"

// 1. Allocate memory
void *mem = malloc(tai_ctx_size());

// 2. Configure
tai_config_t cfg = {
    .host             = "ai-gw.example.com",
    .port             = 8883,
    .device_id        = devid,
    .local_key        = local_key,
    .client_type      = TAI_CLIENT_DEVICE,
    .protocol_version = TAI_VER_21,
    .sign_level       = TAI_SIGN_HMAC_SHA256,
    .pal              = &my_pal,
    .on_audio         = my_on_audio,
    .on_text          = my_on_text,
    .on_image         = my_on_image,
    .on_event         = my_on_event,
    .on_disconnect    = my_on_disconnect,
};

// 3. Initialize + connect
tai_ctx_t *ctx = tai_ctx_init(mem, &cfg);
if (tai_connect(ctx) != TAI_OK) { /* handle error */ }

// 4. Send text
tai_send_text(ctx, "hello", 5);

// 5. Or send an audio stream
tai_send_audio_start(ctx, TAI_AUDIO_PCM, 1, 16, 16000);
while (has_audio) {
    tai_send_audio_chunk(ctx, pcm_frame, frame_len);
}
tai_send_audio_end(ctx);   // Finish in manual mode; do not call audio_end in Server VAD mode

// 6. Receive responses asynchronously through callbacks...

// 7. Clean up
tai_disconnect(ctx);
tai_ctx_deinit(ctx);
free(mem);

The audio-stream call pattern depends on the VAD mode (continuous conversation with Server VAD vs. device-side VAD/manual push-to-talk). See the comparison table in VAD and Interruption.

8. Thread Safety​

• The tai_send_* functions are thread-safe and can be called from any thread
• All callbacks are invoked sequentially on the same background receive thread
• tai_send_*() can be called from a callback (no lock is held)
• tai_connect, tai_disconnect, and tai_ctx_deinit must not be called from a callback (they join the worker thread, causing a self-deadlock)
• To trigger disconnection from a callback, call tai_request_disconnect(), then have the thread that owns the context call tai_disconnect()