Input and state
- Streaming audio and audio end
- Image and video frames
- Text, device Events, and tool execution results
- Device state and Data Point Reports
Core Concepts
Understand the relationships among identity, connections, Sessions, and capability calls before moving on to specific APIs. Agentic-kit is designed to keep devices lightweight while placing platform capabilities and customer-owned capabilities behind clear, replaceable boundaries.
Identity and Activation
Authorization codeLICENSE
The identity credential held by each device at the factory, consisting of a globally unique uuid and its paired authkey. It proves that the device is authorized for Activation; it is not the same as the device ID obtained after Activation.
Factory identity: uuid + authkey -> successful Activation -> official cloud identity: devid + secret_key + local_keyProduct PIDPRODUCT ID
Identifies the shared configuration for a product family. A PID can define Data Points, a panel, and a default AI Agent. Devices of the same product share a PID but have different authorization codes and device IDs.
Device ActivationACTIVATION
The process of moving a device from its factory state to being networked, bound to a user, and registered with the cloud. Activation is normally performed only once. The device should securely persist the returned credentials and use them directly on subsequent starts.
Agents and Sessions
AI AgentBEHAVIOR
Defines how the product understands, decides, and expresses itself. On the platform, you can configure the system prompt, language, TTS, workflows, and callable tools. A product can use a default Agent or switch Agents according to project configuration.
A PID answers "What product is this?" An Agent answers "How does this product communicate and act?"SessionSESSION
The runtime context for one continuous AI interaction. It contains connection state, input and output streams, callback events, and optional Session attributes. The SDK carries Session data; the application and platform configuration jointly define the specific business semantics.
Agent CorePLATFORM RUNTIME
The Agent orchestration and runtime layer in the Tuya Physical AI platform. It handles context assembly, tool orchestration, model routing, the memory framework, and runtime assurance. It is a cloud platform capability and is not compiled into the device with the device-side SDK.
The tRTC real-time channel
tRTC is the encrypted real-time channel between a device and the Tuya AI cloud. Agentic-kit currently provides two client implementations: an open-source TCP implementation and a UDP precompiled static library. They differ in delivery model, platform support, and reconnect ownership. Choose according to the target chip, weak-network requirements, and requirements for control over the source code.
| Implementation | Transport | Integration form | Main characteristics |
|---|---|---|---|
rtc-tcp-client | TCP | Open-source code + PAL | Supports source-level control and cross-platform porting; the application handles disconnections and reconnects as documented |
rtc-client | UDP | Precompiled static library supplied for each chip platform | Suitable for rapid integration and weak-network scenarios; contact Tuya sales for the library matching the target platform |
Confirm the delivery model before choosing
If you need source-level control or must adapt to a new system, evaluate the open-source TCP implementation first. If you need the UDP channel, first confirm the chip, operating system, toolchain, and library version, then obtain the matching precompiled static library from Tuya sales.
A real-time channel is not an AI model
The channel sends audio, images, video, text, events, and tool results to the appropriate capabilities, and returns replies and commands to the device. Models and algorithms can run on the Tuya platform, in the customer's cloud, or on the device.
Multimodal streams and events
Results and actions
- TTS audio and text replies
- Image and video results
- Session, VAD, and interruption Events
- MCP tool calls and device commands
The device application needs explicit state machines for capture, buffering, playback, interruption, and failure recovery. For example, when using cloud VAD, the device continuously sends audio and ends the current uplink turn after receiving the server endpoint Event. Local VAD is an optional optimization for scenarios with battery, bandwidth, or high-interactivity requirements.
PAL: Platform Abstraction Layer
The PAL is Agentic-kit's portability boundary. Rather than binding directly to a particular RTOS or chip, the SDK uses TCP, threading, mutex, time, and memory through function pointers. Official POSIX and FreeRTOS implementations are provided as references for adapting new platforms.
| Category | Typical interfaces | Engineering concerns |
|---|---|---|
| Network | connect / send / recv / close / poll | Timeouts, partial sends and receives, TLS dependencies, network recovery |
| Concurrency | thread_create / join, mutex | Thread stack, callback context, lock granularity |
| System | wal_clock_time, time_ms, malloc / free | UTC synchronization and monotonic timing, peak memory use, and fragmentation |
Physical device control: DPs and device-side MCP
Agentic-kit provides two ways to control physical devices. Customers can choose Data Points or device-side MCP based on whether the device capability is standardized, whether call parameters are dynamic, and whether an Agent must understand the context. A single product can also use each for different types of capability.
Standard device capabilities and state
Use a product Schema to describe stable capabilities such as switches, brightness, temperature, and modes. DPs suit app control, state synchronization, automation, and cross-device linkage. A device refreshes cloud state through Reports and changes local state in response to downlink DP sets.
Dynamic tools for Agents
Register capabilities such as reading a sensor, taking a photo, controlling a motor, or performing a task as tools with names, descriptions, and parameter Schemas. This suits AI interactions that require contextual understanding, structured parameters, or returned execution results.
| Decision factor | Prefer DPs | Prefer device-side MCP |
|---|---|---|
| Capability form | Stable, enumerable device attributes and functions | Dynamic, task-oriented tools or tools with complex parameters |
| Primary caller | App, cloud automation, device linkage | AI Agent |
| Result representation | State values and standard commands | Structured execution results, errors, and context |
The application owns state synchronization after reconnect
The SDK does not automatically Report restored local DP state again. The application should refresh the required state after every successful connection or reconnect to avoid inconsistent cloud and device displays.
Device-side and cloud-side MCP roles
MCP uses a unified protocol to describe tool names, purposes, parameters, and return values. In Physical AI, a tool can reside on a device or in a customer or third-party cloud. Device-side MCP is one of the two physical device control methods in the previous section.
The device is the tool provider
The device registers capabilities such as reading sensors, controlling motors, and taking photos. An Agent initiates a call; the device validates the parameters, performs the action, and returns a structured result.
A cloud service is the tool provider
Cloud services perform weather, search, enterprise knowledge, or core customer business operations. They can use Tuya capabilities or integrate a customer's own implementation.
Physical actions require stricter boundaries
A tool declaration is only the call entry point. Before an embodied robot or spatial assistant performs actions such as operating a door lock or curtains, moving, or grasping, it must also verify device binding, user authorization, parameter ranges, current state, and failure fallback.
Perception, models, and memory
A Physical AI experience emerges from the coordination of perception, reasoning, memory, and execution. These capabilities do not have to be deployed in the same place or provided by the same service.
Device-side perceptionEDGE
Keyword wake-upLocal VAD (optional)Visual preprocessingSensor fusionEmergency stop
Platform capabilitiesTUYA PHYSICAL AI
Cloud VAD / ASR / TTSVision algorithmsAgent CoreOmniMemPAMMCP / A2A
Customer capabilitiesYOUR STACK
Private modelsPrivate-domain knowledgeCore cloud servicesIn-house algorithmsProduct business logic
For example, a voice flow can combine streaming ASR, dynamic hotwords, cloud VAD, and a customer's own TTS; a pet camera can connect individual pet recognition, behavior understanding, and event recording; and a robot can combine platform task planning with robot control, safety policies, and authorized home devices. Deployment location depends on latency, power consumption, bandwidth, privacy, compute capacity, and control of business logic.
Boundary overview
| Layer | Main responsibilities | Controlled by |
|---|---|---|
| Product application | Capture, playback, state machines, action execution, user experience | Customer |
| Agentic-kit | Identity, IoT, tRTC, Session Events, data, and tool interaction | Open SDK + customer integration |
| Tuya Physical AI platform | Agent orchestration, platform models, memory, knowledge, device and industry services | Selected as needed |
| Customer services | Private models, core business, private-domain knowledge, and differentiating algorithms | Customer, integrated through open capabilities |
Next steps
- System architecture - Learn how the modules work together
- Quick start - Run the first example on your computer
- Create and configure an Agent - Prepare your product and Agent on the IoT Platform