Welcome to DOFBOT-Ultra repository
3. Embodied Intelligent Robot System Architecture
Embodied Intelligent Robot System Architecture
Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!
Before starting this feature, you need to close the web control program backend process.
1. Course Content
Explain the large model inference architecture of AI embodied intelligent robots.
Explain the basic concepts in the system, laying the groundwork for subsequent hands-on courses.
2. System Components
3. AI Large Model Inference Architecture Diagram
The robot adopts a dual-model inference and dynamic feedback inference design in its programming, improving the robot's ability to handle long-flow complex tasks and providing stronger system robustness compared to single-model architecture. Here are several key concepts: decision layer large model, execution layer large model, task cycle, and dialogue history, which will be explained one by one below.
4. Large Model Inference Architecture Explanation
4.1 Dual Model Architecture Principle
The robot's large model control system is designed based on a text generation large model and a multimodal large model. The text generation large model serves as the decision layer, acting as a high-level task planner that decomposes complex, abstract human instructions into tasks.
The multimodal large model serves as the execution layer, receiving the task steps generated by the decision layer and the user's temporary instructions (generally simple instructions asking the robot to end a task or rest), supervising the robot's task execution progress, adjusting execution actions in real-time, and generating action function lists that the robot can parse as well as responses to the user.
The action function list contains pre-written basic action functions that can directly output parameters to control the robot's movement.
4.2 Advantages of the Dual Model Inference Architecture
4.2.1 Decoupling of Decision Logic and Action Control
Multimodal large models are inferior to text generation large models in natural language semantic understanding and logical reasoning, and are prone to modality interference when executing complex tasks. Therefore, the system design decouples decision logic from action control.
The text generation large model focuses on semantic understanding and task planning (e.g., parsing "Can you help me get the blue block in room A?" and breaking it down into task steps: "Record current position -> Navigate to room A -> Get current robot perspective image -> Locate blue block coordinates -> Grasp blue block -> Return to starting position -> Place blue block"), avoiding the complexity of simultaneously handling action details during semantic parsing in traditional single models.
The multimodal large model is responsible for action generation and environment interaction (e.g., obtaining visual images to locate object coordinates, outputting action functions and parameters), reducing the likelihood of action logic confusion and large deviations caused by task complexity in single models.
4.2.2 Reducing Model Training Sample Complexity
When using a single model for inference, the multimodal large model must simultaneously perform "natural language understanding + environment perception + process decomposition + action function output," which is prone to modality interference (language ambiguity or overly long instructions leading to comprehension errors). The dual model handles this through phased processing, allowing the decision layer to focus on language space mapping and the execution layer to focus on visual-motion space mapping.
4.2.3 Flexible Expansion
The decision layer and execution layer large models can be flexibly paired with various large models on the Alibaba Bailian Large Model Platform (sometimes called the Tongyi Qianwen platform). Through customized training samples and RAG knowledge bases, the models can adapt to tasks in different scenarios, greatly increasing the generalization ability of AI embodied intelligent robots across various scenarios.
4.3 Decision Layer Large Model
4.3.1 Role of the Decision Layer Model
The robot's decision layer large model defaults to using the Tongyi Qianwen-Max series model.
The decision layer large model is primarily responsible for task planning. It can understand complex human instructions and decompose them into specific task steps.
Each task step corresponds to the smallest action the robot can perform, which is then executed by the execution layer large model by calling API functions from the robot's action function library.
4.3.2 Robot Action Function Library (Simplified)
The robot action library specifies all the smallest actions the robot can physically perform. When planning task steps, the decision layer model selects appropriate actions from this library for combination.
Basic Action Class
Turn on red light
Turn on green light
Turn on blue light
Activate alarm
Deactivate alarm
Turn off light
Robotic Arm Class
Adjust joint x to y degrees
Adjust the specified servo to the specified angle.
Move robotic arm Description: Move the end of the robotic arm a specified distance in a specified direction.
Move robotic arm upward
Nod robotic arm
Shake robotic arm
Dance robotic arm Similar semantics: Shake head, wave head.
Applaud robotic arm Similar semantics: Clap, applaud.
Grip, grasp object with robotic arm Similar semantics: Pick up, grab.
Release item from robotic arm Description: Release the grasped item from the robotic arm.
Open gripper of robotic arm Description: Adjust the angle of servo 6 to 0 degrees.
Close gripper of robotic arm Description: Adjust the angle of servo 6 to 180 degrees.
Sort, grasp machine code x Description: Sort and grasp the specified machine code.
Track object, Description: Track the specified object.
Remove machine code of specified height
Remove color block of specified height. Description: Automatically remove the specified color whose height exceeds x cm. color values: 'red', 'green', 'blue', 'yellow'.
Sort garbage of type x Description: Call the garbage sorting function to sort garbage of type x. x values are 'rec', 'tox', 'wet', or 'dry'.
Move A to the side of B Description: Change the spatial position of grasped object A to the left side of object B. Parameters: (x1, y1) are the pixel coordinates of the top-left corner of object A's outer bounding box, (x2, y2) are the pixel coordinates of the bottom-right corner of object A's outer bounding box, (x3, y3) are the pixel coordinates of the top-left corner of object B's outer bounding box, (x4, y4) are the pixel coordinates of the bottom-right corner of object B's outer bounding box. src represents the name of object A; if A is a color block, src represents the color of the block. tar represents the name of object B; similarly, if B is a color block, tar represents the color of the block. side represents one of five directions: front, back, left, right, top. Values correspond to 1=front, 2=back, 3=left, 4=right, 5=top. Similar semantics: Place A on the side of B, put A on the side of B.
Stack color blocks Description: Grasp the color blocks in the image in the specified order.
Return color block to original position Description: Return the color block to its original position. Similar semantics: Return the color block to its place, put the color block back.
Track object
Track machine code
Cancel tracking machine code
Cancel tracking object
Remember current position of object
Point with robotic arm
Sort garbage
Clean garbage
Sort color blocks in given order
Image Acquisition Class
Get current perspective image Description: Get the current perspective image for observing the environment or locating objects.
Record x-second video Description: Record a video of the specified duration. x represents the duration in seconds.
Play a guessing/small ball game Description: Record a 20-second video.
Video understanding Description: Parse the content of the recorded 20-second video.
Other Classes
Query weather
Calculate math problem
Guess object English name
Tell a joke
4.4 Execution Layer Large Model
The robot's execution layer large model defaults to using the Tongyi Qianwen-VL series model.
The execution layer large model is primarily responsible for generating the action list for controlling the robot and the response to the user. During the execution of the action list, it continuously receives feedback (success/failure) from the robot about action results and visual images, inferring the next action to execute based on the action execution status.
The execution layer large model acts as a supervisor, constantly monitoring the robot's progress in executing task steps. Based on feedback from action results and environmental information, it determines the next action to execute, until the task is successfully completed or prematurely terminated due to special circumstances.
4.4.1 Task Cycle
In a new task cycle, the user's instruction first goes through the decision layer large model, then the execution layer large model. If the execution layer large model determines that the robot has completed all task steps, it enters the waiting state. At this point, subsequent user instructions are directly passed to the execution layer large model.
For example: When the user asks to end the current task or tells the robot to rest, the robot resets the dialogue history and robotic arm state, starting a new task cycle. At this point, the user's instruction starts from the decision layer large model. Below is an illustration of the waiting state:
4.4.2 Historical Context
The robot maintains a dialogue history context locally in its program. The dialogue history context includes the user's instructions, task steps generated by the decision layer model, action lists generated by the execution layer model, and robot feedback status information. Each time the robot requests the execution layer large model, it sends the complete dialogue history, allowing the execution layer large model to understand the robot's task execution progress. The execution layer large model combines the dialogue history context to infer the next action to execute. In summary, the execution layer large model derives the next future action based on the sum of past states.
Under normal circumstances, the execution layer large model determines task progress based on dialogue history. When it determines that the robot has completed all task steps, it enters the waiting state, awaiting temporary instructions from the user. Temporary instructions no longer pass through the decision layer large model but are directly sent to the execution layer large model. There are three situations that cause the robot to clear the dialogue history context, end the current task cycle, and start a new task cycle.
Situation 1: The user actively requests to end the task cycle. In the waiting state, if the user says something like "Take a rest now" or "You may leave" indicating the robot is no longer needed, the execution layer large model calls the finish_dialogue() function to end the current task cycle, reset the dialogue history, and start a new task cycle. The same applies to the following situations.
Situation 2: The robot fails twice consecutively when executing a certain action. If an action fails, the execution layer large model controls the robot to retry at most once. If it fails again, the execution layer large model controls the robot to prematurely end the task cycle and start a new one.
Situation 3: The robot encounters an impossible task, e.g., when the user asks the robot to go to a location not mentioned in the "map mapping", the robot will request to prematurely end the task cycle and inform the user that the task cannot be completed.
4.4.4 Robot Action Function Library
The API functions in the robot's action function library serve as the bridge for the large model to control the robot's interaction with the real world. These API functions specify the smallest actions the physical robot can perform in the physical world. The principle of the API functions is to control the underlying hardware of the physical robot through the ROS2 system to perform various functions.
All action functions and their corresponding functionalities are shown in the table below:
| Function Name | Parameters | Implementation Function | Invocation Instruction |
|---|---|---|---|
| arm_dance() | - | Trigger robotic arm dance action | Robotic arm dance |
| arm_up() | - | Move robotic arm upward | Robotic arm up |
| arm_down() | - | Move robotic arm downward | Robotic arm down |
| arm_nod() | - | Nod robotic arm | Nod |
| arm_shake() | - | Shake robotic arm | Shake head |
| arm_applaud() | - | Applaud robotic arm | Applaud |
| grip_pose() | - | Robotic arm grasping posture | Adjust arm to grasping posture |
| track_pose() | - | Robotic arm tracking posture | Adjust arm to tracking posture |
| cancel_KCF_follow | - | Cancel object tracking | Cancel tracking object |
| grasp_obj(x1, y1, x2, y2) | (x1, y1, x2, y2): top-left and bottom-right corner coordinates of the target object's outer bounding box | Robotic arm grasp an object | Grasp xxx |
| remove_obj(x1, y1, x2, y2,color) | (x1, y1, x2, y2): top-left and bottom-right corner coordinates of the top color block's outer bounding box. color: color of the obstacle color block. | Remove obstacle color block | Remove x color block |
| putdown() | - | Release the grasped item from the robotic arm | Put down xxx |
| apriltag_sort(x) | x: machine code number | Sort and retrieve the specified machine code | Sort out machine code x |
| track(x1, y1, x2, y2) | (x1, y1, x2, y2): top-left and bottom-right corner coordinates of the target object's outer bounding box | Track the specified object within view | Track xx |
| apriltag_remove_higher(x) | x: height | Remove machine code above specified height | Remove machine code higher than x cm |
| color_remove_higher(color, target_high) | color: color target_high: height | Remove color block of specified height | Remove color block higher than target_high cm |
| garbage_sort(type_) | type_: type of garbage to sort | Sort specified type of garbage | Sort recyclable garbage |
| change_pose(x1, y1, x2, y2, x3, y3, x4, y4, src, tar, side) | x1, y1, x2, y2, x3, y3, x4, y4: coordinates of top-left and bottom-right corners of target color block and target placement color block; src: color of target color block; tar: color of target placement color block; side: placement direction | Move color block to the side of another color block | Place blue block above yellow block |
| record_video(time_) | time_: video recording duration | Record video | Record a x-second video |
| apriltag_follow(target_id) | target_id: machine code ID | Track the machine code with specified ID | Track machine code x |
| compute_pose(x1, y1, x2, y2, name) | x1, y1, x2, y2: top-left and bottom-right corner coordinates of the color block whose position needs to be recorded; name: color of the color block | Record the current position of the specified color block | Record current position of x color block |
| compute_pose_order(x1, y1, x2, y2, name, order) | x1, y1, x2, y2: top-left and bottom-right corner coordinates of the top color block; name: color of the top color block; order: stacking order from top to bottom | Record the position of each stacked color block | Record current position of desktop color blocks |
| return_to_orin(name) | name: color of the color block to return | Return the specified color block to its original position | Return x color block to original position |
| light_on(color) | color: LED light color | Turn on LED light | Turn on x color light |
| light_off() | - | Turn off LED light | Turn off light |
| beep_on() | - | Turn on buzzer | Buzzer alarm |
| beep_off() | - | Turn off buzzer | Turn off buzzer |
| adjust_joint(joint_id, angle) | joint_id: servo ID, angle: servo angle | Set servo angle | Set servo x to y degrees |
| gripper_open() | - | Open gripper | Open gripper |
| gripper_close() | - | Close gripper | Close gripper |
| color_back_to_orin() | - | Publish color block return order | Return desktop color blocks to original positions |
| grasp_from_rm_list(color) | color: color block color | Grasp color block from the removal list | Grasp x color block from removal list |
| arm_move(Dir, Dist) | Dir: direction for adjusting the end of the robotic arm (up, down, left, right, front, back); Dist: adjustment distance in cm | Adjust the position of the end of the robotic arm | Move robotic arm x by y cm |
| point_to(x1, y1, x2, y2) | x1, y1, x2, y2: top-left and bottom-right corner coordinates of the object to point to | Robotic arm points to object | Point robotic arm to x |
| garbage_sort_all() | - | Sort garbage labels | Clean desktop garbage |
| grasp_from_down_list(color) | color: color block color | Grasp color block from the placement list | Grasp x color block from placement list |
| check_remove(color) | - | Check if the specified color block exists in the removal list | Check if x color block exists in removal list |
| seewhat() | - | Take a photo of the robot's current perspective and upload it to the execution layer large model | Observe environment |
| video_understanding() | - | Understand video content | What is the content of the video just recorded? |
| finish_dialogue() | - | Reset history context, start a new task cycle | - |
| wait(x) | x: time in seconds | Wait for a period of time, perform no action | Wait x seconds |
| finishtask() | - | Automatically called after the robot completes a task to stop sending status feedback to the execution layer model | - |
5. Application of Voice Models in the System
5.1 Speech Recognition
Since the decision layer and execution layer large models are respectively a text generation model and a vision multimodal model, they cannot directly receive voice information from the user. Therefore, a speech recognition large model is needed to convert the user's voice instructions into corresponding text before passing it to the AI large model.
5.2 VAD Voice Activity Detection
VAD (Voice Activity Detection) is a technology that automatically identifies speech segments and non-speech segments (such as silence, noise) in a speech signal. It locates the start and end positions of speech from the audio stream and filters out irrelevant background sounds.
In subsequent hands-on courses, when the user uses the wake word "Hello yahboom" to wake up the robot, it will enter VAD voice activity detection. The system automatically detects the user's speaking duration, saves the valid sound segment as a WAV audio file, and then sends the audio file to the speech recognition model for conversion to text.
5.3 Speech Synthesis
The text response generated by the large model for the user is converted into audio through a speech synthesis model, which is then played through the hardware speaker.
