Multi-Agent Embodied Architecture

1. Course Content

Understand the multi_brains multi-agent embodied framework.

2. Introduction to multi_brains

• multi_brains is an embodied intelligence framework developed in-house by Yahboom Intelligent that is compatible with general-purpose AI large models. It avoids the issues of VLA models and end-to-end models, which require dedicated hardware, local computing power support, and long data collection and training time. By merely integrating the general-purpose AI models of cloud model providers, the edge robot can gain powerful embodied intelligence capabilities.

• multi_brains expands training samples through the RAG knowledge base, allowing the robot to quickly adapt to different scenario tasks. It can be deployed on the vast majority of small-scale host devices, placing the massive model inference in the cloud or on a local server.

   

• The core philosophy of multi_brains is to decouple the different stages of a task through multiple AI models in a layered manner. The current package uses terminal text interaction as the default entry point: terminal_client.py publishes user text, and llm_agent_node.py completes Dify calls, action execution, and the feedback closed loop. Its main improvements are:

  • A more stable text debugging entry point that does not rely on microphones, wake words, or ASR/TTS services, making it suitable for development, teaching, and experiment reproduction.

  • A more flexible decision-making mechanism with task routing added, enabling the AI agent to autonomously choose between multi-layer reasoning and simplified reasoning based on task difficulty, giving the AI higher autonomy and a better user experience.

  • An easily understandable visualized architecture flow. multi_brains builds visualized agents through Dify, and the data flow of the AI agent during operation can be observed in real time during debugging, allowing users to intuitively understand the specific process of multi-layer reasoning.

  • A fully localized RAG knowledge base, with no need to worry about sensitive data issues.

  • Seamless integration with most AI models worldwide. Through Dify's model supplier plugins, hundreds of AI models worldwide can be quickly and seamlessly integrated (if you need locally deployed models, local models can also be integrated).

  • Traceable historical records. By querying Dify's backend access history, users can query all the execution steps and history of every task instruction.

  • Terminal-traceable interaction process. terminal_client.py displays progress, final replies, and executed actions, making it easy to determine whether Dify decisions and robot actions are consistent.

  • A complete testing toolbox for quickly verifying core functional modules.

  • Simpler onboarding steps, with most operation steps streamlined and shortcut commands configured.

3. Multi-Layer Reasoning Architecture

• The current workflow is mainly composed of a decision layer and an execution layer, with a frame-selection layer newly added to visual tracking tasks. Ordinary actions still run through the "decision layer -> execution layer -> ROS action feedback" closed loop; when a task requires tracking a visual target in the frame, a "frame-selection layer -> execution layer starts tracking" visual branch is added.

• The design of layered reasoning and dynamic feedback reasoning provides stronger system robustness than a single-model architecture.

3.1 Advantages of Layered Reasoning

3.1.1 Decoupling of Task Decision and Action Conversion

• The decision-layer AI focuses on thinking, planning, and decomposing task instructions.

• The execution-layer AI focuses on chat replies and converting task steps into JSON text.

• The frame-selection-layer AI is only responsible for outputting bbox coordinates based on the image and the target description, and does not participate in task planning or action control.

3.1.2 Improving the Success Rate of Long-Flow Tasks

When reasoning with a single-model approach, "natural language understanding + environment perception + process decomposition + bbox estimation + action function output" must be performed simultaneously, which tends to cause modality interference (language ambiguity or overlong instructions leading to comprehension errors). Layered reasoning processes the task in stages, enabling the decision layer to focus on task planning, the execution layer to focus on action execution, and the frame-selection layer to focus on visual target localization.

3.2 Decision-Layer AI

• The decision-layer large model is mainly responsible for task planning. It can understand complex human instructions and decompose them into concrete task steps. For example, when it receives the instruction "Can you please get me a bottle of mineral water from the kitchen?", the large language model will decompose it into several tasks, as shown in the figure below.

3.3 Execution-Layer AI

• The execution-layer large model is mainly responsible for chat replies and converting task steps into JSON text. It continuously receives feedback (success/failure) and images from the results of the robot's action execution, and issues the next action instruction according to whether the executed action succeeded or failed.

• The execution-layer large model acts as a supervisor that continuously monitors the progress of the robot in executing task steps, and, based on the result feedback and environment information after the robot executes an action, thinks and judges the next action to execute, until the task is successfully completed or is ended early due to special circumstances.

3.4 Frame-Selection-Layer AI

• The frame-selection-layer AI only participates in work when visually tracking, locking on to, or following a target. It receives the current image, the user's original instruction, and the decision layer's current step, and outputs whether the target was found, the target name, bbox coordinates, confidence, and a brief reason.

• The output format of the frame-selection layer is strictly JSON, for example:

{"found":true,"target":"Green square","bbox":[120,80,360,300],"confidence":0.86,"reason":"The goal is clearly visible"}

• After reading the frame-selection-layer output, the execution layer will output track_kcf(x1,y1,x2,y2) only if found=true and the bbox is valid; if there is no frame-selection result or the target is ambiguous, the execution layer will end the task and explain the reason to the user.

4 . Task Cycle

• After the user inputs a natural language instruction in the terminal client, a task cycle begins. Within this cycle, the user instruction, action result feedback, and image feedback all continue reasoning in the same Dify conversation_id, which acts as the temporary memory of the robot and allows it to "remember" what has happened in the current task.

• When the execution layer outputs finishtask(), or the user enters reset in the terminal to reset the conversation, the robot resets the historical context of the AI model, which is equivalent to the robot "losing its memory" and forgetting what happened before.

• If a new instruction is entered during execution, llm_agent_node will give priority to the new instruction and cancel the current navigation, line-following, or tracking actions as soon as possible.

5. Historical Context

• The robot's conversation history is stored in the AI application variables of Dify, with a default maximum of 50 rounds of conversation memory.

6. Map Mapping

• The robot uses a grid map for navigation. If the robot needs to understand position areas in the real world, a mapping relationship must be established between the grid map and the real-world environment areas; this relationship is called map mapping.

• Suppose we use the robot to generate a grid map through SLAM mapping in a factory environment, and the real factory has several artificially divided areas, as shown in the figure below.

We establish a one-to-one correspondence between these real-world environment areas and letter symbols:

A: "Area 1", B: "Area 2", C: "Area 3", D: "Area 4", E: "Area 5", F: "Area 6", G: "Area 7", H: "Area 8", I: "Area 9"

Then, in the yaml file, the map coordinates are written for each letter symbol, for example:

xxxxxxxxxx
A: 
  name: 'Area 1'
  position:
    x: 4.4034953117370605
    y: 0.4879316985607147
  orientation:
    x: 0.0
    y: 0.0
    z: 0.701498621044694
    w: 0.7126708108744126

When we ask the robot to go to a certain actual area, we only need the large model to convert it into the corresponding letter symbol, so that the robot can understand the location areas in the real environment.

7. Action Function Library

• The API functions in the robot's action function library are the core bridge through which the large model controls the robot to interact with the real physical world.

• These API functions define the smallest atomic action units that the physical robot can perform in the physical world.

• The AI large model converts the action sequence after thinking and planning into strict JSON text and sends it to the robot base (e.g., {"action": "set_cmdvel(0.2, 0.0, 0.0, 2.0)"}). After parsing, the robot distributes the calls to the corresponding underlying ROS 2 services or actions.

According to the different vision sensor hardware installed, the system provides the USB camera version (action_service_usb.py) and the Nuwa depth camera version (action_service_nuwa.py) of the action service implementation. The specific function list and functions are as follows:

7.1 Common Basic Action Functions for Both Versions (Shared by USB and Nuwa)

The following actions are common basic motion, navigation, robotic arm interaction, and task lifecycle functions for both sets of vision hardware:

7.2 USB Camera Exclusive Action Functions (action_service_usb.py)

For the regular 2D monocular USB camera solution, object tracking and fixed-height planar visual servo grasping are mainly achieved by relying on the 2D pixel bounding box and the KCF algorithm:

7.3 Nuwa Depth Camera Exclusive Action Functions (action_service_nuwa.py)

The solution equipped with the Nuwa 3D depth camera (RGB-D) has spatial depth point-cloud perception capabilities and supports 3D physical-space positioning and inverse-kinematics (IK) 3D grasping and placement:

7.4 Technical Difference Comparison of the Two Camera Solutions

The following table intuitively summarizes the core differences between the USB camera and the Nuwa depth camera in multi-agent embodied control: