2. Multimodal Visual Understanding (Text Version)

1. Course Content

1. Learn to use the robot's visual understanding function.
2. Learn the newly appearing key source code.

2. Preparation

2.1 Content Description

The virtual machine used by default with this product starts the Dify docker environment automatically at boot, so no additional startup is needed.

⚠️ The same test instructions will not produce exactly the same reply content from the large model, and will differ slightly from the screenshots in the tutorial. If you need to enhance or reduce the diversity of the large model's replies, refer to 【AI Large Model Development】-【RAG Knowledge Base Deployment】 to modify the knowledge base and sample library.

Virtual machine terminal, manually start the Dify environment script:

sh ~/bringup_dify.sh

3. Running the Case

3.1 Starting the Program

Open the virtual machine terminal and enter the command to start the underlying proxy + camera topic:

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

Open another virtual machine terminal and enter the command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

Then open another terminal and start:

xxxxxxxxxx
ros2 run multi_brains terminal_client

3.2 Test Cases

Here are two reference test cases. Users can create their own test instructions.

• Help me see what items are in front of you, and tell me what these items are used for.
• Please look to see if there is a blue block and a pack of tissues in front of you. If so, nod your head; if not, shake your head.

⚠️ Do not add a period or any character at the end of the input text!

3.2.1 Case 1

Enter the test case in the text interaction terminal. After waiting for the model to think, the model replies to the user and executes actions according to the user's instructions.

3.2.2 Case 2

The method is the same as Case 1. Enter Case 2 in the terminal, and the model will reply and execute actions according to the instructions.

Case 2 uses the pan-tilt servo camera version as an example. Nodding and shaking the head can also be replaced with commands such as the car moving forward or moving backward.

4. Source Code Analysis

Visual understanding is completed uniformly by llm_agent_node.py: the node subscribes to the camera topic /camera/image_raw and caches the latest image; when the execution layer outputs seewhat(), the node saves the current frame as an image and uploads it to Dify as a feedback attachment, and Dify then continues the next round of reasoning based on the image.

Current relevant source code paths:

xxxxxxxxxx
~/microros_ws/src/multi_brains/multi_brains/llm_agent_node.py
~/microros_ws/src/multi_brains/multi_brains/services/dify_coordinator.py
~/microros_ws/src/multi_brains/multi_brains/controllers/ptz_controller.py

Core flow:

xxxxxxxxxx
User inputs a vision question in terminal_client
  -> Execution layer outputs seewhat()
  -> llm_agent_node saves the current image from /camera/image_raw
  -> Uploads the image through DifyCoordinator and feeds back "robot image feedback completed"
  -> Dify generates a reply based on the image or continues to output actions
  -> terminal_client displays the final reply and action list

If the user asks to "look around", the execution layer will output observe_surroundings(), and the current node will control the pan-tilt to take photos of the front, right, and left in sequence, and feed multiple images to Dify at once.