Multimodal visual understand + visual following (Text Version)

1. Course Content

1. Learn to use the robot's vision combined with the robot's follow-up function
2. Analyze new key source code

2. Preparation

2.1 Content Description

This lesson uses the Raspberry Pi 5 as an example. For Raspberry Pi and Jetson-nano , you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this lesson in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For RDK X5 and Orin boards , simply open a terminal and enter the commands mentioned in this lesson.

📝 This example uses model："qwen/qwen2.5-vl-72b-instruct:free","qwen-vl-latest"

⚠️ The responses from the large model may not be exactly the same for the same test command and may differ slightly from the screenshots in the tutorial. To increase or decrease the diversity of the large model's responses, refer to the section on configuring the decision-making large model parameters in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base] .

⚡️ It is recommended that you first try the previous visual example. This example adds voice functionality to the singleton example. The functionality is largely the same, so I will not elaborate on the debugging details of the program. The results will be detailed here! ! !

3. Running the Example

3.1 Starting the Program

Open the host terminal and start the Dify environment:

xxxxxxxxxx
sh ~/bringup_dify.sh

For Raspberry Pi 5 and Jetson Nano controllers, you must first enter the Docker container. For RDKX5 and Orin board, this is not necessary.

Open a terminal in Docker and enter the following command:

x
ros2 launch multi_brains llm_agent_control.launch.py text_chat_mode:=True

After initialization, the following content will be displayed:

Open the same Docker container in multiple terminals and start it.

xxxxxxxxxx
ros2 run text_chat text_chat

3.2 Test Cases

Here are some test cases for reference; users can create their own test instructions.

• Start xx tracking

  Color/Face/Object/Machine Code/QR Code/Gesture Recognition/Human Posture

  Color tracking, including red, green, blue, and yellow (requires color calibration according to the AI Large Model Preparation tutorial).

  Object tracking

⚠️ Please do not end the text with a period or any other characters!

• ⚡️ Note for gimbal servo USB camera users: No object following!

  • This tracking example uses the principle of near-increase-increase-difference. The size of the object being followed and the tracking effect are related to the recognized area. For program debugging, please refer to the single example.

  • Tracking Examples	Recommended Reference Object Size
    Color Following	3x3cm Building Blocks
    Face Following	Face from a Mobile Phone Image
    AprilTag Machine Code Following	3x3cm Building Blocks
    QR Code Tracking	QR Code Printed on A5 Paper
    Mediapipe Gesture Following	Real-person gestures
    Meidpipe Human Posture Following	Real-person posture

3.2.1 Example 1: "Start following red"

Type "Start following red" in the terminal. The terminal will print the following information.

A window titled frame will open on the VNC screen, displaying the current robot-viewing image.

Move the object slowly, and the robot will follow.

To end the task, press ENTER in the terminal. Press the [Enter] key to continue the conversation, enter [Stop Following], or [End Following].

If the camera doesn't see the red object after 10 seconds, a countdown will appear on the terminal. When the countdown ends, the program will automatically stop following.

After completing a task, the robot will enter a waiting state. Pressing the ENTER key will continue the conversation. The command will be directly passed to the execution layer model, and all conversation history will be retained. We can enter the "End Current Task" command again to end the robot's current task cycle and start a new one.

3.2.2 Example 2: "Follow the Object in My Hand" (Depth Camera)

⚠️ The coordinates obtained by following the object in this example are entirely derived from the inference of the large AI model. Therefore, it is recommended to use a newer model for better results!

Enter "Please follow the object in my hand" in the terminal. The terminal will print the following information:

A window titled frame will open on the VNC screen, displaying the image from the robot's current perspective.

Move the object slowly, and the robot will follow.

If the screen If there are no targets to follow, the program will count down for 10 seconds, a 5-second countdown will be printed on the terminal, and the process will automatically end, indicating that the task has been completed.

To manually end the task, press the ENTER key in the terminal and continue the conversation, entering either [Stop Following] or [End Following].

After completing a task, the robot enters a waiting state. Commands at this point are directly passed to the execution layer model, and all conversation history is retained. You can enter the "End Current Task" command again to end the current task cycle and start a new one.

4. Source Code Parsing

Source code is located at:

Jetson Orin Nano:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

Jetson Nano, Raspberry Pi:

You need to first enter Docker.

xxxxxxxxxx
#NUWA Camera User
/root/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/root/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

RDKX5：

x
#NUWA Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

4.1 Example 1

4.1.1 Nuwa Depth Camera, action_service_nuwa.py

Example 1 uses the seewhat,