Multimodal visual understand + visual following (Text Version)

1. Course Content

1. Learn to use the robot's vision combined with the robot's follow-up function
2. Analyze new key source code

2. Preparation

2.1 Content Description

This lesson uses the Raspberry Pi 5 as an example. For Raspberry Pi and Jetson-nano , you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this lesson in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For RDK X5 and Orin boards , simply open a terminal and enter the commands mentioned in this lesson.

📝 This example uses model："qwen/qwen2.5-vl-72b-instruct:free","qwen-vl-latest"

⚠️ The responses from the large model may not be exactly the same for the same test command and may differ slightly from the screenshots in the tutorial. To increase or decrease the diversity of the large model's responses, refer to the section on configuring the decision-making large model parameters in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base] .

⚡️ It is recommended that you first try the previous visual example. This example adds voice functionality to the singleton example. The functionality is largely the same, so I will not elaborate on the debugging details of the program. The results will be detailed here! ! !

3. Running the Example

3.1 Starting the Program

Open the host terminal and start the Dify environment:

xxxxxxxxxx
sh ~/bringup_dify.sh

For Raspberry Pi 5 and Jetson Nano controllers, you must first enter the Docker container. For RDKX5 and Orin board, this is not necessary.

Open a terminal in Docker and enter the following command:

x
ros2 launch multi_brains llm_agent_control.launch.py text_chat_mode:=True

After initialization, the following content will be displayed:

Open the same Docker container in multiple terminals and start it.

xxxxxxxxxx
ros2 run text_chat text_chat

3.2 Test Cases

Here are some test cases for reference; users can create their own test instructions.

• Start xx tracking

  Color/Face/Object/Machine Code/QR Code/Gesture Recognition/Human Posture

  Color tracking, including red, green, blue, and yellow (requires color calibration according to the AI Large Model Preparation tutorial).

  Object tracking

⚠️ Please do not end the text with a period or any other characters!

• ⚡️ Note for gimbal servo USB camera users: No object following!

  • This tracking example uses the principle of near-increase-increase-difference. The size of the object being followed and the tracking effect are related to the recognized area. For program debugging, please refer to the single example.

  • Tracking Examples	Recommended Reference Object Size
    Color Following	3x3cm Building Blocks
    Face Following	Face from a Mobile Phone Image
    AprilTag Machine Code Following	3x3cm Building Blocks
    QR Code Tracking	QR Code Printed on A5 Paper
    Mediapipe Gesture Following	Real-person gestures
    Meidpipe Human Posture Following	Real-person posture

3.2.1 Example 1: "Start following red"

Type "Start following red" in the terminal. The terminal will print the following information.

A window titled frame will open on the VNC screen, displaying the current robot-viewing image.

Move the object slowly, and the robot will follow.

To end the task, press ENTER in the terminal. Press the [Enter] key to continue the conversation, enter [Stop Following], or [End Following].

If the camera doesn't see the red object after 10 seconds, a countdown will appear on the terminal. When the countdown ends, the program will automatically stop following.

After completing a task, the robot will enter a waiting state. Pressing the ENTER key will continue the conversation. The command will be directly passed to the execution layer model, and all conversation history will be retained. We can enter the "End Current Task" command again to end the robot's current task cycle and start a new one.

3.2.2 Example 2: "Follow the Object in My Hand" (Depth Camera)

⚠️ The coordinates obtained by following the object in this example are entirely derived from the inference of the large AI model. Therefore, it is recommended to use a newer model for better results!

Enter "Please follow the object in my hand" in the terminal. The terminal will print the following information:

A window titled frame will open on the VNC screen, displaying the image from the robot's current perspective.

Move the object slowly, and the robot will follow.

If the screen If there are no targets to follow, the program will count down for 10 seconds, a 5-second countdown will be printed on the terminal, and the process will automatically end, indicating that the task has been completed.

To manually end the task, press the ENTER key in the terminal and continue the conversation, entering either [Stop Following] or [End Following].

After completing a task, the robot enters a waiting state. Commands at this point are directly passed to the execution layer model, and all conversation history is retained. You can enter the "End Current Task" command again to end the current task cycle and start a new one.

4. Source Code Parsing

Source code is located at:

Jetson Orin Nano:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

Jetson Nano, Raspberry Pi:

You need to first enter Docker.

xxxxxxxxxx
#NUWA Camera User
/root/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/root/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

RDKX5：

x
#NUWA Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

4.1 Example 1

4.1.1 Nuwa Depth Camera, action_service_nuwa.py

Example 1 uses the seewhat, colcor_follow, and stop_follow() methods in the CustomActionServer class.

The seewhat function primarily retrieves the color image from the depth camera.

The colcor_follow(self, color) function performs color following.

The stop_follow() function issues a stop command to follow.

This section focuses on the colcor_follow(self, color) function, which requires a color parameter, which can be 'red', 'green', 'blue', or 'yellow'.

xxxxxxxxxx
#Start the color line patrol subprocess
    process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_colorTracker','--ros-args','-p',f'colcor:={target_color}'])

The source code path of the subprocess in the program,

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl_depth/yahboomcar_voice_ctrl_depth/follow/voice_colorTracker.py

​x
def colcor_follow(self,color):
    self.colcor_follow_future = Future() #Reset Future object
    color = color.strip("'\"")  # Remove single and double quotes
    if color == 'red':
        target_color = float(1)
    elif color == 'green':
        target_color = float(2)
    elif color == 'blue':
        target_color = float(3)
    elif color == 'yellow':
        target_color = float(4)
    else:
        self.get_logger().info('color_sort:error')
        return
​
    #Start the color line patrol subprocess
    process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_colorTracker','--ros-args','-p',f'colcor:={target_color}'])
    
    #Waiting to stop following instructions
    while not self.colcor_follow_future.done():
        if self.interrupt_flag:
            break
        time.sleep(0.1)
​
    self.kill_process_tree(process_1.pid)
    self.cancel()

When the main model receives the user input of the "Stop Following" or "End Following" command,

or if the target being followed is lost for more than 10 seconds,

the stop_follow method is called, sending the future.done signal. The while not self.colcor_follow_future.done() block in the colcor_follow function then exits. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the execution action is reported to the main model at the execution layer.

4.1.2 USB Camera, action_service_usb.py

Example 1 uses the seewhat, colorFollow, and stop_track() methods in the CustomActionServer class.

• The seewhat function primarily retrieves the camera's color image.
• The colorFollow(self, color) function performs color tracking.
• The stop_track() function issues a stop tracking command.

This section focuses on the colorFollow(self, color) function, which requires a color parameter (red, green, blue, and yellow).

xxxxxxxxxx
#Start the color line patrol subprocess
process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'colorFollow','--ros-args','-p',f'target_color:={target_color}'])

The source code path of the subprocess in the program,

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl/yahboomcar_voice_ctrl/colorFollow.py

xxxxxxxxxx
def colorFollow(self,color):
    try:
        self.colorFollow_future = Future() 
        color = color.strip("'\"")
        if color == 'red':
            target_color = int(1)
        elif color == 'green':
            target_color = int(2)
        elif color == 'blue':
            target_color = int(3)
        elif color == 'yellow':
            target_color = int(4)
        else:
            target_color = int(1)
        process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'colorFollow','--ros-args','-p',f'target_color:={target_color}'])
        while not self.colorFollow_future.done():
            if self.interrupt_flag:
                break
            time.sleep(0.1)
        self.get_logger().info(f'killed process_pid') 
        self.kill_process_tree(process_1.pid)
        self.cancel()     
    except:
        self.get_logger().error('colorFollow Startup failure')
        return

When the main model receives the user input of the "Stop Following" or "End Following" command,

or if the tracking target is lost for more than 10 seconds,

the stop_track method is called, sending the future.done signal. The while not self.colorFollow_future.done() block in the colorFollow function then exits. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the action execution is reported to the main model at the execution layer.

4.2 Example 2

4.2.1 nuwa Depth Camera, action_service_nuwa.py

Example 2 uses the seewhat, KCF_follow, and stop_track methods in the CustomActionServer class.

• The seewhat function primarily obtains the camera's color image.
• The KCF_follow(self,x1,y1,x2,y2) function performs object tracking. - stop_track() issues a stop command to follow the function.

The seewhat function primarily retrieves the camera's color image. The KCF_follow(self,x1,y1,x2,y2) function takes as parameters the coordinates of the upper-left and lower-right vertices of the object's bounding box (the upper-left corner of the image is the pixel origin). For example, the coordinates of the outer bounding box of the identified green square in Example 2 can be found from the large model's response: the upper-left corner is (230, 345) and the lower-right corner is (235, 350).

xxxxxxxxxx
#Start the object tracking subprocess
process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_KCF_Tracker','--ros-args','-p',f'x1:={x1}','-p',f'y1:={y1}','-p',f'x2:={x2}','-p',f'y2:={y2}'])

The startup program source code path is:

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl_depth/yahboomcar_voice_ctrl_depth/kcf/voice_KCF_Tracker.py

xxxxxxxxxx
def KCF_follow(self,x1,y1,x2,y2):
    self.KCF_follow_future = Future() #Reset Future object
    x1 = int(x1)
    y1 = int(y1)
    x2 = int(x2)
    y2 = int(y2)
​
    process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_KCF_Tracker','--ros-args','-p',f'x1:={x1}','-p',f'y1:={y1}','-p',f'x2:={x2}','-p',f'y2:={y2}'])
    # time.sleep(1.0)#Sleep for 2 seconds to wait for the thread to stabilize
​
    while not self.KCF_follow_future.done():
        if self.interrupt_flag:
            break
        time.sleep(0.1)
​
    self.kill_process_tree(process_1.pid)
    self.cancel()

When the main model receives the user input of the "Stop Following" or "End Following" command,

or if the target is lost for more than 10 seconds,

the stop_follow method is called, sending the future.done signal. The while not self.KCF_follow_future.done() block in the KCF_follow function is then exited. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the execution action is reported to the main model at the execution layer.