Multimodal visual understand + visual line patroling(Text Version)

1. Course Content

1. Learn to use the robot's visual understanding capabilities in conjunction with line-following autonomous driving.
2. Analyze newly discovered key source code.

2. Preparation

2.1 Content Description

This course uses the Jetson Orin NANO as an example. This example uses the gimbal servo USB camera version. For Raspberry Pi and Jetson Nano boards, you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this course in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For RDK X5 and Orin boards, simply open a terminal and enter the commands mentioned in this course.

📝 This example uses model: "qwen/qwen2.5-vl-72b-instruct:free", "qwen-vl-latest"

⚠️ The responses from the large model for the same test command may not be exactly the same and may differ slightly from the screenshots in the tutorial. To increase or decrease the diversity of the large model's responses, refer to the section on configuring the decision-making large model parameters in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base] .

⚡️ It is recommended that you first try the previous visual example. This example adds voice functionality to the singleton example, and while the functionality is largely the same, I will not elaborate on the implementation process, code debugging, or results.

3. Running the Example

3.1 Starting the Program

Open the host terminal and start the Dify environment:

xxxxxxxxxx
sh ~/bringup_dify.sh

For Raspberry Pi 5 and Jetson Nano controllers, you must first enter the Docker container. For RDKX5 and Orin board, this is not necessary.

Open a terminal on the vehicle and enter the following command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py text_chat_mode:=True

After initialization, the following content will be displayed:

Open another terminal and start the program:

xxxxxxxxxx
ros2 run text_chat text_chat

3.2 Test Cases

Here are some reference test cases; users can create their own dialogue commands.

• Start Color Line Patrol

Colors include: red, green, blue, and yellow. (Color calibration is required according to the AI Large Model Preparation tutorial.)

⚠️ Please do not include a period or any other characters at the end of the text you enter!

3.2.1 Example: "Start Green Line Patrol"

• For users of the gimbal, servo, and USB camera versions, when the program starts, the gimbal and servo will be facing downward by default.
• For users of the depth camera version, you need to adjust the camera angle appropriately.

Enter "Start Green Line Patrol" in the terminal. The terminal will print the following information:

A window titled frame will open on the VNC screen, displaying the image from the current robot's perspective.

Then, the robot will begin calculating its speed and automatically patrol the line. If it encounters an obstacle during patrol, it will stop and the buzzer will sound.

If there are no targets to track in the image, the program will count down for 10 seconds, a 5-second countdown will be printed on the terminal, and the process will automatically end, indicating that the task is complete.

To manually end the task, press the ENTER key in the terminal and continue the conversation.

• For USB Camera users, use [Stop Tracking], [End Tracking], or [Stop Line Patrol]
• For Depth Camera users, use [Stop Following], [End Following], or [Stop Line Patrol]

After completing a task, the robot enters a waiting state. In this state, commands are directly passed to the execution layer model, and all conversation history is retained. We can enter the "End Current Task" command again to end the robot's current task cycle and start a new one.

4. Source Code Analysis

Source code location:

Jetson Orin Nano:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

Jetson Nano, Raspberry Pi:

You need to first enter Docker.

xxxxxxxxxx
#NUWA Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

RDKX5：

xxxxxxxxxx
#NUWA相机用户
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB相机用户
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

4.1 USB Camera, action_service_usb.py

• Example 1 uses the seewhat, follow_line(self, color), and stop_track methods in the CustomActionServer class.

  • The seewhat function primarily retrieves the camera's color image.
  • The follow_line(self, color) function performs color line tracking.
  • stop_track() issues a stop tracking command function.

This section focuses on the follow_line(self, color) function, which requires a color parameter, which can be 'red', 'green', 'blue', or 'yellow'.

xxxxxxxxxx
# Start the line-following autonomous driving subprocess program
process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'follow_line', '--ros-args', '-p', f'target_color:={target_color}'])

The startup program source code path is:

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl/yahboomcar_voice_ctrl/follow_line.py

xxxxxxxxxx
def follow_line(self,color):
    try:
        self.follow_line_future = Future()
        color = color.strip("'\"")
        if color == 'red':
            target_color = int(1)
        elif color == 'green':
            target_color = int(2)
        elif color == 'blue':
            target_color = int(3)
        elif color == 'yellow':
            target_color = int(4)
        else:
            target_color = int(1)
        process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'follow_line','--ros-args','-p',f'target_color:={target_color}'])
        while not self.follow_line_future.done():
            if self.interrupt_flag:
                break
            time.sleep(0.1)
        self.get_logger().info(f'killed process_pid')   
        self.kill_process_tree(process_1.pid)
        self.cancel() 
    except:
        self.get_logger().error('follow_line Startup failure')
        return

When the main model receives the user input of the [Stop Tracking] or [End Tracking] command,

or if the tracking target is lost for more than 10 seconds,

the stop_track method is called, sending the future.done signal. The while not self.follow_line_future.done() block in the colorTrack function then exits. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the action execution is reported to the main model at the execution layer.

4.2 nuwa Depth Camera, action_service_nuwa.py

• Example 1 uses the seewhat, follow_line(self, color), and stop_follow methods in the CustomActionServer class.
• The seewhat function primarily retrieves the color image from the depth camera.
• The follow_line(self, color) function performs color line tracking.
• stop_follow() issues a stop command to follow the line.

This section focuses on the follow_line(self, color) function, which requires a color parameter, which can be 'red', 'green', 'blue', or 'yellow'.

xxxxxxxxxx
#Start the line patrol automatic driving subprocess program
process_1= subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_follow_line','--ros-args','-p',f'colcor:={target_color}'])

The startup program source code path is:

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl_depth/yahboomcar_voice_ctrl_depth/follow/voice_follow_line.py

def follow_line(self,color):
    self.follow_line_future = Future() #Reset Future object
    color = color.strip("'\"")  # Remove single and double quotes
    if color == 'red':
        target_color = float(1)
    elif color == 'green':
        target_color = float(2)
    elif color == 'blue':
        target_color = float(3)
    elif color == 'yellow':
        target_color = float(4)
    else:
        self.get_logger().info('color_sort:error')
        return
    process_1= subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_follow_line','--ros-args','-p',f'colcor:={target_color}'])
    while not self.follow_line_future.done():
        if self.interrupt_flag:
            break
        time.sleep(0.1)
​
    self.kill_process_tree(process_1.pid)
    self.cancel()

When the main model receives the user input of the "Stop Following" or "End Following" command,

or if the tracking target is lost for more than 10 seconds,

the stop_follow method is called, sending the future.done signal. The while not self.follow_line_future.done() block in the follow_line function is then exited. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the execution action is reported to the main model at the execution layer.