Multimodal visual understand + visual line patroling(Voice Version)

1. Course Content

1. Learn to use the robot's visual understanding capabilities in conjunction with line-following autonomous driving.
2. Analyze newly discovered key source code.

2. Preparation

2.1 Content Description

This course uses the Jetson Orin Nano as an example. This example uses the gimbal servo USB camera version. For Raspberry Pi and Jetson Nano boards, you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this course in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For RDKX5 and Orin boards, simply open a terminal and enter the commands mentioned in this course.

📝 This example uses model:"qwen/qwen2.5-vl-72b-instruct:free","qwen-vl-latest"

⚠️ The responses from the big model for the same test command may not be exactly the same and may differ slightly from the screenshots in the tutorial. If you need to increase or decrease the diversity of the big model's responses, refer to the section on configuring the decision-making big model parameters in the [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base].

⚡️ It is recommended that you first try the previous visual example. This example adds voice functionality to the singleton, and while the functionality is largely the same, I will not elaborate on the program implementation, code debugging, or results in detail! ! !

3. Running the Example

3.1 Starting the Program

Open the host terminal and start the Dify environment.

xxxxxxxxxx
sh ~/bringup_dify.sh

For Raspberry Pi 5 and Jetson Nano controllers, you must first enter the Docker container. For RDKX5 and Orin board, this is not necessary.

Open a terminal on the vehicle and enter the following command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed:

3.2 Test Case

• This is a reference test case; users can create their own dialogue commands.

  • Start color line tracking

    Colors include: red, green, blue, and yellow (color calibration is required according to the AI Large Model Preparation tutorial).

3.2.1 Example: "Start Green Line Patrol"

First, use "Hi, yahboom" to wake the robot. The robot responds: "I'm here, please tell me." After the robot responds, the buzzer beeps briefly (beep—). The user can speak, and the robot will detect sound activity. If there is sound activity, it will print 1; if there is no sound activity, it will print -. When the speech ends, it will detect the end of the voice and stop recording if there is silence for more than 450ms.

The dynamic sound detection (VAD) is shown in the image below:

The robot will first communicate with the user, then respond to the user's instructions. The terminal will also print the following information:

For users of the gimbal servo USB camera version, upon program startup, the servo gimbal will default to pointing downward.

Depth Camera users need to adjust the camera angle appropriately.

A window titled frame will open on the VNC screen, displaying the image from the robot's current perspective.

Then, speed calculation begins, and the robot automatically patrols the line. If it encounters an obstacle during patrol, it stops and the buzzer sounds.

If there are no targets to track on screen, the program will count down for 10 seconds, a 5-second countdown will be printed on the terminal, and the process will automatically end, indicating the mission is complete.

To manually end the mission, wake up the robot with the voice command "Hi, yahboom" The robot will respond, "I'm here, please." This will interrupt the program, automatically terminate the process, and allow you to proceed to the next command.

The robot will now enter a free conversation state, but all conversation history will be retained. You can wake Xiaoya up again and select "End Current Mission" to end the current mission cycle, clear the conversation history, and start a new one.

4. Source Code Analysis

Source code location:

Jetson Orin Nano host:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

Jetson Nano, Raspberry Pi host:

You need to first enter Docker.

x
#NUWA Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

RDKX5：

#NUWA Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB Camera User
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

4.1 USB Camera, action_service_usb.py

• Example 1 uses the seewhat, follow_line(self, color), and stop_track methods in the CustomActionServer class.

  • The seewhat function primarily retrieves the camera's color image.
  • The follow_line(self, color) function performs color line tracking.
  • stop_track() issues a stop tracking command function.

This section focuses on the follow_line(self, color) function, which requires a color parameter, which can be 'red', 'green', 'blue', or 'yellow'.

xxxxxxxxxx
# Start the line-following autonomous driving subprocess program
process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'follow_line', '--ros-args', '-p', f'target_color:={target_color}'])

The startup program source code path is:

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl/yahboomcar_voice_ctrl/follow_line.py

xxxxxxxxxx
def follow_line(self,color):
    try:
        self.follow_line_future = Future()
        color = color.strip("'\"")
        if color == 'red':
            target_color = int(1)
        elif color == 'green':
            target_color = int(2)
        elif color == 'blue':
            target_color = int(3)
        elif color == 'yellow':
            target_color = int(4)
        else:
            target_color = int(1)
        process_1 = subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl', 'follow_line','--ros-args','-p',f'target_color:={target_color}'])
        while not self.follow_line_future.done():
            if self.interrupt_flag:
                break
            time.sleep(0.1)
        self.get_logger().info(f'killed process_pid')   
        self.kill_process_tree(process_1.pid)
        self.cancel() 
    except:
        self.get_logger().error('follow_line Startup failure')
        return

When the main model receives the user input of the [Stop Tracking] or [End Tracking] command,

or if the tracking target is lost for more than 10 seconds,

the stop_track method is called, sending the future.done signal. The while not self.follow_line_future.done() block in the colorTrack function then exits. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the action execution is reported to the main model at the execution layer.

4.2 nuwa Depth Camera, action_service_nuwa.py

• Example 1 uses the seewhat, follow_line(self, color), and stop_follow methods in the CustomActionServer class.

  • The seewhat function primarily retrieves the color image from the depth camera.
  • The follow_line(self, color) function performs color line tracking.
  • stop_follow() issues a stop command to follow the function.

This section focuses on the follow_line(self, color) function, which requires a color parameter, which can be 'red', 'green', 'blue', or 'yellow'.

xxxxxxxxxx
# Start the line-following autonomous driving subprocess program.
process_1= subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_follow_line','--ros-args','-p',f'colcor:={target_color}'])

The startup program source code path is:

~/yahboomcar_ros2_ws/yahboomcar_ws/src/yahboomcar_voice_ctrl_depth/yahboomcar_voice_ctrl_depth/follow/voice_follow_line.py

​x
def follow_line(self,color):
    self.follow_line_future = Future() #Reset Future object 
    color = color.strip("'\"")  # Remove single and double quotes
    if color == 'red':
        target_color = float(1)
    elif color == 'green':
        target_color = float(2)
    elif color == 'blue':
        target_color = float(3)
    elif color == 'yellow':
        target_color = float(4)
    else:
        self.get_logger().info('color_sort:error')
        return
    process_1= subprocess.Popen(['ros2', 'run', 'yahboomcar_voice_ctrl_depth', 'voice_follow_line','--ros-args','-p',f'colcor:={target_color}'])
    while not self.follow_line_future.done():
        if self.interrupt_flag:
            break
        time.sleep(0.1)
​
    self.kill_process_tree(process_1.pid)
    self.cancel()

When the main model receives the user input of the "Stop Following" or "End Following" command,

or if the tracking target is lost for more than 10 seconds,

the stop_follow method is called, sending the future.done signal. The while not self.follow_line_future.done() block in the follow_line function is then exited. The kill_process_tree method is then called to recursively kill the child process tree. Finally, the status of the execution action is reported to the main model at the execution layer.