Multimodal visual understand(Voice Version)

1. Course Content

1. Learn to use the robot's visual understanding capabilities
2. Learn new key source code

2. Preparation

2.1 Content Description

This course uses the Jetson Orin NANO as an example. For users of the gimbal servo USB camera version, this is used as an example. For Raspberry Pi and Jetson Nano boards, you need to open a terminal on the host computer and enter the command to enter the Docker container. Once inside the Docker container, enter the commands mentioned in this course in the terminal. For instructions on entering the Docker container from the host computer, refer to [01. Robot Configuration and Operation Guide] -- [5.Enter Docker (For JETSON Nano and RPi 5)]. For RDKX5 and Orin boards, simply open a terminal and enter the commands mentioned in this course. >

📝 This example uses model:"qwen/qwen2.5-vl-72b-instruct:free","qwen-vl-latest"

⚠️ The responses from the big model for the same test command may not be exactly the same and may differ slightly from the screenshots in the tutorial. To increase or decrease the diversity of the big model's responses, refer to the section on configuring the decision-making big model parameters in [04.AI Large Model Development] -- [5.Deploy the RAG knowledge base].

3. Running the Example

3.1 Starting the Program

Open the host terminal and start the Dify environment.

sh ~/bringup_dify.sh

For Raspberry Pi 5 and Jetson Nano controllers, you must first enter the Docker container. For RDKX5 and Orin board, this is not necessary.

Open a terminal on the car and enter the following command:

xxxxxxxxxx
ros2 launch multi_brains llm_agent_control.launch.py

After initialization is complete, the following content will be displayed.

3.2 Test Case

• Here are two reference test cases. Users can create their own test commands.

  • Show me what items are in front of you and describe their functions.
  • Please look in front of you to see if there are blue squares and a pack of tissues. If so, nod; if not, shake your head.

3.2.1 Example 1

First, use "Hi, yahboom" to wake the robot. The robot responds: "please give me order" After the robot responds, the buzzer beeps briefly (beep—). The user can then speak. The robot will then detect active sound, printing 1 if there is sound activity and - if there is no sound activity. When the speech ends, it will detect the end of the sound and stop recording if there is silence for more than 450ms.

The following figure shows the Voice Active Detection (VAD):

The robot will first communicate with the user, then follow the instructions. The terminal will print the following information:

The presence of finishtask() in the action list indicates that the execution layer model has determined that the robot has completed the user's instructions and entered the waiting state. At this point, you can wake yahboom up again to complete the current task:

3.2.2 Example 2

Similar to Example 1, first wake the robot with "Hi, yahboom." After the robot responds, the buzzer beeps briefly (beep). The user can then speak. After speaking, the robot responds and moves according to the command.

Example 2 uses the gimbal servo USB camera version as an example. Nodding and shaking the head can also be used to indicate forward or reverse movement, etc.

4. Source Code Analysis

Source code located at:

Jetson Orin Nano host:

xxxxxxxxxx
#NUWA camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

Jetson Nano, Raspberry Pi host:

You need to first enter Docker.

xxxxxxxxxx
#NUWA camera user
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/root/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

RDKX5：

xxxxxxxxxx
#NUWA camera user
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_nuwa.py
#USB camera user
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/action_service_usb.py

xxxxxxxxxx
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/src/multi_brains/multi_brains/model_service.py

action_service.py

In the init_ros_communication method of the CustomActionServer class, a topic publisher for the seewhat_handle topic is created to publish visual processing signals.

xxxxxxxxxx
# Create a publisher and publish the seewhat_handle topic
self.seewhat_handle_pub = self.create_publisher(String, 'seewhat_handle', 1)

• USB camera, action_service_usb.py

Creates a topic subscriber for /usb_cam/image_raw to receive the camera's color image.

xxxxxxxxxx
#Image topic subscriber
self.subscription = self.create_subscription(
    Image, self.image_topic, self.image_callback, 1
)

• nuwa depth camera, action_service_nuwa.py

  Creates a topic subscriber for the /ascamera_hp60c/camera_publisher/rgb0/image command to receive color images from the depth camera.

xxxxxxxxxx
#Image topic subscriber
self.subscription = self.create_subscription(
    Image, self.image_topic, self.image_callback, 1
)

The image_callback callback function continuously subscribes to the depth camera's color image.

When the action server calls the seewhat method, it publishes a signal to the seewhat_handle topic, notifying the model server's model_service node to upload an image to the multimodal large model.

xxxxxxxxxx
def seewhat(self):
    self.save_single_image()
    msg = String(data="seewhat")
    self.seewhat_handle_pub.publish(
        msg
    )  # Normalize, publish to the seewhat topic, and call the large model from model_service.

At this point, the save_single_image method is called to save an image.png file.

Image save path:

jetson Orin Nano :

x
/home/jetson/yahboomcar_ros2_ws/yahboomcar_ws/cache/

For the Jetson Nano and Raspberry Pi hosts, you must first enter Docker:

x
/root/yahboomcar_ros2_ws/yahboomcar_ws/cache/

RDKX5：

x
/home/sunrise/yahboomcar_ros2_ws/yahboomcar_ws/cache/

​x
def image_callback(self, msg):#Image callback function
        self.image_msg = msg
​
def save_single_image(self):
    """
    保存一张图片 / Save a single image
    """
    if self.image_msg is None:
        self.get_logger().warning("No image received yet.")  # 尚未接收到图像...
        return
    try:
        # 将ROS图像消息转换为OpenCV图像 / Convert ROS image message to OpenCV image
        cv_image = self.bridge.imgmsg_to_cv2(self.image_msg, "bgr8")
        # 保存图片 / Save the image
        cv2.imwrite(self.image_save_path, cv_image)
        display_thread = threading.Thread(target=self.display_saved_image)
        display_thread.start()
​
    except Exception as e:
        self.get_logger().error(f"Error saving image: {e}")  # Error saving image...

model_service.py

In the init_ros_communication method of the LargeModelService class,

a subscriber to the topic "seewhat_handle" is created to receive visual processing signals.

xxxxxxxxxx
#Create a Seehat subscriber
self.seewhat_sub = self.create_subscription(String,'seewhat_handle', self.seewhat_callback,1)

When the seewhat_callback callback function receives the visual processing signal, it calls the dual_large_model_mode method, and then calls the instruction_process method to request inference and upload an image from the depth camera to the multimodal large model.

xxxxxxxxxx
def seewhat_callback(self, msg):
    if msg.data == "seewhat":
        if (
            self.regional_setting == "China"
        ):  # 在线模型推理方式：决策层推理+执行层监督 / Online model inference method: Decision layer reasoning + Execution layer supervision
            self.dual_large_model_mode(type="image")
        else:
            self.dual_large_model_international_model(type="image")

xxxxxxxxxx
def instruction_process(self, prompt, type, conversation_id=None):
    """
    根据输入信息的类型（文字/图片），构建不同的请求体进行推理，并返回结果）
    Based on the type of input information (text/image), construct different request bodies for inference and return the result.
    """
    json_str = None
    prompt_seewhat = self.prompt_dict[self.language].get("prompt_2")
    if self.regional_setting == "China":  
        if type == "text":
            raw_content = self.model_client.multimodalinfer(prompt)
        elif type == "image":
            raw_content = self.model_client.multimodalinfer(
                prompt_seewhat, image_path=self.image_save_path
            )
        json_str = self.extract_json_content(raw_content)
    elif self.regional_setting == "international":  
        if type == "text":
            result = self.model_client.TaskExecution(
                input=prompt,
                map_mapping=self.map_mapping,
                language=self.language_dict[self.language],
                conversation_id=conversation_id,
            )
            if result[0]:
                json_str = self.extract_json_content(result[1])
                self.conversation_id = result[2]
            else:
                self.get_logger().info(f"ERROR:{result[1]}")
        elif type == "image":
            result = self.model_client.TaskExecution(
                input=prompt_seewhat,
                map_mapping=self.map_mapping,
                language=self.language_dict[self.language],
                image_path=self.image_save_path,
                conversation_id=conversation_id,
            )
            if result[0]:
                json_str = self.extract_json_content(result[1])
                self.conversation_id = result[2]
            else:
                self.get_logger().info(f"ERROR:{result[1]}")