Welcome to DOFBOT-Ultra repository

Multimodal Visual Understand

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this feature, you need to close the web control program backend process.

Note: If no image is displayed, the Camera type may not be set to the correct type.

1. Feature Description

After the program runs, enter visual-related questions or describe the current scene through the terminal. The program will take a photo of the current environment and upload it to the Alibaba Cloud platform. The visual model will then analyze the image with the question and provide an answer.

2. Startup

For the Raspberry Pi version, you need to enter the Docker container and run:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and enter the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then enter questions related to visual understanding in the text_chat terminal. Refer to the following example:

x
Describe the scene you see.

The saved photo path is:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/install/largemodel/share/largemodel/resources_file/image.png

RDK:/home/sunrise/dofbot_ultra_ws/install/largemodel/share/largemodel/resources_file/image.png

3. Core Code Analysis - seewhat

The program calls the seewhat function to take a photo. The source path of this function is:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

​x
def seewhat(self):
    #Save image
    self.save_single_image()
    time.sleep(3.0)
    msg = String(data="seewhat")
    self.seewhat_handle_pub.publish(
        msg
    )  # Normalize, publish seewhat topic, model_service calls the large model

def save_single_image(self):
    """
    Save a single image
    """
    self.IS_SAVING=True
    time.sleep(0.5)
    if self.image_msg is None:
        self.get_logger().warning("No image received yet.")
        return
    try:
        # Convert ROS image message to OpenCV image
        cv_image = self.bridge.imgmsg_to_cv2(self.image_msg, "bgr8")
        # Save the image
        cv2.imwrite(self.image_save_path, cv_image)
        display_thread = threading.Thread(target=self.display_saved_image)
        display_thread.start()

    except Exception as e:
        self.get_logger().error(f"Error saving image: {e}")
    self.IS_SAVING=False