Welcome to DOFBOT-Ultra repository

Multimodal Visual Understand

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The web control program backend process must be closed before running this function.

1. Function Description

After the program starts, ask visual questions or describe the current scene via voice input. The program will take a photo of the current environment and upload it to the Alibaba Cloud platform, then the visual model will analyze the image with the question and finally provide an answer.

2. Startup

Raspberry Pi version needs to enter docker to run the command. RDK runs the command directly on the host.

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=False

After waking up the module, wait for a "beep" sound from the buzzer, then give voice commands by speaking. Refer to the following example:

xxxxxxxxxx
Please describe what do you see?

The saved photo path is:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/install/largemodel/share/largemodel/resources_file/image.png

RDK:/home/sunrise/dofbot_ultra_ws/install/largemodel/share/largemodel/resources_file/image.png

3. Core Code Analysis - seewhat

The program calls the seewhat function to take a photo. The source path of this function is:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

​x
def seewhat(self):
    #Save image
    self.save_single_image()
    time.sleep(3.0)
    msg = String(data="seewhat")
    self.seewhat_handle_pub.publish(
        msg
    )  # Normalize, publish seewhat topic, call large model via model_service

def save_single_image(self):
    """
    Save a single image
    """
    self.IS_SAVING=True
    time.sleep(0.5)
    if self.image_msg is None:
        self.get_logger().warning("No image received yet.")
        return
    try:
        # Convert ROS image message to OpenCV image
        cv_image = self.bridge.imgmsg_to_cv2(self.image_msg, "bgr8")
        # Save the image
        cv2.imwrite(self.image_save_path, cv_image)
        display_thread = threading.Thread(target=self.display_saved_image)
        display_thread.start()

    except Exception as e:
        self.get_logger().error(f"Error saving image: {e}")
    self.IS_SAVING=False