Welcome to DOFBOT-Ultra repository

Multimodal Large Model + Video Understanding

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before running the feature, you need to close the web control program backend process.

1. Feature Description

After the program runs, enter questions about a video through the terminal. The large model will first record a video, then analyze the video content with the question, and finally return the answer. You can also record a video of a specified duration first, then ask the large model questions related to the video content.

2. Startup

For the Raspberry Pi version, you need to enter the Docker container and run:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and enter the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then enter visual understanding questions in the text_chat terminal. Refer to the following example:

xxxxxxxxxx
Record a 20-second video

After the buzzer beeps,

After the buzzer beeps again, recording ends. Then, based on the video content, enter the following:

xxxxxxxxxx
Did you see my phone in the video I just recorded?

The recorded video file is saved in the /home/yahboom directory inside the Docker container. For RDK, it is saved in the /home/sunrise directory on the host machine.

3. Core Code Analysis

3.1. Video Recording Function - record_video

Source path:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

xxxxxxxxxx
#Open another terminal, start the Record_Video node program, pass in the parameter time_, which is the video recording duration
def record_video(self,time_):
    record_time_ = float(time_)
    cmd1 = f"ros2 run largemodel_arm Record_Video --ros-args -p time_:{record_time_:.2f}"
    subprocess.Popen(
        [
            "gnome-terminal",
            "--title=recording",
            "--",
            "bash",
            "-c",
            f"{cmd1}; exec bash",
        ]
    )        
#Raspberry Pi: The source path of the Record_Video node program is /home/yahboom/dofbot_ultra_ws/src/llm/largemodel_arm/largemodel_arm/Record_Video.py. Mainly look at the color image callback function.
#RDK: The source path of the Record_Video node program is /home/sunrise/dofbot_ultra_ws/src/llm/largemodel_arm/largemodel_arm/Record_Video.py. Mainly look at the color image callback function.

def get_RGBCallBack(self,msg):   
    if self.beep_start==False:
        self.start_time = time.time()
        self.beep_start = True
        self.Arm.Arm_Buzzer_On()
        time.sleep(0.5)
        self.Arm.Arm_Buzzer_Off()
        self.record_done = False
        
    if self.record_done!=True: 
        #Convert image topic data to image data
        rgb_image = self.rgb_bridge.imgmsg_to_cv2(msg,'bgr8')
        if self.out is None:
            #print("-----------")
            height, width, _ = rgb_image.shape
            fourcc = cv2.VideoWriter_fourcc(*'x264')  # Use MP4 encoder
            #Write image frames to video file
            self.out = cv2.VideoWriter(
                self.Path_video, 
                fourcc, 
                15, 
                (width, height)
            )
        self.out.write(rgb_image)
        cv2.imshow('Video Recording', rgb_image)
        key = cv2.waitKey(1)
        elapsed = time.time() - self.start_time 
        #Check recording duration, stop if it exceeds the set time
        if elapsed >= self.Time_ :
            cv2.destroyAllWindows()
            self.record_done = True
            self.out.release()
            self.Arm.Arm_Buzzer_On()
            time.sleep(0.5)
            self.Arm.Arm_Buzzer_Off()
            self.largemodel_arm_done_pub.publish(String(data='record_video_done'))

3.2. Video Understanding Function - video_understanding

Source path:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

xxxxxxxxxx
#After the model_service node receives video_handle, it sends the video to the Alibaba Cloud large model platform, uses the multimodal large model to analyze the video content with the question, and finally returns the result
def video_understanding(self):
    self.get_logger().info(
                "Publish video handle."
            )
    msg = String(data="video_handle")
    self.video_handle_pub.publish(
        msg
    )  # Normalize, publish video_handle topic, model_service calls the large model

4. Combining with the Robotic Arm

We can also combine robotic arm + video understanding to develop new gameplay. For example, prepare several opaque paper cups. After the video starts, we place an object under one of the opaque paper cups, then slowly change the positions of the paper cups. After the video recording ends, we ask the large model which cup the object is under at the end, and finally make the robotic arm point to that cup. To test this function, you can input in the terminal:

x
Let's play a game. First record a video, then guess which cup has the red block at the end of the video? 

Replace the red block with an actual object.

The task plan should be:

1. Call the `record_video` function, pass in the parameter "20", and record a 20-second video; 

2. Call the `video_understanding` function to analyze the recorded 20-second video, and determine which cup has the red block hidden under it at the end of the video;  

3. Call the `point_to(x1, y1, x2, y2)` function to point to the cup with the red block, where x1, y1, x2, y2 represent the outer frame coordinates of the cup with the red block at the end of the video.