Welcome to DOFBOT-Ultra repository

Multimodal Large Model + Video Understanding

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The web control program backend process must be closed before starting this function.

1. Function Description

After the program starts, input questions about a video via voice. The large model will first record a video, then analyze the video content with the question, and finally reply with the answer. You can also record a video of a specified duration first, and then ask the large model questions related to the video content.

2. Startup

Raspberry Pi version needs to enter docker to run the command. RDK runs the command directly on the host.

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=False

After waking up the module, give voice commands related to visual understanding questions. Refer to the following example:

x
Record a 20-second video

After the buzzer beeps,

After the buzzer beeps again, the recording ends. Then, based on the video content, ask the following question:

x
Did you see the red block in the video I just recorded?

The recorded video file is saved in the /home/yahboom directory inside the Docker container.

3. Core Code Analysis

3.1. Video Recording Function - record_video

Source path:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

xxxxxxxxxx
#Open another terminal, start the Record_Video node program, passing the parameter time_ (video recording duration)
def record_video(self,time_):
    record_time_ = float(time_)
    cmd1 = f"ros2 run largemodel_arm Record_Video --ros-args -p time_:{record_time_:.2f}"
    subprocess.Popen(
        [
            "gnome-terminal",
            "--title=recording",
            "--",
            "bash",
            "-c",
            f"{cmd1}; exec bash",
        ]
    )        
#Raspberry Pi: Record_Video node program source path is /home/yahboom/dofbot_ultra_ws/src/llm/largemodel_arm/largemodel_arm/Record_Video.py, mainly look at the color image callback function,
#RDK: Record_Video node program source path is /home/sunrise/dofbot_ultra_ws/src/llm/largemodel_arm/largemodel_arm/Record_Video.py, mainly look at the color image callback function,

def get_RGBCallBack(self,msg):   
    if self.beep_start==False:
        self.start_time = time.time()
        self.beep_start = True
        self.Arm.Arm_Buzzer_On()
        time.sleep(0.5)
        self.Arm.Arm_Buzzer_Off()
        self.record_done = False
        
    if self.record_done!=True: 
        #Convert image topic data to image data
        rgb_image = self.rgb_bridge.imgmsg_to_cv2(msg,'bgr8')
        if self.out is None:
            #print("-----------")
            height, width, _ = rgb_image.shape
            fourcc = cv2.VideoWriter_fourcc(*'x264')  # Use MP4 encoder
            #Write image frames to video file
            self.out = cv2.VideoWriter(
                self.Path_video, 
                fourcc, 
                15, 
                (width, height)
            )
        self.out.write(rgb_image)
        cv2.imshow('Video Recording', rgb_image)
        key = cv2.waitKey(1)
        elapsed = time.time() - self.start_time 
        #Check recording duration, stop if it exceeds the set recording time
        if elapsed >= self.Time_ :
            cv2.destroyAllWindows()
            self.record_done = True
            self.out.release()
            self.Arm.Arm_Buzzer_On()
            time.sleep(0.5)
            self.Arm.Arm_Buzzer_Off()
            self.largemodel_arm_done_pub.publish(String(data='record_video_done'))

3.2. Video Understanding Function - video_understanding

Source path:

Raspberry Pi:/home/yahboom/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

RDK:/home/sunrise/dofbot_ultra_ws/src/llm/largemodel/largemodel/action_service.py

xxxxxxxxxx
#After the model_service node receives video_handle, it will upload the video to the Alibaba Cloud large model platform, use the multimodal large model to analyze the video content with the question, and finally return the result
def video_understanding(self):
    self.get_logger().info(
                "Publish video handle."
            )
    msg = String(data="video_handle")
    self.video_handle_pub.publish(
        msg
    )  # Normalize, publish video_handle topic, call large model via model_service

4. Combining with Robotic Arm

We can also combine the robotic arm + video understanding to develop new gameplay. For example, prepare several opaque paper cups. After the video starts, place an object under one of the opaque paper cups, then slowly change the positions of the paper cups. After recording the video, ask the large model which cup the object is under, and finally have the robotic arm point to that cup. To test this function, say after waking up:

x
Let's play a game. First record a video, then guess which cup has the red block at the end of the video?

Replace the little yellow duck with an actual object.

The task plan should be:

1. Call the record_video(20) function to record a 20-second video;
2. Call the `video_understanding()` function to analyze the recorded 20-second video and determine under which cup the object is hiding at the end of the video.

If the analysis result shows that an object is indeed hidden under a certain cup, then continue with the following steps:

3. Call the `point_to(x1, y1, x2, y2)` function to point to the cup containing the object, where `x1, y1, x2, y2` are the outer bounding box coordinates of the cup containing the object.