Welcome to DOFBOT SE repository

Multimodal Large Model+video Understanding

Function Description

After the program runs, input video-related questions through the terminal. The large model will first record a video, then parse the video content with the question, and finally reply with the answer to the question. You can also first record a video of specified duration, then ask the large model questions related to the video based on the video content.

Startup

Open the terminal and enter the following command:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking up the module, give a video recording command, reference the following example:

xxxxxxxxxx
Record a 20 seconds video

After the buzzer beeps once,

After the buzzer beeps once, recording ends. Then, based on the video content, wake up the voice module and say:

xxxxxxxxxx
Did you see the red block in the video I just recorded?

The question here should be asked based on the actual recorded video content.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + video Understanding】. The voice version and text version have the same action functions; only the input method for task commands is different.

Combine With Robotic Arm

We can also combine the robotic arm with video understanding to develop new玩法. For example, prepare several opaque paper cups. After the video starts, place an object under one of the opaque paper cups, then slowly change the positions of the paper cups. After the video recording ends, ask the large model which cup the object is finally under, and then have the robotic arm point to that cup. To test this function, after waking up the voice module, input by voice:

xxxxxxxxxx
Let's play a game. First record a video, then guess which cup has the red block at the end of the video?

The program will record a video, then analyze which cup contains the red block, and finally control the robotic arm to point to the cup with the hidden duck.

The task planning should be:

​x
1. Call the `record_video` function, pass in the parameter "20", and record a 20-second video; 

2. Call the `video_understanding` function to analyze the recorded 20-second video, and determine which cup has the red block hidden under it at the end of the video;  

3. Call the `point_to(x1, y1, x2, y2)` function to point to the cup with the red block, where x1, y1, x2, y2 represent the outer frame coordinates of the cup with the red block at the end of the video.