Welcome to DOFBOT SE repository

Multimodal Visual Understand

Function Description

After the program runs, after waking up the voice module, ask questions about vision or describe the current scene. The program will take a photo of the current environment and upload it to the Alibaba Cloud platform. Then the vision model will parse the image with the question and finally provide an answer.

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py

After waking up the module, give a vision-related command, reference the following example:

xxxxxxxxxx
Please describe what do you see?

The path where the captured photo is saved is: LargeModel_ws/install/largemodel/share/largemodel/resources_file/image.png

Core Code Analysis Seewhat

Please refer to the Core Code Analysis Seewhat content in tutorial 【AI Model-Text Version】-【2.Multimodal Visual Understand】. The voice version and text version have the same action functions; only the input method for task commands is different.