Welcome to DOFBOT-Pi repository

Multimodal Visual Understanding（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and ask questions about vision or describe the current scene. The program will capture an image of the current environment and upload it to the Alibaba Cloud platform, then the vision model will analyze the image with the question and finally provide an answer.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

x
ros2 launch largemodel largemodel_control.launch.py

After waking up the module, give vision-related commands. You can refer to the following example,

x
Please describe what do you see?

The captured photo is saved at: LargeModel_ws/install/largemodel/share/largemodel/resources_file/image.png

3. Core Code Analysis seehat

You can refer to the content in 3. Core Code Analysis seehat from tutorial [17. AI Model - Text Version] - [2. Multimodal Visual Understand]. The voice version and text version have the same action functions, only the task command input method is different.