Welcome to DOFBOT-JetsonNANO repository

Multimodal Large Model+robotic arm handling（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and input color block handling commands by voice. The large model will plan the actions to complete the color block handling task, then the program will control the robotic arm to handle the color blocks in sequence.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking up the module, input the command by voice:

xxxxxxxxxx
Put the green block on top of the red block

The block can be placed in one of the five positions: front, back, left, right, or top.

The program will first take a photo to get the positions of the green block and red block; then it will control the robotic arm to grip the green block and place it on top of the red block.

3. Task Planning

1. Call seewhat() to observe the environment and check the positions of the green block and the red block.
2. Call grasp_obj(x1, y1, x2, y2) to grasp the green block, where (x1, y1, x2, y2) are the outer frame coordinates of the green block.
3. Call change_pose(x1, y1, x2, y2, "green", "red", 5, tar_size) to place the green block on top of the red block, where (x1, y1, x2, y2) are the outer frame coordinates of the red block, tar_size indicates the height of the red block.

4. Core Code Analysis

You can refer to the content in 4. Core Code Analysis from tutorial [18. AI Model - Text Version] - [Multimodal Large Model+robotic arm handling]. The voice version and text version have the same action functions, only the task command input method is different.

5. Advanced Gameplay

We can have the robotic arm complete a more complex task: remember the current position of the color block, then place the color block on another block, and finally put this block back to its original position. After waking up the module, input by voice:

x
Remember the current position of the red square, then place the red square on top of the yellow square, and finally return the red square to its original position.

The program will first take a photo to find the red block and calculate its current position, then take another photo to find the red block and yellow block, then control the robotic arm to put the red block on top of the yellow block. Then grip the red block and finally put the red block back to its original position.

5.1. Task Planning

1. Call seewhat() to observe the environment;
2. Call the compute_pose(x1, y1, x2, y2) function to record the current position of the yellow block;
3. Call grasp_obj(x1, y1, x2, y2) to grasp the yellow block, where x1, y1, x2, y2 are the outer frame coordinates of the yellow block;
4. Call seewhat() to observe the environment and check the position of the red block;
5. Call the change_pose(x1, y1, x2, y2, "yellow", "red", side, tar_size) function to place the yellow block on top of the red block, where x1, y1, x2, y2 are the outer frame coordinates of the red block, side indicates the placement position, and tar_size indicates the height of the red block;
6. Call grasp_from_down_list("yellow") to find the current position of the yellow block in the placement list;
7. Call the return_to_orin("color") function, where "color" is the color of the target block to be reset, valid values are "red", "blue", "green", or "yellow".

5.2. Core Code Analysis

You can refer to the content in 5.2. Core Code Analysis from tutorial [18. AI Model - Text Version] - [Multimodal Large Model+robotic arm handling]. The voice version and text version have the same action functions, only the task command input method is different.