Welcome to DOFBOT SE repository

Multimodal Large Model+robotic Arm Handling

Function Description

After the program runs, after waking up the voice module, input color block handling commands by voice. The large model will plan the actions to complete the color block handling task, then the program will control the robotic arm to handle the color blocks in order.

Startup

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py

After waking up the module, input command by voice:

xxxxxxxxxx
Place the green block on the desktop above the red block

The available placement directions are: front, back, left, right, and top - these five directions.

Task Planning

1. Call seewhat() to observe the environment and check the positions of the green block and the red block.

2. Call grasp_obj(x1, y1, x2, y2) to grasp the green block, where (x1, y1, x2, y2) are the outer frame coordinates of the green block.

3. Call change_pose(x1, y1, x2, y2, "green", "red", 5, tar_size) to place the green block on top of the red block, where (x1, y1, x2, y2) are the outer frame coordinates of the red block, tar_size indicates the height of the red block.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Handling】. The voice version and text version have the same action functions; only the input method for task commands is different.

Advanced Play

We can have the robotic arm complete a more complex task: remember the current position of the color block, then place the color block on another color block, then put this block back to its original position. After waking up the module, input by voice:

xxxxxxxxxx
Remember the current position of the red square, then place the red square on top of the yellow square, and finally return the red square to its original position.

Task Planning

1. Call seewhat() to observe the environment;

2. Call the compute_pose(x1, y1, x2, y2) function to record the current position of the yellow block;

3. Call grasp_obj(x1, y1, x2, y2) to grasp the yellow block, where x1, y1, x2, y2 are the outer frame coordinates of the yellow block;

4. Call seewhat() to observe the environment and check the position of the red block;

5. Call the change_pose(x1, y1, x2, y2, "yellow", "red", side, tar_size) function to place the yellow block on top of the red block, where x1, y1, x2, y2 are the outer frame coordinates of the red block, side indicates the placement position, and tar_size indicates the height of the red block;

6. Call grasp_from_down_list("yellow") to find the current position of the yellow block in the placement list;

7. Call the return_to_orin("color") function, where "color" is the color of the target block to be reset, valid values are "red", "blue", "green", or "yellow".

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Handling】. The voice version and text version have the same action functions; only the input method for task commands is different.