Welcome to DOFBOT SE repository

Multimodal Large Model+robotic Arm Sorting

Function Description

After the program runs, after waking up the voice module, input the color block sorting order by voice. The large model will plan the steps to complete the color block sorting task, then the program will control the robotic arm to sort the color blocks in order and place them at the designated positions.

Startup

Open the terminal and enter the command:

ros2 launch largemodel largemodel_control.launch.py

After waking up the module, input by voice:

xxxxxxxxxx
Sort the color blocks on the desktop in the order of yellow, red, green

The program will sort according to the given order of yellow, red, blue, green. Before sorting the yellow color block, it must first detect whether it is in the removal list. If it is, it will directly grip from the removal list and place it at the yellow color block placement point; if not, it will determine whether there are other color blocks on top of the yellow color block. If there are, it will first remove the other color blocks on top of the yellow color block. If not, it will directly grip and place it at the yellow color block placement point. Other color blocks are also sorted according to this execution order until the task ends.

Task Planning

1. Call check_remove(color) function to check if the current target color block to be sorted exists in the removal list (where color is the color of the current target color block to be sorted, value range is red, blue, green or yellow);

2. If the current target color block to be sorted exists in the removal list, execute the following operations: (1) Call grasp_from_rm_list(color) function to grip the current target color block to be sorted from the removal list (meaning of color parameter is the same as above); (2) Call arm_stack(color) to place the target color block at the specified position (where color parameter meaning is the same as above).

3. If the target color block does not exist in the removal list, follow these steps to sort the current color block to be sorted (only output one action function at a time): (1) Call seewhat() function to find the current target color block to be sorted; (2) Call seewhat() function to determine if there are other objects on top of the current target color block to be sorted; (3) If there are no other objects on top of the current target color block to be sorted, execute the following steps (only output one action function at a time): ① Call grasp_obj(x1, y1, x2, y2) function to grip the current target color block to be sorted (where (x1, y1, x2, y2) are the top surface border coordinates of the current target color block to be sorted); ② Call arm_stack(color) to place the current target color block to be sorted at the specified position (where color parameter meaning is the same as above). (4) If there are other objects on top of the current target color block to be sorted, execute the following steps to first remove that object then sort the current target color block to be sorted, here only one action function can be output at a time: ① Call remove_obj(x1, y1, x2, y2, color) function to remove that object (where (x1, y1, x2, y2) are the top surface border coordinates of that object, color is the color of that object, value range is red, blue, green or yellow); ② Call seewhat() function to observe the environment; ③ If there are no other objects on top of the current target color block to be sorted, call grasp_obj(x1, y1, x2, y2) function to grip the current target color block to be sorted (where (x1, y1, x2, y2) are the top surface border coordinates of the current target color block to be sorted); ④ Call arm_stack(color) to place the target color block at the specified position (where color parameter meaning is the same as above).

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Sorting】. The voice version and text version have the same action functions; only the input method for task commands is different.