Welcome to DOFBOT-JetsonNANO repository

Multimodal Large Model+Color Blocks Back In Place（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and input the command to remember the current positions of color blocks. After the program completes recording, scramble the positions of the color blocks. Then wake up the voice module again and input the command to end the current recording task. Then wake up the voice module again and input the color block return command. After thinking, the large model will plan an action list, and the program will control the robotic arm to complete the color block return task according to the action functions in the action list.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking up the voice module, input by voice:

xxxxxxxxxx
Remember the current positions of the color blocks

After recording the positions of the color blocks, wake up the voice module again and input by voice:

x
End the current recording task

Then you can scramble the positions of the color blocks. After scrambling, wake up the voice module and input by voice:

x
Return the blocks on the desktop to their original positions

After the buzzer beeps once, it will start gripping the color blocks and returning them to their original positions according to the order recorded in the program. If there are other color blocks on top of the current color block that needs to be returned, it will first remove the color blocks on top, then grip and return to the original position.

3. Task Planning

3.1. Record Current Positions of Color Blocks

1. Call seewhat() to observe the environment;
2. Call seewhat() again to observe the environment, then determine which color block is the top block;
3. If there are two or more piles of color blocks in the environment, select and execute the following content based on whether a color block is a single one. Calculate for each pile of color blocks. Only one action function can be output at a time here:
   (1) If it is a single color block, pass the outer frame coordinates of the color block into the function compute_pose(x1, y1, x2, y2, 'name'), where name represents the color of the color block, and its value can be red, blue, green, or yellow;
   (2) If it is not a single color block, pass the outer frame coordinates of the top color block into the function compute_pose_order(x1, y1, x2, y2, 'name', 'order'), where name represents the color of the color block (with values red, blue, green, or yellow), and order represents the top-to-bottom arrangement order of the color blocks. Its value is a string composed of the initial letters of the colors of each layer of color blocks from top to bottom. For example, if the top-to-bottom arrangement order of the color blocks is red, blue, green, yellow, then the value of order is rbgy, and so on.
4. If there is only one pile of color blocks, pass the outer frame coordinates of the top color block into the function compute_pose_order(x1, y1, x2, y2, 'name', 'order'), where name represents the color of the color block (with values red, blue, green, or yellow), and order represents the top-to-bottom arrangement order of the color blocks. Its value is a string composed of the initial letters of the colors of each layer of color blocks from top to bottom. For example, if the top-to-bottom arrangement order of the color blocks is red, blue, green, yellow, then the value of order is rbgy, and so on.

3.2. Color Block Return

1. First, call the color_back_to_orin() function to control the color blocks to return to their original positions.
2. Call the check_remove(color) function to check whether the target color block exists in the removal list (where color is the color of the target color block, and its value range is red, blue, green, or yellow).
3. If the target color block exists in the removal list, perform the following operations:
   (1) Call the grasp_from_rm_list(color) function to grasp the target color block from the removal list (the meaning of the color parameter is the same as above);
   (2) Pass the color of the target color block as a parameter into the return_to_orin(color) function. For example, when the target color block is red, call return_to_orin(red) (the value range of the color parameter is the same as above).
4. If the target color block does not exist in the removal list, follow the steps below to put the color block back to its original position (only one action function can be output at a time):
   (1) Call the seewhat() function to observe the environment;
   (2) Determine whether there are other color blocks on top of the target color block that needs to be returned to its original position;
   (3) If there are no other color blocks on top of the target color block that needs to be returned to its original position, perform the following steps: ①Call the grasp_obj(x1, y1, x2, y2) function to grasp the target color block that needs to be returned to its original position (where (x1, y1, x2, y2) are the border coordinates of the top surface of the target color block that needs to be returned to its original position);
   ②Pass the color of the target color block as a parameter into the return_to_orin(color) function (the value range of the color parameter is the same as above).
   (4) If there are other color blocks on top of the target color block that needs to be returned to its original position, perform the following steps:
   ①Call the remove_obj(x1, y1, x2, y2, color) function to remove the color block (where (x1, y1, x2, y2) are the border coordinates of the top surface of this color block, and color is the color of this color block, with a value range of red, blue, green, or yellow);
   ②Call the seewhat() function to observe the environment;
   ③If there are no other color blocks above the target color block that needs to be returned to its original position, call the grasp_obj(x1, y1, x2, y2) function to grasp the target color block (where (x1, y1, x2, y2) are the border coordinates of the top surface of the target color block); ④Pass the color of the target color block as a parameter into the return_to_orin(color) function (the value range of the color parameter is the same as above).

4. Core Code Analysis

You can refer to the content in 4. Core Code Analysis from tutorial [18. AI Model - Text Version] - [Multimodal Large Model+Color Blocks Back In Place]. The voice version and text version have the same action functions, only the task command input method is different.