Welcome to DOFBOT SE repository

Multimodal Large Model+Color Blocks Back In Place

Function Description

After the program runs, after waking up the voice module, input the command to remember the current positions of the color blocks by voice. After the program finishes recording, shuffle the positions of the color blocks. Then wake up the voice module again and input by voice to end the current recording task. Then wake up the voice module again and input the color blocks return-to-place command by voice. After the large model thinks it through, it will plan an action list, and the program will control the robotic arm to complete the color blocks return-to-place task based on the action functions in the action list.

Startup

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py

After waking up the voice module, input by voice:

xxxxxxxxxx
Remember the current positions of the color blocks

After recording the color block positions, after waking up the voice module, input by voice:

xxxxxxxxxx
End the current recording task

Then you can shuffle the positions of the color blocks. After shuffling, after waking up the voice module, input by voice:

xxxxxxxxxx
Return the blocks on the desktop to their original positions

After the buzzer beeps once, it starts gripping the color blocks according to the order in which the color block positions were recorded in the program and placing them back to their original positions. If there are other color blocks on top of the color block that needs to be returned to its original position, it will first remove the color blocks on top, then grip and place back to the original position.

Task Planning

Record Current Positions of Color Blocks

1. Call seewhat() to observe the environment;

2. Call seewhat() again to observe the environment, then determine which color block is the top block;

3. If there are two or more piles of color blocks in the environment, select and execute the following content based on whether a color block is a single one. Calculate for each pile of color blocks. Only one action function can be output at a time here:
   (1) If it is a single color block, pass the outer frame coordinates of the color block into the function compute_pose(x1, y1, x2, y2, 'name'), where name represents the color of the color block, and its value can be red, blue, green, or yellow;
   (2) If it is not a single color block, pass the outer frame coordinates of the top color block into the function compute_pose_order(x1, y1, x2, y2, 'name', 'order'), where name represents the color of the color block (with values red, blue, green, or yellow), and order represents the top-to-bottom arrangement order of the color blocks. Its value is a string composed of the initial letters of the colors of each layer of color blocks from top to bottom. For example, if the top-to-bottom arrangement order of the color blocks is red, blue, green, yellow, then the value of order is rbgy, and so on.

4. If there is only one pile of color blocks, pass the outer frame coordinates of the top color block into the function compute_pose_order(x1, y1, x2, y2, 'name', 'order'), where name represents the color of the color block (with values red, blue, green, or yellow), and order represents the top-to-bottom arrangement order of the color blocks. Its value is a string composed of the initial letters of the colors of each layer of color blocks from top to bottom. For example, if the top-to-bottom arrangement order of the color blocks is red, blue, green, yellow, then the value of order is rbgy, and so on.

Color Blocks Back In Place

1. First, call the color_back_to_orin() function to control the color blocks to return to their original positions.

   Multimodal Large Model+robotic arm handling

   Multimodal Large Model + Intent Prediction + Robotic Arm Gripping