Welcome to DOFBOT SE repository

AI Top Chef

1. Function Description

After the program starts, give the large model a command asking what delicious food can be made from the ingredients on the desktop. The program will check the ingredients on the desktop and reply with what delicious dishes can be made. Finally, it will grip the ingredients and place them to the left of the robotic arm.

2. Startup

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and enter the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then in the text_chat terminal, enter the following content:

xxxxxxxxxx
What delicious food can be made from the ingredients on the desktop?

Press Enter. For the voice version, wake up the voice module and directly say "What delicious food can be made from the ingredients on the desktop?" to the voice module. Wait for the large model to think and reply, as shown in the figure below:

The robotic arm will select two ingredients that can be used to make a delicious dish based on what it sees, grip these two ingredients, and place them to the left of the robotic arm.

3. Task Planning

1. Call seewhat() to observe the environment and check what ingredients are on the desktop;

2. Reply with what two ingredients can be used to make a dish and the dish name;

3. Call seewhat() to observe the position of the first ingredient;

4. Call grasp_obj(x1, y1, x2, y2) to grip the first ingredient;

5. Call putdown() to place down the first ingredient;

6. Call seewhat() to observe the position of the second ingredient;

7. Call grasp_obj(x1, y1, x2, y2) to grip the second ingredient;

8. Call putdown() to place down the second ingredient.