Welcome to DOFBOT-Pi repository

AI Top Chef

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program starts, give the large model an instruction, ask the large model what delicious food can be made with the ingredients on the desktop. The program will check the ingredients on the desktop and reply with what delicious food dishes can be made, and finally will grip the ingredients and place them on the left side of the robotic arm.The width of the ingredients should preferably not exceed 3 centimeters.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

x
ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and input the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then input the following content in the text_chat terminal:

xxxxxxxxxx
Check what delicious food can be made with the ingredients on the desktop

Press Enter. If it's the voice version, wake up the voice module and directly say to the voice module "Check what delicious food can be made with the ingredients on the desktop", then wait for the large model to think and reply, as shown in the figure below:

The robotic arm will select two ingredients that can make delicious food based on what it sees, and will grip these two ingredients and place them on the left side of the robotic arm.

3. Task Planning

1. Call seewhat() to observe the environment and see what ingredients are on the desktop.
2. Reply with the two ingredients that can be used to make a dish and the name of the dish.
3. Call seewhat() to observe the position of the first ingredient.
4. Call grasp_obj(x1, y1, x2, y2) to pick up the first ingredient.
5. Call putdown() to put down the first ingredient.
6. Call seewhat() to observe the position of the second ingredient.
7. Call grasp_obj(x1, y1, x2, y2) to pick up the second ingredient.
8. Call putdown() to put down the second ingredient.