Welcome to DOFBOT Pro repository

AI Top Chef

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program starts, give the large model an instruction, ask the large model what delicious food can be made with the ingredients on the desktop. The program will check the ingredients on the desktop and reply with what delicious food dishes can be made, and finally will grip the ingredients and place them on the left side of the robotic arm.

2. Startup

Taking the text version as an example, users with Jetson-Nano mainboard version need to enter the docker container first and then input the following command. Users with Orin mainboard can directly open the terminal and input the following command:

ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=True

Then open a second terminal and input the following command:

xxxxxxxxxx
ros2 run text_chat text_chat

Then input the following content in the text_chat terminal:

xxxxxxxxxx
Check what delicious food can be made with the ingredients on the desktop

Press Enter. If it's the voice version, wake up the voice module and directly say to the voice module "Check what delicious food can be made with the ingredients on the desktop", then wait for the large model to think and reply, as shown in the figure below:

The robotic arm will select two ingredients that can make delicious food based on what it sees, and will grip these two ingredients and place them on the left side of the robotic arm.

3. Task Planning

1. Call seehat() to observe the environment and check what ingredients are on the desktop;
2. Reply with what two ingredients can make a dish and the dish name;
3. Call seehat() to observe the position of the first ingredient;
4. Call grasp_obj(x1, y1, x2, y2) to grip the first ingredient;
5. Call putdown() to put down the first ingredient;
6. Call seehat() to observe the position of the second ingredient;
7. Call grasp_obj(x1, y1, x2, y2) to grip the second ingredient;
8. Call putdown() to put down the second ingredient.