Welcome to DOFBOT-JetsonNANO repository

Multimodal Large Model + Intent Prediction + Robotic Arm Gripping（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and say the action command "I'm a little hungry/I'm a little thirsty". The large model will look for food or water based on hunger or thirst, then grasp the food or water.Object Height and Width Limits: Neither shall exceed 3 centimeters.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

ros2 launch largemodel largemodel_control.launch.py

The intent prediction includes the following two action commands:

• I'm hungry
• I'm a little thirsty

After waking up the voice module, input by voice:

xxxxxxxxxx
I'm hungry

First execute the seehat function to check if there is food in the current environment, then call grasp_obj(x1,y1,x2,y2) to grasp the food, where x1,y1,x2,y2 are the bounding box coordinates of the food returned by the large model.

After waking up the voice module, input by voice:

xxxxxxxxxx
I'm a little thirsty

First execute the seehat function to check if there is water or drinks in the current environment, then call grasp_obj(x1,y1,x2,y2) to grasp the water or drinks, where x1,y1,x2,y2 are the bounding box coordinates of the water or drinks returned by the large model.

The grasp_obj function was analyzed in previous courses and will not be analyzed here again.

3. Custom Intent Understanding

First, log in to the dify management interface. Enter the robotic arm's IP address in the browser and press Enter to log in to the dify management interface. For the first login, you need to enter email and password. The email is yahboom@163.com and the password is yahboom123. After successful login, it will be shown as below:

Then click on Knowledge Base and select [决策层训练样例en.xlsx], as shown below:

After entering this [决策层训练样例en.xlsx], click on [决策层训练样例en.xlsx], then modify the file content in the next step, as shown below:

After opening this xlsx file, you can see the content inside. Click [Add Chunk] on the right side, as shown below:

Then add the following content. Suppose we add a scenario: I feel a little hot, then the robotic arm grasps the small fan on the desktop. You can input the following content:

xxxxxxxxxx
query: "I'm feeling a bit hot."; answer: "1. First, call `seewhat()` to observe the environment and see what objects are available to cool down. 2. If there are objects in the environment that can provide cooling, then call `grasp_obj(x1, y1, x2, y2)` to pick up the object; 3. Call `putdown()` to place the object down."

Modify according to actual problems and expected output steps. The content in query is the actual problem, and the content in answer is the expected output steps.

After adding the content, click Save, as shown below:

Then we return to the studio and click on Decision Layer to test, as shown below:

After entering the decision layer large model, input [I'm feeling a bit hot now] in the chat box and press Enter:

The decision layer will reply with the following content, which is consistent with our expected output, indicating that the correct steps have been successfully planned. Addition completed:

After starting the AI large model launch file, you can give commands through text terminal or voice, and the robotic arm will execute corresponding actions according to the planned steps.