Welcome to DOFBOT Pro repository

Multimodal Large Model+robotic arm tracking（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and input robotic arm tracking commands by voice. Based on the type of object to be tracked, the large model will start external programs to control the robotic arm to track the target object.

2. Startup

Users with Jetson-Nano mainboard version need to enter the docker container first and then input the following command. Users with Orin mainboard can directly open the terminal and input the following command:

ros2 launch largemodel largemodel_control.launch.py

The robotic arm tracking has two types of objects:

• Track machine code: Track machine code x, where x is the machine code ID with values 1-4, for example, track machine code 1.Wood blocks used: 40x40x40mm wood blocks.
• Track other objects: Track any object, for example, track the small yellow duck on the green wooden block.

2.1. Track Machine Code

2.1.1. Startup

After waking up the voice module, input the command by voice:

xxxxxxxxxx
Track machine code 2

First, the robotic arm will adjust to the tracking posture, then capture the current image and start recognizing and tracking the target machine code. If the recognized machine code is not 2, the robotic arm will not track and will print "Not target apriltag" in the terminal; if the recognized machine code is 2, the robotic arm will start tracking. Move the machine code slowly, and the robotic arm will follow the movement. To exit tracking, wake up the voice module and input by voice: Cancel tracking.

2.1.2. Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;
2. Call apriltag_follow(2) function, where the parameter 2 represents the ID of the machine code to be tracked.

2.1.3. Core Code Analysis

You can refer to the content in 2.1.3. Core Code Analysis from tutorial [17. AI Model - Text Version] - [Multimodal Large Model+robotic arm tracking]. The voice version and text version have the same action functions, only the task command input method is different.

2.2. Track Other Objects

2.2.1. Startup

After waking up the voice module, input the command by voice:

xxxxxxxxxx
Track the small yellow duck in hand

First, the robotic arm will adjust to the tracking posture, then capture the current frame to find the small yellow duck in hand, then control the robotic arm to track the small yellow duck. Move the position of the small yellow duck slowly, and the robotic arm will follow the movement. To exit tracking, wake up the voice module and input by voice: Cancel tracking.

2.2.2. Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;
2. Call seehat() to observe the environment and find the small yellow duck in hand;
3. Call track(x1, y1, x2, y2) function, where x1, y1, x2, y2 are the outer bounding box coordinates of the small yellow duck.

2.2.3. Core Code Analysis

You can refer to the content in 2.2.3. Core Code Analysis from tutorial [17. AI Model - Text Version] - [Multimodal Large Model+robotic arm tracking]. The voice version and text version have the same action functions, only the task command input method is different.