Welcome to DOFBOT-Pi repository

Multimodal Large Model+robotic arm tracking（Voice Version）

Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program runs, wake up the voice module and input robotic arm tracking commands by voice. Based on the type of object to be tracked, the large model will start external programs to control the robotic arm to track the target object.

2. Startup

After entering the Docker container, enter the following commands in the Docker terminal.

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

The robotic arm tracking has two types of objects:

• Track machine code: Track machine code x, where x is the machine code ID with values 1-4, for example, track machine code 1.
• Track other objects: Track any object, for example, track the small yellow duck on the green wooden block.

2.1. Track Machine Code

2.1.1. Startup

After waking up the voice module, input the command by voice:

xxxxxxxxxx
Track machine code 2

First, the robotic arm will adjust to the tracking posture, then capture the current image and start recognizing and tracking the target machine code. If the recognized machine code is not 2, the robotic arm will not track and will print "Not target apriltag" in the terminal; if the recognized machine code is 2, the robotic arm will start tracking. Move the machine code slowly, and the robotic arm will follow the movement. To exit tracking, wake up the voice module and input by voice: Cancel tracking.

2.1.2. Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;
2. Call apriltag_follow(2) function, where the parameter 2 represents the ID of the machine code to be tracked.

2.1.3. Core Code Analysis

You can refer to the content in 2.1.3. Core Code Analysis from tutorial [18. AI Model - Text Version] - [Multimodal Large Model+robotic arm tracking]. The voice version and text version have the same action functions, only the task command input method is different.

2.2. Track Other Objects

2.2.1. Startup

After waking up the voice module, input the command by voice:

x
Track the red block in my hand

First, the robotic arm will adjust to the tracking posture, then capture the current frame to find the red block in hand, then control the robotic arm to track the red block. Move the position of the small yellow duck slowly, and the robotic arm will follow the movement. To exit tracking, wake up the voice module and input by voice: Cancel tracking.

2.2.2. Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;
2. Call seehat() to observe the environment and find the red block in hand;
3. Call track(x1, y1, x2, y2) function, where x1, y1, x2, y2 are the outer bounding box coordinates of the small yellow duck.

2.2.3. Core Code Analysis

You can refer to the content in 2.2.3. Core Code Analysis from tutorial [18. AI Model - Text Version] - [Multimodal Large Model+robotic arm tracking]. The voice version and text version have the same action functions, only the task command input method is different.