Welcome to DOFBOT SE repository

Multimodal Large Model+robotic Arm Tracking

Function Description

After the program runs, after waking up the voice module, input robotic arm tracking commands by voice. According to the type of object to be tracked, the large model will start an external program to control the robotic arm to track the target object.

Startup

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py

Robotic arm tracking is divided into two types of objects:

• Track machine code: Track machine code x, where x is the machine code ID with values 1-4, for example, track machine code 1.

• Track other objects: Track any object, for example, track the red block on the green wooden block.

Track Machine Code

Startup

After waking up the voice module, input command by voice:

xxxxxxxxxx
Track machine code 2

First, the robotic arm will adjust to the tracking posture, then capture the current image and start recognizing and tracking the target machine code. If the recognized machine code is not number 2, then the robotic arm will not track and will print "Not target apriltag" in the terminal; if the recognized machine code is number 2, then the robotic arm starts tracking, slowly moving the machine code, and the robotic arm will follow the movement. If you want to exit tracking, then after waking up the voice module, input by voice: Cancel tracking.

Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;

2. Call apriltag_follow(2) function, where the parameter 2 represents the ID of the machine code to be tracked.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Tracking】. The voice version and text version have the same action functions; only the input method for task commands is different.

Track Other Objects

Startup

After waking up the voice module, input command by voice:

xxxxxxxxxx
Track the red block in my hand

First, the robotic arm will adjust to the tracking posture, then capture one frame of the current image to find the red block in hand, then control the robotic arm to track the red block, slowly moving the position of the red block, and the robotic arm will follow the movement. If you want to exit tracking, then after waking up the voice module, input by voice: Cancel tracking.

Task Planning

1. Call track_pose() function to adjust the robotic arm to tracking posture;

2. Call seehat() to observe the environment and find the red block in hand;

3. Call track(x1, y1, x2, y2) function, where x1, y1, x2, y2 are the outer bounding box coordinates of the red block.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Tracking】. The voice version and text version have the same action functions; only the input method for task commands is different.