Welcome to DOFBOT SE repository

Multimodal Large Model+robotic Arm Gripping

Function Description

After the program runs, after waking up the voice module, input robotic arm gripping commands by voice. According to the type of object to be gripped, the large model will start an external program to control the robotic arm to grip the target object.

Block height and width limit: no more than 3 centimeters each.

Startup

Open the terminal and enter the following command:

ros2 launch largemodel largemodel_control.launch.py

The robotic arm gripping has the following types of action commands:

• Grip by machine code ID: Sort machine code 1/2/3/4

• Grip other objects: Grip the red block on the desktop, grip the blue cuboid on the desktop

Grip By Machine Code ID

Startup

After waking up the module, input by voice:

xxxxxxxxxx
Sort machine code 2

The program will open two terminals, start the machine code recognition program and the gripping program respectively. After recognizing machine code 2, the gripping program will control the robotic arm to lower and grip machine code 2, then place it at the designated position, and finally return to the machine code recognition posture to prepare for the next grip. Only after no longer detecting machine code 2 will it feedback to the large model and end the task.

Task Planning

Plan the action function apriltag_sort(2), where the parameter 2 represents the ID of the machine code to be gripped.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Gripping】. The voice version and text version have the same action functions; only the input method for task commands is different.

Grip Other Objects

Startup

After waking up the module, input by voice:

xxxxxxxxxx
Grip the red block on the desktop

The program will first take a photo to determine the position of the red block, then open two terminals, start the KCF tracking and positioning program and the object gripping program respectively. After calculation, the robotic arm will lower and grip the red block and finally return to the initial posture.

Task Planning

1. Call seehat() to check the position of the red block;

2. Call grasp_obj(x1,y1,x2,y2) gripping function, where x1,y1,x2,y2 represent the top-left and bottom-right corner coordinates of the red block's outer bounding box.

Core Code Analysis

Please refer to the Core Code Analysis content in the tutorial 【AI Model-Text Version】-【Multimodal Large Model + robotic Arm Gripping】. The voice version and text version have the same action functions; only the input method for task commands is different.