Welcome to DOFBOT Pro repository

Voice Control Yolov11 Garbage Recognition Sorting

This course is specifically for Orin board. Jetson-Nano board cannot run this function due to insufficient performance. Before running the function, you need to close the App and large programs. For the closing method, refer to [4. Preparation] - [1. Manage APP control services].

1. Function Description

After the program starts, voice commands are issued to sort garbage. The program will recognize garbage label blocks in the image according to the command, calculate their positions in the world coordinate system, then control the robotic arm to grasp them, classify them according to the recognized garbage label names, and place them at the designated corresponding positions. At the same time, it will announce what type of garbage the grasped garbage label block is.

2. Startup and Operation

2.1. Startup

Orin board opens the terminal and inputs the following commands,

#Start camera and inverse kinematics
ros2 launch dofbot_pro_info camera_arm_kin.launch.py
#Start speech recognition and broadcast
ros2 launch yahboom_speech speech.launch.py
#Start image conversion program
ros2 run dofbot_pro_voice_ctrl msgToimg
#Start yolov11 sorting program
ros2 run dofbot_pro_voice_ctrl yolov11_sortation_VC
#Start yolov11 recognition program (startup may be slow)
python3 /home/jetson/dofbot_pro_ws/src/dofbot_pro_voice_ctrl/dofbot_pro_voice_ctrl/Yolov11/yolov11_VC.py

2.2. Operation Steps

After all programs are running, place four wooden blocks with garbage labels in the image. Say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Start garbage sorting" to the speech module. The speech module will reply "OK". The robotic arm will grasp garbage blocks and place them at corresponding positions, and the speech module will announce what garbage it is and the type of garbage. After placement is completed, it will announce "Placement complete" and the robotic arm returns to the recognition posture.

3. Core Code Analysis

3.1. Robotic Arm Grasping Node yolov11_sortation_VC

Source code path: ~/dofbot_pro_voice_ctrl/dofbot_pro_voice_ctrl/Yolov11/yolov11_sortation_VC

Mainly explains how to publish voice broadcast topics.

​x
#Create publisher for voice broadcast topic
self.pub_playID = self
 Voice Control Tracking and Grasping Color Block
Voice Control Object Tracking and Grasping