Welcome to DOFBOT-Ultra repository

Voice Control Yolov26 Garbage Recognition Sorting

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

After the program starts, issue a garbage sorting command via voice. The program will identify the garbage label blocks in the image based on the command, calculate their positions in the world coordinate system, and then control the robotic arm to pick them up. It will classify them according to the recognized garbage label name and place them in the corresponding designated positions. At the same time, it will broadcast what type of garbage the picked-up garbage label block is.

All functions involving picking up cubic blocks must use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo #6 stall and damage.

2. Startup and Operation

2.1. Startup

Open a terminal and enter the following commands:

This tutorial uses the generic startup command camera_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes via depth camera pipeline;

• Other values: Executes via standard USB camera pipeline.

ros2 launch dofbot_ultra_info camera_arm_kin.launch.py
ros2 run dofbot_ultra_yolo msgToimg
ros2 run dofbot_ultra_yolo yolo_detect --ros-args -p model:=best_yolo26.pt -p imgsz:=320
ros2 launch dofbot_yahboom_speech speech.launch.py
ros2 run dofbot_ultra_voice_ctrl voice_router
ros2 run dofbot_ultra_yolo yolo_sortation
# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

For RDK:

xxxxxxxxxx
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py
ros2 run dofbot_ultra_yolo msgToimg
ros2 run dofbot_ultra_yolo yolo_detect # RDK uses a yolo26-based bpu acceleration solution
# If you want to use the yolov11 version, you can use the following command
# RDK: ros2 run dofbot_ultra_yolo yolo_detect --ros-args -p model_path:=/home/sunrise/ultralytics/ultralytics/best_bayese_640x640_nv12.bin
ros2 launch dofbot_yahboom_speech speech.launch.py
ros2 run dofbot_ultra_yolo yolo_sortation

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

2.2. Operation Steps

After all programs are running, place four wooden blocks with garbage labels in the image. Say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Start garbage sorting" to the speech module. The speech module will reply "OK". The robotic arm will grasp garbage blocks and place them at corresponding positions, and the speech module will announce what garbage it is and the type of garbage. After placement is completed, it will announce "Placement complete" and the robotic arm returns to the recognition posture.

The grasp point calculation method automatically switches based on the current CAMERA_TYPE:

• Depth camera mode: Calculates grasp point via depth pipeline;

• Standard USB camera mode: Calculates grasp point based on planar calibration results.

3. Core Code Analysis

This function is mainly completed by the coordination of three core nodes: Voice Router (voice_router), YOLO Detection (yolo_detect), and Sorting Execution (yolo_sortation).

Flow

1. Voice Trigger: Speak the command, the voice module recognizes it and notifies the router node.

2. Signal Distribution: The router node simultaneously wakes up the "detection node" and "sorting node".

3. Visual Recognition: The detection node uses YOLO to recognize object coordinates and sends them to the sorting node.

4. Grasp Execution: The sorting node controls the robotic arm to grasp based on the coordinates and broadcasts the garbage type.

1. Voice Trigger and Routing (voice_router.py)

When the voice module recognizes "Start garbage sorting" (M2C code 108), voice_router is responsible for distributing the task.

xxxxxxxxxx
# dofbot_ultra_voice_ctrl/voice_router.py

if code == 108:  # Recognized "Start garbage sorting"
    self._set_mode('yolo_sortation')
    
    # 1. Publish sort_flag=True, this is the start signal
    self._publish_bool(self.pub_sort_flag, True) 
    
    # 2. Manually make the voice module reply "OK" (ID 45)
    self._publish_int(self.pub_player_id, 45) 
    return

2. YOLO Detection Node (yolo.py)

Raspberry Pi: When the sort_flag signal is received, it starts sending the recognized object coordinates.

xxxxxxxxxx
# dofbot_ultra_yolo/yolo.py

# Subscribe to the sort_flag topic, receive trigger signals from voice or keyboard
self.sort_flag_sub = self.create_subscription(Bool, 'sort_flag', self.sort_flag_callback, 10)

def sort_flag_callback(self, msg):
    if msg.data:
        # Signal received, enable detection flag
        self.start_flag = True 

RDK: When the sort_flag signal is received, it starts sending the recognized object coordinates.

xxxxxxxxxx
def sort_flag_callback(self, msg):
    if msg.data and not self.target_locked and not self.waiting_for_grasp:
        self.get_logger().info("External trigger received, starting new detection")
        self.start_flag = True
        self.start_time = time.time()
        # Clear old target info
        self.cx = self.cy = 0
        self.name = None
        self.pub_sort_flag.publish(Bool(data=True))
    else:
        self.get_logger().warn(f"Trigger ignored: locked={self.target_locked}, waiting={self.waiting_for_grasp}")

Publishing coordinates

xxxxxxxxxx
# In the image processing callback
if boxes != [None] and self.start_flag == True:
    # ... calculate center point coordinates ...
    center = Yolov5Detect()
    center.centerx = float(center_x)
    center.centery = float(center_y)
    center.result = str(self.yolo_model.names[class_id])
    
    # Publish detection results to the sorting node
    self.pubDetect.publish(center) 
    
    # Close the flag immediately after one send, wait for next trigger (single grasp mode)
    self.start_flag = False 

3. Sorting Execution Node (yolo_sortation.py)

Converts camera coordinates to the robotic arm's world coordinates and compensates for object height.

xxxxxxxxxx
# dofbot_ultra_yolo/yolo_sortation.py

# 1. Grasp point descent compensation
# The detected point is on the object surface. For a secure grasp, lower the gripper center by 3.7 cm
raw_z = raw_z - 0.037 

# 2. Apply tilt and height error compensation
tilt_correction = (global_config.slope_x * raw_x) + (global_config.slope_y * raw_y)
height_comp_z = global_config.z_comp_z * raw_z

# 3. Calculate final robotic arm target coordinates (x, y, z)
pose_T[0] = raw_x + global_config.x_offset + ...
pose_T[1] = raw_y + global_config.y_offset + ...
pose_T[2] = raw_z + global_config.z_offset + ... - tilt_correction - height_comp_z

Classification Broadcast and Placement After successful grasping, broadcast based on the recognized object name (self.name) and place it in the corresponding area.

​x
if self.name in self.recyclable_waste: # If it is recyclable waste
    if self.name == "Zip_top_can":     # If it is a can
        self.play_id.data = 94         # Broadcast "This is a can..."
        self.pub_playID.publish(self.play_id)
    
    # Move to the recycling bin position
    self.set_joint = [140, 20, 90, 3, 50.0, 143.0] 

# ... after placement action is complete ...
self.play_id.data = 81 # Broadcast "Placement complete"
self.pub_playID.publish(self.play_id)