Voice Control AprilTag ID Sorting

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

The voice issues an AprilTag ID sorting command, and the robotic arm will pick up the AprilTag wooden block corresponding to the specified ID based on the command.

This tutorial uses the generic startup command camera_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes via depth camera pipeline;

• Other values: Executes via standard USB camera pipeline.

All functions involving picking up cubic blocks must use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo 6 stall and damage.

2. Startup and Operation

2.1. Startup

Step-by-step startup (Raspberry Pi version)

​x
# ==================== Basic Service Layer ====================
# 1) Start camera and robotic arm inverse kinematics service
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py

# 2) Start robotic arm driver
ros2 run dofbot_ultra_driver arm_driver


# ==================== Voice Input Layer ====================
# 3) Start speech recognition (converts voice commands to digital codes)
ros2 launch dofbot_yahboom_speech speech.launch.py


# ==================== Core Control Layer ====================
# 4) Start voice router (core bridge: converts voice_result to control signals for each module)
#    Function: receives voice code → maps to corresponding function module (AprilTag/Color/KCF etc.)
ros2 run dofbot_ultra_voice_ctrl voice_router


# ==================== Visual Perception Layer ====================
# 5) Start AprilTag detection
#    Note: Disable terminal trigger mode, use topic trigger instead (avoid input() interference)
ros2 run dofbot_ultra_driver apriltag_detect --ros-args -p terminal_trigger:=false


# ==================== Action Execution Layer ====================
# 6) Start grasp executor (receives position information, controls robotic arm to complete grasping)
#    Reuses the new compensation/calibration/grasping logic
ros2 run dofbot_ultra_driver grasp


# ==================== Voice Feedback Layer ====================
# 7) Start voice broadcast (voice feedback after execution completes)
ros2 run dofbot_yahboom_speech voice_player


# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

For RDK:

xxxxxxxxxx
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py
ros2 run dofbot_ultra_driver arm_driver
ros2 launch dofbot_yahboom_speech speech.launch.py
ros2 run dofbot_ultra_driver apriltag_detect --ros-args -p terminal_trigger:=false
ros2 run dofbot_ultra_driver grasp

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

Notes:

1. camera_arm_kin.launch.py is the generic entry point, suitable for both camera modes.

2. The startup method for nodes like apriltag_detect, grasp, voice_router remains unchanged.

2.2. Operation Steps

After all programs are running, place four AprilTags with different IDs in the image. Say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Sorting machine code number one"/"Sorting machine code number two"/"Sorting machine code number three"/"Sorting machine code number four" to the speech module. The robotic arm will lower its gripper to grasp the target ID AprilTag block, and the speech module will broadcast "OK". After grasping the AprilTag block, it will place it at the designated position. Finally, the robotic arm returns to its initial posture, and the speech broadcast module will announce "Placement complete".

3. Core Code Analysis

3.1. Speech Recognition Node speech_recognize_node

Source path:

Raspberry Pi: ~/dofbot_ultra_ws/src/dofbot_yahboom_speech/dofbot_yahboom_speech/speech_recognize.py

RDK: ~/dofbot_ultra_ws/src/core/dofbot_yahboom_speech/dofbot_yahboom_speech/speech_recognize.py

Mainly explains how to create a publisher for speech recognition results and how to call the library to obtain recognition results, and finally publish the speech recognition result.

xxxxxxxxxx
# Import speech recognition and broadcast library
from Speech_Lib import Speech
# Create speech recognition and broadcast object
mySpeech = Speech()
# Create speech recognition result publisher
self.pub_res = self.create_publisher(Int8, 'voice_result', 1)
# Call the library function to get the recognition result
result = mySpeech.speech_read()
# Publish the speech recognition result topic
self.voice_command.data = result
self.pub_res.publish(self.voice_command)

3.2. Voice Broadcast Node voice_player_node

Source path:

Raspberry Pi: ~/dofbot_ultra_ws/src/dofbot_yahboom_speech/dofbot_yahboom_speech/voice_player.py

RDK: ~/dofbot_ultra_ws/src/core/dofbot_yahboom_speech/dofbot_yahboom_speech/voice_player.py

Mainly explains how to create a subscriber for voice broadcast and how to call the library to broadcast voice.

xxxxxxxxxx
# Import speech recognition and broadcast library
from Speech_Lib import Speech
# Create speech recognition and broadcast object
mySpeech = Speech()

# Create subscriber for voice broadcast results
self.sub_playID = self.create_subscription(Int8, 'player_id', self.playercallback, 1000)

# Callback function
def playercallback(self, msg):
    # Extract voice broadcast topic data
    ID_value = msg.data
    print("ID_value: ", ID_value)
    if ID_value != 0:
        play_id = ID_value
        print("play_id = ", play_id)
        # Call the void_write function in the library to play audio files
        mySpeech.void_write(int(play_id))

3.3. AprilTag Detection Node apriltag_detect.py

This tutorial actually starts with:

xxxxxxxxxx
ros2 run dofbot_ultra_driver apriltag_detect --ros-args -p terminal_trigger:=false

Therefore, the actual source code being run is dofbot_ultra_driver/apriltag_detect.py. In the current implementation, voice commands are first interpreted into control topics by voice_router, and the detection node is only responsible for receiving triggers, identifying AprilTags, and sending the target position to the grasping node.

xxxxxxxxxx
self.trigger_sub = self.create_subscription(
    Bool, 'apriltag_trigger', self._trigger_callback, 10)
self.target_id_sub = self.create_subscription(
    Int16, 'apriltag_target_id', self._target_id_callback, 10)

# Publish position after detecting the target
self.pos_info_pub = self.create_publisher(AprilTagInfo, "PosInfo", qos_profile=10)

The working method is:

1. voice_router receives the "Sort AprilTag 1-4" command;

2. Publishes grasp_enable=True, apriltag_target_id=<target ID>, apriltag_trigger=True;

3. apriltag_detect.py only identifies the target with the corresponding ID;

4. After detection, publishes PosInfo to the grasping node.

3.4. Robotic Arm Grasping Node grasp.py

This tutorial actually starts with:

xxxxxxxxxx
ros2 run dofbot_ultra_driver grasp

That is, the source code that actually performs the grasping and placing actions is dofbot_ultra_driver/grasp.py. This node is responsible for receiving the target pose from the vision node, calling the inverse kinematics solver, and finally controlling the robotic arm to complete the entire grasping sequence.

xxxxxxxxxx
self.sub = self.create_subscription(
    AprilTagInfo, 'PosInfo', self.pos_callback, 1)
self.enable_sub = self.create_subscription(
    Bool, 'grasp_enable', self._enable_callback, 10)
self.pubGraspStatus = self.create_publisher(Bool, 'grasp_done', 1)

The core action flow is as follows:

1. After receiving PosInfo, calculate the target's position in the robotic arm coordinate system;

2. Request the dofbot_kinematics service to solve for joint angles;

3. Execute descending, closing gripper, lifting, placing, and returning to zero;

4. Publish grasp_done=True to signal the detection node to stop sending targets;

5. The voice broadcast node announces "Placement complete" based on player_id.

Voice Control Function Structure Analysis

Here we take AprilTag ID sorting as an example to look at the actual call flow of the voice control series code.

1. System Architecture

xxxxxxxxxx
Speech Recognition → voice_router → Function Modules (AprilTag Detection, Grasp Execution, etc.)
              ↓
         voice_player (Voice Broadcast Feedback)

2. Core Components

Speech Recognition Module

• Uses I2C bus (address 0x0f) to communicate with the hardware speech recognition module

• Reads voice phrase configuration from [ID.csv]

• Recognition results are published to the voice_result topic (Int8 type)

• Supports wake-word trigger mode to avoid false triggering

Voice Router [voice_router.py]

• Core function: Converts voice commands into standard control signals

• Subscribes to the voice_result topic

• Maps voice codes to different control topics:

  • apriltag_trigger - Triggers AprilTag detection

  • apriltag_target_id - Specifies the target tag ID

  • grasp_enable - Enables grasping function

  • Enable signals for other function modules

AprilTag Detection Module [apriltag_detect.py]

• Subscribes to RGB and depth camera images

• Uses the dt_apriltags library to detect AprilTags

• Publishes position information to the PosInfo topic

• Key feature: Controls whether to publish position information via the grasp_done topic (avoids repeated triggering)

Grasp Execution Module [grasp.py]

• Subscribes to PosInfo to get the target position

• Subscribes to grasp_enable to control whether motion is allowed

• Uses the inverse kinematics service to calculate joint angles

• Reads the offset configuration file for position compensation

• Controls the robotic arm to execute the complete grasping sequence

3. Voice Command Mapping (Using AprilTag Grasping as Example)

According to [ID.csv]:

4. Detailed Workflow

Taking "Sort AprilTag 1" as an example:

x
1. User says "hi,yahboom, Sorting machine code number one"
   ↓
2. voice_recognition recognizes code 95
   ↓
3. voice_router receives code 95, executes:
   - Sets mode to 'apriltag_sort'
   - Publishes grasp_enable = True
   - Publishes apriltag_target_id = 1
   - Publishes apriltag_trigger = True (one-shot trigger)
   ↓
4. apriltag_detect receives the trigger signal, starts detection
   - Finds the tag with ID=1
   - Publishes position information to PosInfo
   ↓
5. grasp executor receives PosInfo and target ID=1
   - Verifies tag ID matches
   - Computes inverse kinematics
   - Executes grasping action
   - Publishes grasp_done = True
   ↓
6. apriltag_detect receives grasp_done, stops publishing position information