Welcome to DOFBOT-Ultra repository

Voice Control Tracking and Grasping AprilTag

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

After issuing a voice command for the robotic arm to track and grasp an AprilTag, the program will control the robotic arm to track the AprilTag according to the command. In DaBai depth camera mode, it will grasp the AprilTag block when conditions are met; in standard USB camera mode, the same voice command only starts tracking without performing grasping.

This tutorial uses the generic startup command camera_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes tracking and grasping via depth camera pipeline;

• Other values: Executes RGB-only tracking pipeline via standard USB camera, tracking only, no grasping.

All functions involving picking up cubic blocks must use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo 6 stall and damage.

2. Startup and Operation

2.1. Startup

Raspberry Pi version

​x
# ==================== Basic Service Layer ====================
# 1) Start camera and robotic arm inverse kinematics service
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py

# ==================== Voice Input Layer ====================
# 2) Start speech recognition (converts voice commands to digital codes)
ros2 launch dofbot_yahboom_speech speech.launch.py

# ==================== Core Control Layer ====================
# 3) Start voice router (core bridge: converts voice_result to apriltag_follow_enable)
ros2 run dofbot_ultra_voice_ctrl voice_router

# ==================== Visual Perception Layer ====================
# 4) Start tracking and grasping (voice control mode)
ros2 run dofbot_ultra_apriltag apriltag_follow --ros-args -p voice_control_mode:=true

# ==================== Voice Feedback Layer ====================
# 5) Start voice broadcast (voice feedback after execution completes)
ros2 run dofbot_yahboom_speech voice_player

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

For RDK:

xxxxxxxxxx
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py
ros2 run dofbot_ultra_driver arm_driver
ros2 launch dofbot_yahboom_speech speech.launch.py
ros2 run dofbot_ultra_apriltag apriltag_follow --ros-args -p voice_control_mode:=true
# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

2.2. Operation Steps

After all programs are running, the robotic arm will move to the tracking posture. At this time, hold an AprilTag and place it in the image. Say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Track and clip machine code" to the speech module. Slowly move the AprilTag, and the robotic arm will track the AprilTag. In DaBai depth camera mode, after the robotic arm stops tracking and the target stabilizes, it will beep and pick up the machine-coded wooden block, place it in the set position, and finally return to the tracking posture of the robotic arm. In normal USB camera mode, the robotic arm will continue to track, but will not perform the beeping, picking, and placing actions.

The runtime behavior automatically switches based on the current CAMERA_TYPE:

• Depth camera mode: Calculates grasp point via depth pipeline and executes grasping;

• Standard USB camera mode: Continuously tracks via RGB-only pipeline, no grasp action calculation.

3. Core Code Analysis

3.1. Voice Router Node voice_router.py

In this tutorial, the logic for "understanding commands and switching modes" is completed by dofbot_ultra_voice_ctrl/voice_router.py. It receives the voice_result published by the speech recognition module and translates "Track and grasp AprilTag" into standard control topics.

xxxxxxxxxx
if code == 105:
    self._set_mode('apriltag_follow')
    self._publish_bool(self.pub_apriltag_follow_enable, True)
    return

That is, in the current architecture:

1. The voice module still recognizes the original phrase "Track and clip machine code";

2. voice_router.py receives code 105 and publishes apriltag_follow_enable=True;

3. The downstream node that actually executes the tracking/grasping logic is apriltag_follow.py.

3.2. Tracking Node apriltag_follow.py

This tutorial actually starts with:

xxxxxxxxxx
ros2 run dofbot_ultra_apriltag apriltag_follow --ros-args -p voice_control_mode:=true

Therefore, the actual source code being run is dofbot_ultra_apriltag/apriltag_follow.py. This node no longer directly subscribes to voice_result; instead, it enters the working state via apriltag_follow_enable. At the same time, it automatically switches between depth camera and USB camera behavior based on CAMERA_TYPE.

xxxxxxxxxx
self.enable_sub = self.create_subscription(
    Bool, 'apriltag_follow_enable', self._enable_callback, 10)

can_grasp = (
    USE_DEPTH_CAMERA
    and self.enabled
    and (not self.grasping)
    ...
)

This logic determines the actual behavior of the current tutorial:

1. In depth camera mode, when the target is centered and the distance condition is met, apriltag_follow.py will trigger grasping;

2. In standard USB camera mode, can_grasp is gated by USE_DEPTH_CAMERA, so the node only tracks, does not grasp;

3. The voice version simply opens enabled, and the downstream behavior fully inherits apriltag_follow.py's own implementation.