Welcome to DOFBOT-Ultra repository

Voice Control Tracking and Grasping Color Block

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

1. Function Description

After the program runs, issue a voice command for the robotic arm to track and grasp a color block. The program will control the robotic arm to track the color block according to the command. In DaBai depth camera mode, it will grasp the color block when conditions are met; in standard USB camera mode, the same voice command only starts tracking without performing grasping.

This tutorial uses the generic startup command camera_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes tracking and grasping via depth camera pipeline;

• Other values: Executes RGB-only tracking pipeline via standard USB camera, tracking only, no grasping.

All functions involving picking up cubic blocks must use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo #6 stall and damage.

2. Startup and Operation

2.1. Startup

1. Start camera and robotic arm basic services

ros2 launch dofbot_ultra_info camera_arm_kin.launch.py

2. Start robotic arm driver (core control)

xxxxxxxxxx
ros2 run dofbot_ultra_driver arm_driver

3. Start speech recognition and broadcast (interaction layer)

xxxxxxxxxx
ros2 launch dofbot_yahboom_speech speech.launch.py

4. Start voice router (command distribution)

xxxxxxxxxx
ros2 run dofbot_ultra_voice_ctrl voice_router

5. Start color tracking and grasping node

xxxxxxxxxx
ros2 run dofbot_ultra_color color_follow --ros-args -p start_enabled:=false

6. Start voice broadcaster

xxxxxxxxxx
ros2 run dofbot_yahboom_speech voice_player

For RDK:

​x
ros2 launch dofbot_ultra_info camera_arm_kin.launch.py
ros2 run dofbot_ultra_driver arm_driver
ros2 launch dofbot_yahboom_speech speech.launch.py
ros2 run dofbot_ultra_color color_follow --ros-args -p start_enabled:=false

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

2.2. Startup Steps

2.2.1. Calibrating Color Block

After the program starts, the robotic arm will move to the tracking pose. Then hold a color block within the image frame, select a region of the color block with the mouse, release the mouse, and calibration is complete.

2.2.2. Voice Control

After color calibration is completed, say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here". Then say "Track and clip color blocks" to the speech module. The robotic arm will then track the held color block. Slowly move the color block, and the robotic arm will move correspondingly. In DaBai depth camera mode, once the target is stable, the robot arm will beep, grasp and place the color block, and finally return to its initial posture. In normal USB camera mode, the robot arm will continue to track, but will not perform the beeping, grasping and placing actions.

The runtime behavior automatically switches based on the current CAMERA_TYPE:

• Depth camera mode: Calculates grasp point via depth pipeline and executes grasping;

• Standard USB camera mode: Continuously tracks via RGB-only pipeline, no grasp action calculation.

3. Core Code Analysis

3.1. Voice Router Node voice_router.py

In the current version, the voice control logic is handled by dofbot_ultra_voice_ctrl/voice_router.py, which converts speech recognition results into standard control topics.

xxxxxxxxxx
if code == 106:
    self._set_mode('color_follow')
    self._publish_bool(self.pub_color_follow_enable, True)
    return

Therefore, in this tutorial, when you say "Track and grasp color block", what actually happens is:

1. The voice module recognizes the command code 106;

2. voice_router.py publishes color_follow_enable=True;

3. color_follow.py starts the tracking flow.

3.2. Tracking Node color_follow.py

This tutorial actually starts with:

xxxxxxxxxx
ros2 run dofbot_ultra_color color_follow --ros-args -p start_enabled:=false

Therefore, the source code that actually executes the tracking logic is dofbot_ultra_color/color_follow.py. It does not directly subscribe to voice_result; instead, it subscribes to color_follow_enable and automatically switches between depth camera and USB camera behavior based on CAMERA_TYPE.

xxxxxxxxxx
self.enable_sub = self.create_subscription(
    Bool, 'color_follow_enable', self._enable_callback, 10)

can_grasp = (
    USE_DEPTH_CAMERA
    and (not self.grasping)
    and (self.cur_distance != 0.0)
    ...
)

This means the actual behavior of the current tutorial is:

1. In depth camera mode, when the color block is centered and the distance condition is met, it will continue to execute grasping;

2. In standard USB camera mode, can_grasp is restricted by USE_DEPTH_CAMERA, so the node only tracks and does not grasp;

3. The voice version does not have a separate copy of the tracking code; it directly reuses color_follow.py.