Welcome to DOFBOT-Ultra repository

Voice Control Object Tracking and Grasping

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this function, you need to close the web control program backend process.

The example in this section is a tracking-only version: it uses voice commands to invoke KCF/CSRT visual tracking and robotic arm following, without performing any grasping actions. The existing voice module keyword "Track and grasp object" is retained for compatibility with the current voice model vocabulary, avoiding the need for retraining; the actual behavior is only "tracking + robotic arm following the target".

The default tracking algorithm is CSRT (high accuracy, good OpenCV compatibility). The course name retains "KCF" as an industry convention; you can switch KCF_Tracker to KCF / MOSSE / MIL etc. via parameters.

1. Function Description

After the program starts, speak the wake word to the voice module, then say "Tracking and picking up objects". The voice router will trigger the tracking flow via /kcf_enable, and the robotic arm will start following the selected target object. The entire process will not trigger any grasping actions — no descending, no gripper closing, no placement.

This tutorial uses the generic startup command base_arm_kin.launch.py.

This launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: Executes tracking pipeline via color image + depth image;

• Other values: Executes RGB-only tracking pipeline via standard USB camera.

2. Startup and Operation

2.1. Startup

Open multiple terminals and execute in order:

​x
# Terminal 1: Basic drivers (robotic arm + camera + inverse kinematics)
ros2 launch dofbot_ultra_driver base_arm_kin.launch.py

# Terminal 2: KCF/CSRT visual tracking (opens image window, select target with mouse)
ros2 run dofbot_ultra_kcf KCF_Tracker

# Terminal 3: Robotic arm pure follow node (subscribes to /pos_xyz, no grasping)
ros2 run dofbot_ultra_kcf KCF_TrackOnly --ros-args -p voice_mode:=true

# Terminal 4: Speech recognition service
ros2 launch dofbot_yahboom_speech speech.launch.py

# Terminal 5: Voice router (converts voice commands to corresponding topic signals)
ros2 run dofbot_ultra_voice_ctrl voice_router

# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

For RDK:

xxxxxxxxxx
# Terminal 1: Basic drivers (robotic arm + camera + inverse kinematics)
ros2 launch dofbot_ultra_driver base_arm_kin.launch.py

# Terminal 2: KCF/CSRT visual tracking (opens image window, select target with mouse)
ros2 run dofbot_ultra_kcf KCF_Tracker

# Terminal 3: Robotic arm pure follow node (subscribes to /pos_xyz, no grasping)
ros2 run dofbot_ultra_kcf KCF_TrackOnly --ros-args -p voice_mode:=true

# Terminal 4: Speech recognition service
ros2 launch dofbot_yahboom_speech speech.launch.py


# Depth camera exposure settings
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

Notes:

1. base_arm_kin.launch.py is the generic entry point, suitable for both camera modes.

2. When CAMERA_TYPE is standard USB mode, KCF_Tracker automatically switches to RGB-only tracking implementation, and the startup method for KCF_TrackOnly remains unchanged.

2.2. Operation Steps

2.2.1. Selecting an Object

After KCF_Tracker starts, an image window will pop up. Hold the object to be tracked within the camera frame, use the left mouse button to drag a rectangular box around the object, then release. A green tracking box will immediately appear in the window, and /pos_xyz will start continuously publishing the target coordinates.

2.2.2. Voice Control

• After selecting the object and pressing the spacebar, say "Hello, yahboom" to the speech recognition module, and the speaker will broadcast "here".

• Then say "Tracking and picking up objects" (using the original phrase), and the speaker will announce "OK".

• At this point, the voice_router will issue kcf_enable=True; after receiving KCF_TrackOnly, it will maintain the tracking state, and the robotic arm will follow the object you selected.

• Move the target object slowly, and the robotic arm will adjust its posture synchronously.

• It will not perform gripping: even if the object is stable in the center of the screen for a period of time, it will not trigger actions such as downward movement, closing the gripper, or placement.

• To reselect an object, press the R key to reset KCF_Tracker, and then drag the selection box again.

3. Node and Topic Relationships

xxxxxxxxxx
+----------------+      /pos_xyz       +----------------+      Serial       +-----------+
|  KCF_Tracker   | ------------------> | KCF_TrackOnly  | --------------> | Robotic Arm |
+----------------+                     +----------------+                  +-----------+
                                            ^
                                            | /kcf_enable
                                            |
                                       +-------------+
                                       | voice_router|
                                       +-------------+
                                            ^
                                            | Voice command
                                       +-------------+
                                       |  speech node |
                                       +-------------+

4. Core Code Analysis

4.1. Voice Router (voice_router.py)

After receiving command code 107 ("Tracking and picking up objects"), it publishes kcf_enable=True and replies "OK".

xxxxxxxxxx
# dofbot_ultra_voice_ctrl/voice_router.py
if code == 107:
    self._set_mode('kcf')
    self._publish_bool(self.pub_kcf_enable, True)  # Trigger tracking
    self._publish_int8(self.pub_player_id, 45)     # Reply "OK"

This code segment is exactly the same as the original "Track and grasp" version; no retraining of voice vocabulary is needed. The difference is that the downstream node no longer performs grasping.

4.2. Visual Tracking (KCF_Tracker.py, default CSRT)

xxxxxxxxxx
self.declare_parameter('tracker_type', 'CSRT')
self.gTracker = Tracker(tracker_type=self.tracker_type)

# Continuously publish the image center coordinates of the target
pos = Position()
pos.x = center_x
pos.y = center_y
pos.z = self.cur_distance
self.pub_pos.publish(pos)

Notes:

1. In depth camera mode, pos.z is the target distance.

2. In standard USB mode, pos.z is an invalid placeholder value 999, used to maintain /pos_xyz message format compatibility; the pure follow node KCF_TrackOnly only uses x/y, so it is not affected.

4.3. Robotic Arm Pure Follow (KCF_TrackOnly.py)

class KCFTrackOnlyNode(Node):
    def __init__(self):
        super().__init__('KCF_track_only')
        self.dofbot_tracker = DofbotTrack()
        self.pos_sub = Subscriber(self, Position, '/pos_xyz')
        self.time_sync = ApproximateTimeSynchronizer(
            [self.pos_sub], queue_size=10, slop=0.5, allow_headerless=True)
        self.time_sync.registerCallback(self.track_only)
        self.enabled = True  # Enabled by default, can be paused by kcf_enable=false
        self.enable_sub = self.create_subscription(
            Bool, 'kcf_enable', self._enable_callback, 10)

    def track_only(self, position):
        if not self.enabled:
            return
        center_x, center_y = position.x, position.y
        if abs(center_x - 320) > 8 or abs(center_y - 240) > 8:
            self.dofbot_tracker.XY_track(center_x, center_y)

Key points:

• Only calls XY_track — no Clamping, no gripper action, no buzzer, no distance calculation.

• 8-pixel dead zone to avoid jitter.

• kcf_enable is optional: default True lets the robotic arm follow directly; the voice router publishing True has no side effects, publishing False can temporarily pause.