Welcome to DOFBOT-Ultra repository

KCF Object Tracking

Please make sure the web control program backend process is not running in the background. To close it, refer to [Get Started Quickly / 3. Access the Terminal] → [Disable the web control program backend process that starts on boot]. After closing, please restart DofbotUltra!!!

The web control program backend process must be closed before starting this function.

About the tracking algorithm: This example uses the CSRT algorithm by default (high accuracy, more robust to deformation, and better OpenCV compatibility on the Pi than KCF). The section title retains "KCF" following industry convention. The KCF_Tracker node actually encapsulates multiple trackers uniformly, and can be switched to KCF / MOSSE / MIL etc. via the ROS2 parameter tracker_type.

This tutorial covers tracking only, not grasping.

1. Function Description

After the program starts, use the mouse to draw a bounding box around the object to be tracked in the image. Release the mouse to start tracking (no need to press Space). KCF_Tracker continuously publishes the center coordinates of the target in the image to the /pos_xyz topic. KCF_TrackOnly subscribes to this topic and calls DofbotTrack.XY_track(x, y) to coordinate joints 1/2/3/4 of the robotic arm, keeping the tracked object's center aligned with the image center. Throughout the process, no gripper closing, descending, placement, or other grasping actions are triggered.

This tutorial uses the generic launch command base_arm_kin.launch.py.

The launch file automatically switches implementation based on the CAMERA_TYPE environment variable:

• CAMERA_TYPE=DABAI_DCW2: executes the tracking pipeline using color image + depth image;
• Other values: executes the RGB-only tracking pipeline for standard USB cameras.

2. Launch and Operation

2.1. Launch Commands

Open three terminals and execute in order:

​x
# Terminal 1: Start camera + low-level control + inverse kinematics
ros2 launch dofbot_ultra_driver base_arm_kin.launch.py
​
# Terminal 2: Start KCF/CSRT visual tracking node (opens an image window)
ros2 run dofbot_ultra_kcf KCF_Tracker
​
# Terminal 3: Start robotic arm following node (subscribes to /pos_xyz, no grasping)
ros2 run dofbot_ultra_kcf KCF_TrackOnly
​
# Set depth camera exposure
ros2 service call /camera/set_color_exposure orbbec_camera_msgs/srv/SetInt32 '{data: 30}'
ros2 service call /camera/set_depth_auto_exposure std_srvs/srv/SetBool '{data: true}'

To switch the tracking algorithm, specify the parameter when launching KCF_Tracker, for example:

xxxxxxxxxx
ros2 run dofbot_ultra_kcf KCF_Tracker --ros-args -p tracker_type:=KCF

Notes:

1. base_arm_kin.launch.py is the generic entry point, suitable for both camera modes.
2. When CAMERA_TYPE is set to standard USB mode, KCF_Tracker automatically switches to RGB-only tracking implementation but still continuously publishes /pos_xyz for KCF_TrackOnly to consume.

2.2. Operation

• After the program starts, in the image window opened by KCF_Tracker, hold the left mouse button and drag to draw a rectangular box around the target object, then release.
• A green tracking box will appear in the image. The title bar / terminal will continuously update the pixel coordinates of the target. In depth camera mode, the distance is also displayed. In standard USB mode, the distance field shows an invalid placeholder value, which does not affect pure tracking.
• The KCF_TrackOnly node starts in automatic following mode as soon as it launches. As long as /pos_xyz has data, it drives the robotic arm to follow; no additional key press is required.
• Slowly move the object, and the robotic arm will synchronously adjust its pose to keep the object near the image center.
• To reselect a target, press R to reset KCF_Tracker, then draw a new box.
• To stop the robotic arm, simply close the KCF_TrackOnly node (Ctrl+C).

3. Node and Topic Relationships

xxxxxxxxxx
+----------------+      /pos_xyz       +----------------+      Serial       +-----------+
|  KCF_Tracker   | ------------------> | KCF_TrackOnly  | -------------> | Robotic Arm |
| (Camera+CSRT   |                     | (XY_track only)|                |            |
|  tracking)     |                     +----------------+                +-----------+
+----------------+

• KCF_Tracker: Provides mouse ROI selection and tracking algorithms, publishes /pos_xyz. In depth camera mode, z is the distance value; in standard USB mode, z is an invalid placeholder value, used only to maintain message format compatibility.
• KCF_TrackOnly: Subscribes to /pos_xyz, calls DofbotTrack.XY_track for each frame's position, coordinating servos 1/2/3/4 for following. Does not call Clamping, so no descending, gripper closing, placement, or similar actions occur.

4. Core Code Analysis

4.1. KCF_Tracker.py (Default CSRT)

Code path:

xxxxxxxxxx
Raspberry Pi: /home/yahboom/dofbot_ultra_ws/src/dofbot_ultra_kcf/dofbot_ultra_kcf/KCF_Tracker.py
RDK: /home/sunrise/dofbot_ultra_ws/src/3d/dofbot_ultra_kcf/dofbot_ultra_kcf/KCF_Tracker.py

Key logic:

xxxxxxxxxx
self.declare_parameter('tracker_type', 'CSRT')
self.tracker_type = str(self.get_parameter('tracker_type').value).strip().upper() or 'KCF'
self.gTracker = Tracker(tracker_type=self.tracker_type)
​
# In kcfTrack callback, continuously publish target coordinates
pos = Position()
pos.x = center_x
pos.y = center_y
pos.z = self.cur_distance   # 999 when depth is invalid
self.pub_pos.publish(pos)

4.2. KCF_TrackOnly.py (Robotic Arm Following, No Grasping)

Code path:

xxxxxxxxxx
Raspberry Pi: /home/yahboom/dofbot_ultra_ws/src/dofbot_ultra_kcf/dofbot_ultra_kcf/KCF_TrackOnly.py
RDK: /home/sunrise/dofbot_ultra_ws/src/3d/dofbot_ultra_kcf/dofbot_ultra_kcf/KCF_TrackOnly.py

Complete core code:

xxxxxxxxxx
class KCFTrackOnlyNode(Node):
    def __init__(self):
        super().__init__('KCF_track_only')
        self.dofbot_tracker = DofbotTrack()
​
        self.pos_sub = Subscriber(self, Position, '/pos_xyz')
        self.time_sync = ApproximateTimeSynchronizer(
            [self.pos_sub], queue_size=10, slop=0.5, allow_headerless=True)
        self.time_sync.registerCallback(self.track_only)
​
        # Following enabled by default; can be temporarily paused by publishing /kcf_enable=false
        self.enabled = True
        self.enable_sub = self.create_subscription(
            Bool, 'kcf_enable', self._enable_callback, 10)
​
    def _enable_callback(self, msg):
        self.enabled = bool(msg.data)
        if self.enabled:
            self.dofbot_tracker.stop_flag = False
​
    def track_only(self, position):
        if not self.enabled:
            return
        center_x, center_y = position.x, position.y
        if getattr(self.dofbot_tracker, 'stop_flag', False):
            return
        # Only adjust when the target deviates from the image center by more than 8 pixels, to avoid jitter
        if abs(center_x - 320) > 8 or abs(center_y - 240) > 8:
            self.dofbot_tracker.XY_track(center_x, center_y)

Key points:

• enabled = True — The node starts following immediately upon launch, without relying on the SPACE key or an additional enable signal.
• abs(center_x-320) > 8 or abs(center_y-240) > 8 — An 8-pixel dead zone prevents small jitters from causing frequent micro-adjustments by the arm.
• No call to Clamping — No descending, gripper closing, placement, or other grasping actions are executed. This is the core difference from KCF_TrackAndGrap.