Mediapipe Palm Tracking (No-PTZ)

1. Experiment Objective

MediaPipe Hands is used to detect 21 keypoints of the palm, and the car rotates to track so that the palm stays at the center of the frame. It supports Tracking mode (rotation only) and Follow mode (rotation + distance keeping).

2. Experiment Principle

1. Keyword Explanation

2. Technical Architecture

/camera/image_raw (BEST_EFFORT)
  → hand_tracker_node
    ├── Image processing: image conversion → flip → MediaPipe Hands.process() → extract 21 keypoints
    ├── Target selection: palm center (5th keypoint)
    └── Body control timer (10Hz): pixel error → PD angular velocity + bounding box size distance following → /cmd_vel (BEST_EFFORT)

Requires a MediaPipe Python environment (ROS 2 and workspace environment).

3. Core Topic Quick Reference

Firmware Uplink

Firmware Downlink

3. Experiment Steps

1. Hardware Preparation

2. Start the Agent + Camera + Joint Model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start Palm Tracking (Terminal 2)

Tracking mode:

xxxxxxxxxx
ros2 launch microros_v2_no_ptz_visual_control_pkg hand_tracker.launch.py

6. Operation Instructions

4. Experiment Phenomenon

• When the palm moves in the frame, the car automatically rotates to track it, keeping the palm at the center of the frame
• The 21 hand keypoints and skeleton connections are displayed in real time
• Follow mode: the car backs away when the palm moves closer and moves forward when it moves away
• After the palm moves out of the frame, the car stops; when it reappears, tracking resumes automatically

5. Code Explanation

Source path:

• Launch (track): ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/hand_tracker.launch.py
• Launch (follow): ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/hand_follow.launch.py
• Node source: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_tracker_node.py
• Hand detector: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_detector.py

1. Core Node Analysis

hand_tracker_node uses HandDetector to wrap the MediaPipe Hands model. Core workflow:

• Image processing (15Hz): raw Image frame → cv_bridge conversion → flip → HandDetector.findHands() → extract 21 keypoints
• Target selection: the 5th keypoint (index finger MCP joint) is used as the palm center
• Body control (10Hz): deviation between the palm center x-coordinate and the image center → PD angular velocity; bounding box size deviation → PD linear velocity

Constructor — create the MediaPipe detector and publishers/subscribers:

​x
# Source file: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_tracker_node.py
​
def __init__(self) -> None:
    super().__init__(DEFAULT_NODE_NAME)
    self._tgt_lock = threading.Lock()
    self._state = STATE_IDLE
    self._has_target = False
    self._lost_frames = 0
    self._target_x: Optional[float] = None
    self._target_y: Optional[float] = None
    self._hand_size: Optional[float] = None  # Palm bounding box size
​
    self._declare_params()
    self._load_params()
​
    # MediaPipe hand detector
    self._hand_det = HandDetector()
​
    best_effort = QoSProfile(
        history=QoSHistoryPolicy.KEEP_LAST, depth=1,
        reliability=QoSReliabilityPolicy.BEST_EFFORT,
        durability=QoSDurabilityPolicy.VOLATILE,
    )
    self._img_sub = self.create_subscription(
        Image, self._img_topic, self._on_image, best_effort)
    self._cmd_pub = self.create_publisher(Twist, self._cmd_topic, best_effort)
​
    self._img_timer = self.create_timer(
        1.0 / max(float(self._img_rate), MINIMUM_TIMER_RATE_HZ), self._process_image)
    self._body_timer = self.create_timer(
        1.0 / max(float(self._body_rate), MINIMUM_TIMER_RATE_HZ), self._body_tick)

Image processing callback — MediaPipe detection + keypoint drawing:

xxxxxxxxxx
# Source file: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_tracker_node.py
​
def _process_image(self) -> None:
    """Image processing."""
    with self._img_lock:
        if self._latest_frame is None:
            return
        frame = self._latest_frame.copy()
​
    if frame is None:
        return
    frame = cv.resize(frame, (self._img_w, self._img_h))
    if self._flip_h:
        frame = cv.flip(frame, 1)
​
    # MediaPipe Hands detection + skeleton drawing
    frame, _ = self._hand_det.findHands(frame, draw=True)
​
    with self._tgt_lock:
        if len(self._hand_det.lmList) != 0:
            # Take the 5th keypoint (index finger MCP) as the palm center
            mx, my = self._hand_det.lmList[5][1], self._hand_det.lmList[5][2]
            # Compute the bounding box size (used for distance following)
            x_vals = [pt[1] for pt in self._hand_det.lmList]
            y_vals = [pt[2] for pt in self._hand_det.lmList]
            self._target_x = float(mx)
            self._target_y = float(my)
            self._hand_size = max(max(x_vals) - min(x_vals), max(y_vals) - min(y_vals))
            self._has_target = True
            self._lost_frames = 0
        else:
            self._lost_frames += 1
            if self._lost_frames > self._lost_frame_threshold:
                self._has_target = False

2. Key Parameters

Parameter definition file:

• Launch: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/hand_tracker.launch.py
• Node: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_tracker_node.py

3. Node Description

6. Common Problems