Mediapipe Gesture Tracking (PTZ)

1. Experiment Objective

The PTZ pan-tilt tracks the gesture position while recognizing the gesture class (finger-count code and gesture name) and displaying it in real time.

2. Experiment Principle

1. Key Terms

2. Technical Architecture

​x
ESP32 camera (firmware) → /camera/image_raw → gesture_tracker_ptz_node → /cmd_ptz_angle → firmware servo control
​
The state estimation layer (ekf_filter_node + camera_node) provides the image data stream

Node chain: state estimation (ekf_filter_node + camera_node) → gesture_tracker_ptz_node → /cmd_ptz_angle → firmware servo control

3. Core Topics Quick Reference

Firmware Uplink (data upstream):

Firmware Downlink (command downstream):

3. Experiment Steps

1. Hardware Preparation

2. Start Agent + Camera + Joint Model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start Gesture Tracking (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_ptz_visual_control_pkg gesture_tracker_ptz.launch.py

After startup, the frame shows the idle state; palm detection and gesture recognition run normally but no pan-tilt control is output.

4. Operation Instructions

1. Workflow

After startup, the node uses two independent timers: image processing (image_processing_rate_hz = 15 Hz) + pan-tilt control (control_rate_hz = 20 Hz). Minimal two-state machine: idle ↔ tracking.

(1) Image reception and preprocessing

Subscribes to /camera/image_raw (QoS: BEST_EFFORT, KEEP_LAST); the image callback converts and caches the raw Image. The image processing timer takes a copy of the latest frame, scales it to the processing resolution (processing_width×processing_height = 320×240), and flips/rotates as needed.

(2) MediaPipe palm detection + gesture recognition

Uses HandDetector (detectionCon=hand_detection_confidence=0.8) to detect the palm and extract:

• The 21 hand landmark coordinates (lmList)
• fingersUp(): a 5-bit finger extension state list, 0=bent / 1=extended
• finger_count: the total number of extended fingers
• get_gesture(): the gesture name (e.g., "Scissors", "OK", "Thumbs Up", etc.)
• finger_code: the finger code string (e.g., "[0,1,1,0,0]")

(3) Tracking anchor extraction

Takes landmark 5 (index finger MCP joint) as the tracking target point. Draws the tracking point: a purple solid circle (radius 8 px) + a yellow hollow circle (radius 10 px) + the MediaPipe hand landmarks and connections.

(4) PD pan-tilt control (tracking + hand present + not a fist only)

Fixed-rate 20 Hz PD control: tracking point pixel coordinates → normalized error → PD step → limiting (±max_ptz_step=1.2°) → step smoothing (α=ptz_step_smooth_alpha=0.35) → direction reversal (horizontal not reversed, tilt reversed) → angle smoothing (α=ptz_smooth_alpha=0.85) → clamp the angle → publish.

When there is no hand or a fist (fc==0), no control is output, but the state is not changed and the PD history is not reset — tracking resumes seamlessly when the palm opens again.

(5) Gesture data display

The gesture name, finger code, and finger count are overlaid in real time at the top-left of the frame; "TRACK" is shown in the TRACKING state and "FIST" when a fist is detected.

2. Interaction

On-screen overlay: The debug window shows the state (idle/tracking), frame rate, gesture name, finger code, finger count (with the FIST marker), tracking point error H/V, PD parameters (kp/kd/deadband/step), pan-tilt angle, and control rate. A crosshair is drawn at the frame center + an orange deadband rectangle.

Optional legend: With overlay_show_legend=true, the "SPACE:toggle | i:idle | q:quit" key hint is shown.

3. Safety and Edge Cases

• No hand/fist, no control: When there is no hand or a fist, the PD outputs nothing, but the state is not changed and the PD history is not reset, for a seamless handoff when tracking resumes
• Gesture detection never changes the state: Only the Space and i keys switch states; the detection result only affects whether control is output
• Pan-tilt angle limiting: Horizontal [-90°, 90°], tilt [-90°, 20°]
• Deadband protection: The step is forced to zero when the tracking point offset is within the deadband (20 px)
• Initial PTZ publish at startup: With publish_initial_ptz_command=true, the initial angle (0°, -45°) is published 20 times at 0.2 s intervals
• Exit strategy: Press q/ESC to exit directly with os._exit(0)
• Runtime parameter validation: Range checks cover all PD/angle/image parameters; illegal values are rejected; image_topic/ptz_command_topic/window_name require a node restart

5. How to Stop

Press Ctrl+C in sequence to stop the processes in the three terminals: Terminal 2 (gesture tracking) → Terminal 1 (chassis, agent, camera, and model).

4. Experiment Observations

• idle state: The frame streams in real time; palm detection and gesture recognition run normally but no pan-tilt control is output, and the state is idle
• tracking + hand present + fingers extended: The pan-tilt follows the index finger MCP joint with PD control, the MediaPipe landmarks and connections are overlaid, and the gesture name and finger code update in real time
• tracking + fist: The frame shows "FIST", the pan-tilt stays still at its current position, and the PD history is kept; tracking resumes automatically when the palm opens again
• tracking + palm moved out of frame: The gesture display changes to "None", the pan-tilt stays at its last position, and tracking resumes when the palm reappears
• Palm within the deadband: The step is zeroed and the pan-tilt stays completely still without jitter
• Gesture changes: The gesture name updates in real time when different gestures are made, and tracking is unaffected (gesture detection and pan-tilt control are fully decoupled)
• Press i to return to idle: The pan-tilt stays at its current position; press Space to re-enter tracking

5. Program Analysis

Source code paths:

• Launch file: ~/microros_ws/src/microros_v2_ptz_visual_control_pkg/launch/gesture_tracker_ptz.launch.py
• Node file: ~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/gesture_tracker_ptz_node.py
• Hand detection module: ~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/hand_detector.py

1. Core Node Analysis

The node GestureTrackerPtzNode inherits from rclpy.node.Node and adds the full get_gesture() gesture classification logic on top of HandDetector, so it can recognize and display multiple gesture classes.

Constructor __init__ core logic:

xxxxxxxxxx
def __init__(self) -> None:
    super().__init__(DEFAULT_NODE_NAME)
    self._image_lock = threading.Lock()
    self._gesture_lock = threading.Lock()
​
    # State: IDLE / TRACKING only
    self._state = STATE_IDLE
​
    # Gesture shared data
    self._has_hand = False
    self._finger_count = 0
    self._gesture_name = "None"
    self._finger_code = "[0,0,0,0,0]"
    self._target_x: Optional[float] = None
    self._target_y: Optional[float] = None
​
    # PTZ angles (initial 0°, -45°)
    self._h_deg = 0.0
    self._p_deg = 0.0
​
    # PD history state
    self._prev_err_x = 0.0
    self._prev_err_y = 0.0
    self._last_sx = 0.0
    self._last_sy = 0.0
​
    self._declare_params()
    self._load_params()
    self._h_deg = float(self._init_h_deg)
    self._p_deg = float(self._init_p_deg)
​
    # Initialize the MediaPipe gesture detector (with the confidence parameter)
    self._hand_det = HandDetector(detectorCon=float(self._hand_conf))
​
    # Image subscription (BEST_EFFORT, KEEP_LAST depth=1, VOLATILE)
    self._img_sub = self.create_subscription(
        Image, self._img_topic, self._on_image,
        QoSProfile(history=QoSHistoryPolicy.KEEP_LAST, depth=1,
                   reliability=QoSReliabilityPolicy.BEST_EFFORT,
                   durability=QoSDurabilityPolicy.VOLATILE),
    )
​
    # Pan-tilt angle publishing
    self._ptz_pub = self.create_publisher(Vector3, self._ptz_topic, 10)
​
    # Image processing timer (15 Hz)
    self._img_timer = self.create_timer(
        1.0 / max(float(self._img_rate), MINIMUM_TIMER_RATE_HZ),
        self._process_image,
    )
    # Pan-tilt control timer (20 Hz)
    self._ctrl_timer = None
    self._rebuild_ctrl_timer(False)
​
    # Initial PTZ timed publishing (20 times, 0.2 s interval each)
    if self._pub_init_ptz:
        self._start_init_ptz()

MediaPipe gesture detection _detect:

xxxxxxxxxx
def _detect(self, frame: np.ndarray) -> None:
    self._hand_det.findHands(frame, draw=True)
​
    if len(self._hand_det.lmList) == 0:
        with self._gesture_lock:
            self._has_hand = False
        return
​
    # Extract landmark 5 (index finger MCP) as the tracking target
    mx, my = self._hand_det.findPoint(LANDMARK_INDEX_FINGER_MCP)
​
    # Finger extension state analysis
    fingers = self._hand_det.fingersUp()
    nf = [1 if int(v) != 0 else 0 for v in fingers]
    fc = int(sum(nf))
    fcode = "[" + ",".join(str(v) for v in nf) + "]"
​
    # Gesture class recognition
    gn = str(self._hand_det.get_gesture())
​
    # Draw the tracking point
    cv.circle(frame, (int(mx), int(my)), 8, (255, 0, 255), -1)
    cv.circle(frame, (int(mx), int(my)), 10, (255, 255, 0), 2)
​
    # Update the shared data
    with self._gesture_lock:
        self._has_hand = True
        self._target_x = float(mx)
        self._target_y = float(my)
        self._finger_count = fc
        self._finger_code = fcode
        self._gesture_name = gn

Gesture recognition and pan-tilt control decoupling design:

The key difference from the palm tracking node is that the gesture tracking node additionally displays the following data in the on-screen overlay:

xxxxxxxxxx
State:tracking FPS:15
Gesture:Scissors Finger:[0,1,1,0,0] Count:2 TRACK
Target:(LM5) Error H:12.3 V:-5.6
PD kp:1.50 kd:0.35 dead:(20,20) step:1.2
PTZ s1:2.3 s2:-42.1 hz:20
SPACE:toggle | i:idle | q:quit

The get_gesture() method maps the 5-bit fingersUp() code to a gesture name:

The PD control logic _ctrl_tick is identical to the palm tracking node, so it is not repeated here.

2. Key Parameters

Parameter definition file: ~/microros_ws/src/microros_v2_ptz_visual_control_pkg/launch/gesture_tracker_ptz.launch.py

6. Frequently Asked Questions