Welcome to MicroROS V2 Robot Car repository
5. Mediapipe Gesture Follow
Mediapipe Gesture Follow (No-PTZ)
1. Experiment Objective
Use MediaPipe gesture recognition to track the palm while also recognizing the gesture type, with body rotation tracking + distance following keeping the hand centered in the frame and maintaining a constant distance.
2. Experiment Principle
1. Key Terms
| Term | Description |
|---|---|
| Gesture Recognition | MediaPipe Hands 21 keypoints → fingersUp() computes the states of the 5 fingers → classifies 6 gestures |
| Distance Follow | Estimates distance from the hand bounding box size, with PD control maintaining the desired distance |
| Fingertip Counting | Determines whether a finger is extended from the distance between the fingertip keypoints (4/8/12/16/20) and the corresponding knuckle |
| QoS | best_effort (image input + cmd_vel downlink), depth=1 |
2. Technical Architecture
/camera/image_raw (BEST_EFFORT)→ gesture_tracker_node (enable_distance_follow:=true)├── MediaPipe Hands → 21 keypoints → fingersUp() classification└── PD angular velocity + bounding box size distance following → /cmd_vel
A MediaPipe Python environment is required (for both ROS2 and the workspace environment).
3. Experiment Steps
1. Hardware Preparation
| Hardware | Requirement |
|---|---|
| PTZ Version | ❌ |
| No-PTZ Version | ✅ |
| Required Peripherals | ESP32 camera (mounted on bracket) |
2. Start Agent + Camera + Joint Model (Terminal 1)
xxxxxxxxxxros2 launch microros_v2_bringup car_base.launch.py
3. Start Gesture Follow (Terminal 2)
xxxxxxxxxxros2 launch microros_v2_no_ptz_visual_control_pkg gesture_follow.launch.py
6. Operation Instructions
| Key | Function |
|---|---|
| Space | Toggle follow/standby |
| q / ESC | Exit |
4. Experiment Result
- Hand approaches → the car moves backward; moves away → the car moves forward to follow
- The frame displays the gesture type in real time (5/4/3/2/1 fingers / fist)
- 21 keypoints + skeleton connections visualized
- Hand moves out of the frame → stops moving
5. Code Explanation
Source code paths:
- Launch:
~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/gesture_follow.launch.py- Node source:
~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_tracker_node.py- Hand detector:
~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_detector.py
1. Core Node Explanation
gesture_follow.launch.py launches gesture_tracker_node, which adds fingersUp() gesture classification on top of palm tracking. Follow mode enables both rotation tracking and distance following — the former is based on the palm center x-coordinate error, and the latter on the bounding box size error.
2. Key Parameters
Parameter definition files:
- Launch:
~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/gesture_follow.launch.py- Node:
~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_tracker_node.py
| Parameter | Type | Default | Description |
|---|---|---|---|
hand_detection_confidence | float | 0.8 | Hand detection confidence threshold |
deadband_x_pixel | float | 50.0 px | Horizontal deadband |
angular_kp / angular_kd | float | 1.2 / 0.55 | Angle PD gains |
max_angular_z | float | 0.6 rad/s | Maximum turning speed |
enable_distance_follow | bool | true | Enable distance following |
target_hand_size_px | float | 90.0 px | Desired hand size |
distance_tolerance_hand_size_px | float | 12.0 px | Distance tolerance |
distance_kp / distance_kd | float | 0.05 / 0.03 | Distance PD gains |
max_linear_x | float | 0.2 m/s | Maximum forward/backward speed |
3. Node Description
| Node | Package | Description |
|---|---|---|
gesture_tracker_node | microros_v2_no_ptz_visual_control_pkg | MediaPipe gesture tracking + following, finger classification + dual PD |
6. Common Issues
| Issue | Cause | Solution |
|---|---|---|
| Gesture not recognized | Insufficient lighting or hand too far | Improve lighting and move closer to the camera |
| Incorrect gesture classification | Insufficient fingertip detection accuracy | Keep fingers separated and clearly visible |
| Distance following overshoot | PD gains too large | Lower distance_kp |