Welcome to MicroROS V2 Robot Car repository
3. Gesture Control Car Action Group
Gesture Control Car Action Group (PTZ)
1. Experiment Objective
Recognizing 5 gestures triggers preset chassis action sequences (square/circle/S-curve/back-and-forth/stop), with an odometry-driven PD closed loop completing the determination. The PTZ pan-tilt switches its angle according to the state.
2. Experiment Principle
1. Key Terms
| Term | Description |
|---|---|
| Action Group | A preset chassis motion sequence; odometry data drives the phase transitions |
| Odometry Closed-Loop PD | Subscribes to /odom_raw; each phase drives to the target distance/angle with PD, then cools down before the next phase |
| Gesture-to-Action Mapping | Yes→square / OK→circle / Rock→S-curve / ThumbDown→back-and-forth / 5 fingers→emergency stop |
| Gesture Confirmation Count | A gesture must be held for 3 consecutive frames to trigger an action, preventing false triggers |
| Phase Cooldown Period | After the PD settles, stops for 10 ticks to let the robot stabilize before switching phases |
| PTZ State Switching | waiting state (0°, -15°); action state (0°, -45°) tilts down to observe the ground |
| QoS | best_effort (image/odometry + cmd_vel), depth=1 |
2. Technical Architecture
Three-state machine (idle/waiting/action) + dual timers (image at 15 Hz + action at 30 Hz):
/camera/image_raw (BEST_EFFORT)→ gesture_action_group_control_ptz_node├── Image processing (15 Hz): MediaPipe → gesture recognition → confirmation count → trigger action├── Action control (30 Hz): odometry PD closed loop → phase transition → /cmd_vel (BEST_EFFORT)└── PTZ: pan-tilt angle switched by state (waiting: 0°, -15° / action: 0°, -45°)/odom_raw (BEST_EFFORT) → odometry-driven PD
3. Core Topics Quick Reference
Firmware Downlink
| Topic | Message Type | QoS | Field | Range | Direction |
|---|---|---|---|---|---|
/cmd_vel | geometry_msgs/Twist | best_effort + keep_last(1) | linear.x | -0.20~0.20 m/s | += forward, −= backward |
angular.z | -1.00~1.00 rad/s | += turn left, −= turn right | |||
/cmd_ptz_angle | microros_v2_msgs/PtzAngle | reliable | horizontal_deg | 0° | Fixed horizontal |
pitch_deg | -45°~-15° | Switched by state |
3. Experiment Steps
1. Hardware Preparation
| Hardware | Requirement |
|---|---|
| PTZ version | ✅ |
| No-PTZ version | ❌ |
| Required peripherals | ESP32 camera (pan-tilt) + PTZ servo |
2. Activate the MediaPipe Environment
xxxxxxxxxxsource /opt/ros/humble/setup.bashsource ~/microros_ws/install/setup.bash3. Start Agent + Camera + Joint Model (Terminal 1)
xxxxxxxxxxros2 launch microros_v2_bringup car_base.launch.py
4. Start Gesture Action Group Control (Terminal 2)
xxxxxxxxxxros2 launch microros_v2_ptz_visual_control_pkg gesture_action_group_control_ptz.launch.py
5. Operation Instructions
Gesture-to-Action Mapping
| Gesture | Finger Code | Action |
|---|---|---|
| Yes | [0,1,1,0,0] | square (0.5 m square, 4 sides) |
| OK | ≥3 fingers, index bent | circle (spin 360° in place) |
| Rock | [0,1,0,0,1] | s_curve (S curve) |
| Thumb Down | ≤1 finger, thumb down | backforth (back-and-forth 0.5 m) |
| 5 fingers extended | [1,1,1,1,1] | stop (emergency stop) |
| Key | Function |
|---|---|
| Space | waiting ↔ idle |
| i / I | Back to idle |
| q / Q / ESC | Exit |
4. Experiment Observations
- idle: PTZ (0°, -15°), gestures are detected but do not trigger actions
- waiting: The car body is stationary, waiting for a gesture
- Yes gesture held for 3 frames: 0.5 m square (4 sides), PTZ tilts down to -45°
- OK gesture: Spins 360° in place
- Rock gesture: S curve
- Thumb Down: Back-and-forth 0.5 m
- 5-finger emergency stop: Interrupts immediately and returns to waiting
5. Program Analysis
Source code paths:
- Launch:
~/microros_ws/src/microros_v2_ptz_visual_control_pkg/launch/gesture_action_group_control_ptz.launch.py- Node:
~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/gesture_action_group_control_ptz_node.py- Hand detector:
~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/hand_detector.py
1. Core Node Analysis
gesture_action_group_control_ptz_node adds PTZ state switching on top of the No-PTZ version: the pan-tilt angle is (0°, -15°) in the waiting state, and switches to (0°, -45°) in the action state to tilt down and observe the ground motion.
PTZ state-switching logic:
x# Source file: ~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/gesture_action_group_control_ptz_node.pydef _enter_waiting(self) -> None: """waiting state: pan-tilt (0°, -15°)""" self._state = STATE_WAITING self._set_ptz(self._waiting_h_deg, self._waiting_p_deg) # (0°, -15°) self._pub_zero_cmd()def _start_action(self, gesture_key: str) -> None: """action state: pan-tilt (0°, -45°) tilted down""" self._action_type = gesture_key self._set_ptz(0.0, -45.0) # action: tilt down to observe the ground self._state = STATE_ACTIONKey logic: three-state machine (idle/waiting/action) + odometry PD closed loop + PTZ pan-tilt state linkage. The 5-finger emergency stop can interrupt any action and return to waiting.
2. Key Parameters
Parameter definition files:
- Launch:
~/microros_ws/src/microros_v2_ptz_visual_control_pkg/launch/gesture_action_group_control_ptz.launch.py- Node:
~/microros_ws/src/microros_v2_ptz_visual_control_pkg/microros_v2_ptz_visual_control_pkg/gesture_action_group_control_ptz_node.py
| Parameter | Type | Default Value | Description |
|---|---|---|---|
linear_kp / linear_kd | float | 0.5 / 0.1 | Distance PD gains |
angular_kp / angular_kd | float | 0.08 / 0.15 | Angle PD gains |
max_linear_x | float | 0.2 m/s | Maximum linear velocity |
max_angular_z | float | 1.0 rad/s | Maximum angular velocity |
distance_tolerance | float | 0.03 m | Distance determination tolerance |
angle_tolerance_deg | float | 5.0° | Angle determination tolerance |
settle_frames | int | 5 | Frames for the settle determination |
phase_cooldown_ticks | int | 10 | Cooldown ticks between phases |
gesture_hold_confirm_frames | int | 3 | Gesture confirmation frames |
square_forward_distance | float | 0.5 m | Square side length |
circle_total_angle_deg | float | 360.0° | Total circle angle |
backforth_distance | float | 0.5 m | Back-and-forth distance |
waiting_horizontal_deg | float | 0.0° | waiting pan-tilt horizontal angle |
waiting_pitch_deg | float | -15.0° | waiting pan-tilt pitch angle |
3. Node Description
| Node | Package | Description |
|---|---|---|
gesture_action_group_control_ptz_node | microros_v2_ptz_visual_control_pkg | PTZ gesture action group, three-state machine + PD closed loop + pan-tilt linkage |
6. Frequently Asked Questions
| Problem | Cause | Solution |
|---|---|---|
| Action group interrupted | No data on the /odom_raw odometry topic | Check the odometry topic |
| Gesture does not trigger | Not enough confirmation frames | Hold the gesture for more than 3 frames |
| Square drifts off course | Angle PD parameters not suited to the floor | Tune angular_kp/angular_kd |
| No detection at startup | The MediaPipe environment was not sourced | source /opt/ros/humble/setup.bashsource ~/microros_ws/install/setup.bash |