Gesture Control Car Action Group (No-PTZ)

Experiment Objective

Recognize 5 gestures to trigger preset chassis action sequences (square/circle/S-curve/back-and-forth/stop), with odometry-driven PD closed-loop control for completion determination. The No-PTZ version uses a fixed camera bracket.

Experiment Principle

1. Keyword Explanation

2. Technical Architecture

Three-state machine + dual timers (image 15Hz + action control 30Hz):

/camera/image_raw (BEST_EFFORT)
  → gesture_action_group_control_node
    ├── Image processing timer (15Hz): image conversion → MediaPipe → gesture recognition → confirmation count → trigger action
    └── Action control timer (30Hz): odometry PD closed loop → phase transition → /cmd_vel (BEST_EFFORT)
/odom_raw (BEST_EFFORT)
  → Odometry data drives PD

Gesture → action mapping:

PD closed-loop parameters:

• Distance phase: linear_kp(0.8)×error + linear_kd(0.25)×derivative → clamped to ±0.2 m/s
• Angle phase: absolute yaw target, angular_kp(0.05)×error + angular_kd(0.005)×derivative → clamped to ±0.6 rad/s
• Settle for 3 frames → cooldown for 10 ticks → switch phase

3. Core Topics Quick Reference

Firmware Uplink (Sensor Data)

Firmware Downlink (Control Commands)

Experiment Steps

1. Hardware Preparation

2. Start Agent + Camera + Joint Model (Terminal 1)

Start agent + camera + joint model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start Gesture Action Group Control (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_no_ptz_visual_control_pkg gesture_action_group_control.launch.py

4. Operation Instructions

Interaction

Safety and Boundary Handling

• 5-finger emergency stop interrupts any action
• Gesture must be confirmed for ≥3 frames to trigger
• Odometry drives phase transitions
• Angle wraps ±π
• Stop before quitting

Experiment Result

• idle/waiting: the car is stationary waiting for a gesture
• Yes gesture held for 3 frames: square 0.5m × 4 sides, looped once
• OK gesture: rotate 360° in place
• Rock gesture: S-curve
• Thumb Down: back-and-forth 0.5m
• 5-finger emergency stop: immediately interrupts and returns to waiting

Code Explanation

Source paths:

• Launch: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/gesture_action_group_control.launch.py
• Node source: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_action_group_control_node.py
• Hand detector: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/hand_detector.py

1. Core Node Analysis

gesture_action_group_control_node is the most complex node among the 4 documents, containing a three-state machine (idle/waiting/action_running), dual timers (image processing 15Hz + action control 30Hz), and an odometry PD closed loop.

Data flow:

• _process_image (15Hz) → _detect → MediaPipe gesture recognition → confirmation count → _start_action
• _action_tick (30Hz) → dispatches to _square_tick/_circle_tick/_s_curve_tick/_backforth_tick by action type → PD computation → /cmd_vel
• _odom_callback → caches odometry position and yaw

Constructor — declares parameters, creates dual timers, subscribers/publishers:

​x
# Source file: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_action_group_control_node.py
​
def __init__(self) -> None:
    super().__init__(DEFAULT_NODE_NAME)
    self._image_lock = threading.Lock()
    self._state_lock = threading.Lock()
    self._odom_lock = threading.Lock()
​
    self._state = STATE_IDLE         # State machine initial: idle
    self._latest_frame: Optional[np.ndarray] = None
​
    # Odometry state (drives PD)
    self._odom_x = 0.0
    self._odom_y = 0.0
    self._odom_yaw = 0.0
​
    # Gesture confirmation mechanism
    self._gesture_confirm_count = 0
    self._last_gesture = ""
​
    # Action engine state
    self._action_type = ""
    self._action_phase = 0
    self._phase_start_x = 0.0
    self._phase_start_y = 0.0
    self._phase_start_yaw = 0.0
    self._settle_count = 0
​
    self._declare_params()
    self._load_params()
​
    self._hand_det = HandDetector()
​
    # Subscriber QoS uniformly BEST_EFFORT (image + odometry)
    best_effort_qos = QoSProfile(
        history=QoSHistoryPolicy.KEEP_LAST, depth=1,
        reliability=QoSReliabilityPolicy.BEST_EFFORT,
        durability=QoSDurabilityPolicy.VOLATILE,
    )
    self._img_sub = self.create_subscription(
        Image, self._img_topic, self._on_image, best_effort_qos)
    self._odom_sub = self.create_subscription(
        Odometry, self._odom_topic, self._odom_callback, best_effort_qos)
    self._cmd_pub = self.create_publisher(Twist, self._cmd_topic, best_effort_qos)
​
    # Dual timers: image processing 15Hz + action control 30Hz
    self._img_timer = self.create_timer(
        1.0 / max(float(self._img_rate), MINIMUM_TIMER_RATE_HZ), self._process_image)
    self._action_timer = self.create_timer(
        1.0 / max(float(self._action_rate), MINIMUM_TIMER_RATE_HZ), self._action_tick)
​
    # Keyboard polling timer (50Hz)
    if self._show_win:
        self._ensure_win()
        self._key_timer = self.create_timer(0.05, self._poll_keys)
​
    self._param_cb = self.add_on_set_parameters_callback(self._on_param_update)

Action control timer callback — three-state machine scheduling:

xxxxxxxxxx
# Source file: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_action_group_control_node.py
​
def _action_tick(self) -> None:
    """Action control timer (30Hz), three-state machine scheduling."""
    if self._state == STATE_IDLE:
        self._pub_zero_cmd()       # idle: zero velocity
        return
    if self._state == STATE_WAITING:
        self._pub_zero_cmd()       # waiting: zero velocity, wait for gesture
        return
    if self._state != STATE_ACTION:
        return
​
    with self._odom_lock:
        ox = self._odom_x
        oy = self._odom_y
        oyaw = self._odom_yaw
​
    phase_dist = math.hypot(ox - self._phase_start_x, oy - self._phase_start_y)
​
    # Dispatch to different tick functions by action type
    if self._action_type == GESTURE_SQUARE:
        self._square_tick(oyaw, phase_dist)
    elif self._action_type == GESTURE_CIRCLE:
        self._circle_tick(oyaw)
    elif self._action_type == GESTURE_S:
        self._s_curve_tick(oyaw)
    elif self._action_type == GESTURE_BACKFORTH:
        self._backforth_tick(phase_dist)

Key logic: _distance_pd and _angle_pd compute the PD outputs for distance and angle respectively, with settle counting for determination. The S-curve action is an alternating turn→straight→turn→... sequence, driven by the s_turn_angles and s_straight_distances parameters.

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/launch/gesture_action_group_control.launch.py
• Node: ~/microros_ws/src/microros_v2_no_ptz_visual_control_pkg/microros_v2_no_ptz_visual_control_pkg/gesture_action_group_control_node.py

3. Node Description

Common Problems