Gesture Recognition

1. Experiment Objective

Use MediaPipe Hands combined with custom gesture classification logic to recognize 13 predefined gestures in real time (number gestures, special gestures, etc.).

2. Experiment Principle

1. Key Terms

2. Technical Architecture

Terminal 1: start agent + camera + joint models
Terminal 2: start gesture recognition

3. Experiment Steps

1. Hardware Preparation

2. Start the agent + camera + joint models (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start gesture recognition (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_mediapipe_pkg mediapipe_gesture.launch.py

4. Operating Instructions

• Make one of the 13 gestures at the camera; the recognized result and FPS are shown at the top of the frame
• Press q to exit.

4. Observed Results

• Making number gestures (0-8) shows the corresponding gesture name at the top of the frame (e.g. "Five", "Ok")
• Finger extended/bent state is determined in real time through the fingersUp array
• A semi-transparent black overlay shows the gesture name, FPS and operation hints

5. Code Walkthrough

Source code paths:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/mediapipe_gesture.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/mediapipe_gesture.py
• Shared library: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/media_library.py

1. Core Node Analysis

​x
# Source file: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/mediapipe_gesture.py
class MediapipeGestureNode(Node):
    def __init__(self):
        super().__init__("mediapipe_gesture_node")
        self.declare_parameter("rgb_topic", "/camera/image_raw")
        self.declare_parameter("static_image_mode", False)
        self.declare_parameter("max_hands", 1)
        self.declare_parameter("detector_con", 0.7)
        self.declare_parameter("track_con", 0.7)
        self.declare_parameter("print_gesture", False)
        # Use the shared library HandDetector (wraps MediaPipe Hands + finger state determination)
        self.hand_detector = HandDetector(
            static_image_mode=False, max_hands=1,
            detector_con=0.7, track_con=0.7)
​
    def image_callback(self, msg):
        frame = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        frame = self.hand_detector.findHands(frame)
        fingers_up = self.hand_detector.fingersUp()
        gesture = self.hand_detector.get_gesture(fingers_up)
        if gesture and self.print_gesture:
            self.get_logger().info(f'Gesture: {gesture}')

Key logic:

• HandDetector comes from the shared library media_library.py, encapsulating MediaPipe Hands and the gesture recognition logic
• fingersUp(): compares the fingertip y and pip joint y of the 5 fingers, returning a 5-bit array (1=extended, 0=bent)
• get_gesture(): classifies based on fingersUp + thumb-index distance into one of the 13 gestures
• detector_con=0.7: higher than the 0.5 used in basic hand detection, ensuring gesture recognition accuracy
• print_gesture: when enabled, the recognized gesture name is printed to the ROS2 log (for debugging)

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/mediapipe_gesture.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/mediapipe_gesture.py

3. Node Description

6. Frequently Asked Questions