Hand Detection

1. Experiment Objective

Use MediaPipe Hands to detect the 21 3D hand keypoints in the camera image, with a dual-panel display (original image + skeleton pose).

2. Experiment Principle

1. Key Terms

2. Technical Architecture

Terminal 1: start agent + camera + joint models
Terminal 2: start hand detection

Data flow: ESP32 camera → /camera/image_raw → image conversion → MediaPipe Hands inference (21 keypoints + finger skeleton) → dual-panel display (left = original image overlay, right = skeleton on black background) → real-time display via cv2.imshow.

3. Experiment Steps

1. Hardware Preparation

2. Start the agent + camera + joint models (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start hand detection (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_mediapipe_pkg hand_detector.launch.py

4. Operating Instructions

Real-time dual-panel display: the left panel shows the original image with the hand skeleton overlaid (21 red dots + connecting lines), and the right panel shows the skeleton pose on a black background. Press q to exit.

5. How to Stop

Press Ctrl+C in each terminal, one by one, to stop.

4. Observed Results

• Put your palm in the camera's field of view: 21 keypoints are marked with red dots and the finger skeleton is drawn with colored connecting lines
• In two-hand mode, both hands are detected simultaneously, each annotated independently
• The right panel shows the skeleton on a pure black background, making keypoint position accuracy easy to inspect

5. Code Walkthrough

Source code paths:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/hand_detector.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/HandDetector.py

1. Core Node Analysis

HandDetector uses MediaPipe Hands to detect the 21 hand keypoints and visualizes them in a dual-panel display.

​x
# Source file: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/HandDetector.py
class HandDetector(Node):
    def __init__(self):
        super().__init__("hand_detector_node")
        self.declare_parameter("rgb_topic", "/camera/image_raw")
        self.declare_parameter("static_image_mode", False)
        self.declare_parameter("max_hands", 2)
        self.declare_parameter("detector_con", 0.5)
        self.declare_parameter("track_con", 0.5)
        self.declare_parameter("draw_on_source", False)
​
        # MediaPipe Hands initialization
        self.mp_hand = mp.solutions.hands
        self.hands = self.mp_hand.Hands(
            static_image_mode=static_image_mode,
            max_num_hands=max_hands,
            min_detection_confidence=detector_con,
            min_tracking_confidence=track_con)
​
    def image_callback(self, msg):
        frame = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        frame_rgb = cv.cvtColor(frame, cv.COLOR_BGR2RGB)
        results = self.hands.process(frame_rgb)
​
        # Dual panel: left = original image with skeleton overlaid, right = skeleton on pure black background
        black_board = np.zeros(frame.shape, dtype=np.uint8)
        if results.multi_hand_landmarks:
            for hand_landmarks in results.multi_hand_landmarks:
                self.mp_draw.draw_landmarks(frame, hand_landmarks,
                    self.mp_hand.HAND_CONNECTIONS, lm_draw_spec, conn_draw_spec)
                self.mp_draw.draw_landmarks(black_board, hand_landmarks,
                    self.mp_hand.HAND_CONNECTIONS)
        combined = np.hstack((frame, black_board))
        cv.imshow(self.window_name, combined)

Key logic:

• MEDIAPIPE_DISABLE_TF/GPU: the Ubuntu VM has no GPU, so the TensorFlow/GPU backend is disabled to avoid startup errors
• mediapipe.solutions.hands.Hands(): core detector; max_num_hands=2 supports two hands; min_detection_confidence=0.5 filters low-confidence detections
• static_image_mode=False (default): enables cross-frame tracking, reducing computation, suitable for video streams
• Dual panel via np.hstack: skeleton-overlaid original image + skeleton on a pure black background shown side by side, helping to inspect keypoint position accuracy

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/hand_detector.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/HandDetector.py

3. Node Description

6. Frequently Asked Questions