Brush

1. Experiment Objective

Use MediaPipe hand tracking to implement a virtual paint brush: draw on the camera image with the index fingertip, with support for selecting multiple colors and erasing.

2. Experiment Principle

1. Key Terms

2. Technical Architecture

Terminal 1: start agent + camera + joint models
Terminal 2: start the virtual paint brush

3. Experiment Steps

1. Hardware Preparation

2. Start the agent + camera + joint models (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start the virtual paint brush (Terminal 2)

xxxxxxxxxx
ros2 launch microros_v2_mediapipe_pkg virtual_paint.launch.py

4. Operating Instructions

• Draw: extend the index finger and move the fingertip across the frame to draw lines
• Select color: touch the color bar at the top of the frame with the index fingertip to switch colors
• Erase: select Eraser or overwrite with a specific color
• Press q to exit

4. Observed Results

• As the index fingertip moves across the frame, continuous colored lines are left on the canvas
• The color bar at the top of the frame provides multiple color choices
• The drawing canvas is overlaid on the original image

5. Code Walkthrough

Source code paths:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/virtual_paint.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/VirtualPaint.py

1. Core Node Analysis

​x
# Source file: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/VirtualPaint.py
class VirtualPaint(Node):
    def __init__(self):
        super().__init__("virtual_paint_node")
        self.mp_hand = mp.solutions.hands
        self.hands = self.mp_hand.Hands(
            static_image_mode=False, max_num_hands=1,
            min_detection_confidence=0.5, min_tracking_confidence=0.5)
        # Canvas initialization
        self.canvas = None
        # Color list + top color bar
        self.colors = [(255,0,0), (0,255,0), (0,0,255), ...]
        self.current_color = self.colors[0]
        self.prev_pos = None
​
    def image_callback(self, msg):
        frame = self.bridge.imgmsg_to_cv2(msg, 'bgr8')
        if self.canvas is None:
            self.canvas = np.zeros_like(frame)
        frame = cv.flip(frame, 1)  # mirror the frame for hand-eye coordination
​
        # Draw the top color selection bar
        for i, color in enumerate(self.colors):
            cv.rectangle(frame, (i*80, 0), ((i+1)*80, 60), color, -1)
​
        results = self.hands.process(cv.cvtColor(frame, cv.COLOR_BGR2RGB))
        if results.multi_hand_landmarks:
            for hand_landmarks in results.multi_hand_landmarks:
                # Index fingertip = landmark 8
                cx = int(hand_landmarks.landmark[8].x * frame.shape[1])
                cy = int(hand_landmarks.landmark[8].y * frame.shape[0])
                # Color selection area (y < 60)
                if cy < 60:
                    self.current_color = self.colors[cx // 80]
                else:
                    if self.prev_pos:
                        cv.line(self.canvas, self.prev_pos, (cx, cy),
                            self.current_color, 10)
                    self.prev_pos = (cx, cy)
        frame = cv.addWeighted(frame, 0.5, self.canvas, 0.5, 0)
        cv.imshow(self.window_name, frame)

Key logic:

• cv.flip(frame, 1): mirrors the frame horizontally so hand movement matches intuition (hand-eye coordination)
• Index fingertip = landmark index 8 (the 8th of the 21 points); coordinates are normalized to 0~1 and must be multiplied by the image width/height
• Color selection area y < 60: when the hand is raised to the top of the frame it touches the color bar; the color index is mapped from the x coordinate divided by 80
• The canvas is an independent layer self.canvas; after drawing lines on the canvas, cv.addWeighted blends it semi-transparently with the original image

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_mediapipe_pkg/launch/virtual_paint.launch.py
• Node: ~/microros_ws/src/microros_v2_mediapipe_pkg/microros_v2_mediapipe_pkg/VirtualPaint.py

3. Node Description

6. Frequently Asked Questions