Welcome to DOFBOT-Ultra repository

1. Hand detection

Hand Detection

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Use MediaPipe Hands to detect hand keypoints (21 3D coordinates) in real-time from the camera feed, drawing the hand skeleton on both the original image and a black background. Supports simultaneous detection of two hands.

1、Startup

1.1、Startup

Enter the following command to start the program


ros2 run dofbot_mediapipe 01_HandDetector.py

Press the q key in the image window or Ctrl+C in the terminal to exit the program.

1.2、Source Code

Raspberry Pi source path:


xxxxxxxxxx
~/dofbot_ultra_ws/src/dofbot_mediapipe/scripts/01_HandDetector.py

RDK X5 source path:


xxxxxxxxxx
~/dofbot_ultra_ws/src/2d/dofbot_mediapipe/scripts/01_HandDetector.py


x
#!/usr/bin/env python3
# encoding: utf-8
import cv2 as cv
import numpy as np
import mediapipe as mp
from dofbot_utils.camera import create_camera
from dofbot_utils.fps import FPS


class HandDetector:
    def __init__(self):
        self.mpHand = mp.solutions.hands
        self.mpDraw = mp.solutions.drawing_utils
        self.hands = self.mpHand.Hands(
            static_image_mode=False,
            max_num_hands=2,
            min_detection_confidence=0.5,
            min_tracking_confidence=0.5)
        self.lmDrawSpec = mp.solutions.drawing_utils.DrawingSpec(
            color=(0, 0, 255), thickness=-1, circle_radius=6)
        self.drawSpec = mp.solutions.drawing_utils.DrawingSpec(
            color=(0, 255, 0), thickness=2, circle_radius=2)

    def detect(self, frame, draw=True):
        img = np.zeros(frame.shape, np.uint8)
        img_RGB = cv.cvtColor(frame, cv.COLOR_BGR2RGB)
        results = self.hands.process(img_RGB)
        if results.multi_hand_landmarks:
            for handLms in results.multi_hand_landmarks:
                if draw:
                    self.mpDraw.draw_landmarks(
                        frame, handLms, self.mpHand.HAND_CONNECTIONS,
                        self.lmDrawSpec, self.drawSpec)
                self.mpDraw.draw_landmarks(
                    img, handLms, self.mpHand.HAND_CONNECTIONS,
                    self.lmDrawSpec, self.drawSpec)
        return frame, img


def frame_combine(frame, src):
    if len(frame.shape) == 3:
        frameH, frameW = frame.shape[:2]
        srcH, srcW = src.shape[:2]
        dst = np.zeros((max(frameH, srcH), frameW + srcW, 3), np.uint8)
        dst[:frameH, :frameW] = frame
        dst[:srcH, frameW:frameW + srcW] = src
    else:
        src = cv.cvtColor(src, cv.COLOR_BGR2GRAY)
        frameH, frameW = frame.shape[:2]
        srcH, srcW = src.shape[:2]
        dst = np.zeros((max(frameH, srcH), frameW + srcW), np.uint8)
        dst[:frameH, :frameW] = frame
        dst[:srcH, frameW:frameW + srcW] = src
    return dst


def main():
    detector = HandDetector()
    fps = FPS()

    cap = create_camera()
    if not cap.isOpened():
        raise RuntimeError("Failed to open camera")

    try:
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                continue

            frame, img = detector.detect(frame)
            fps.update_fps()
            fps.show_fps(frame)

            combined = frame_combine(frame, img)
            cv.imshow('HandDetector', combined)

            if cv.waitKey(1) & 0xFF == ord('q'):
                break
    finally:
        cap.release()
        cv.destroyAllWindows()


if __name__ == '__main__':
    main()

2、Introduction

MediaPipe is a data flow processing machine learning application development framework developed and open-sourced by Google. It is a graph-based data processing pipeline for building applications that use various forms of data sources such as video, audio, sensor data, and any time series data. MediaPipe is cross-platform, running on embedded platforms (Jetson Nano, etc.), mobile devices (iOS and Android), workstations, and servers, and supports mobile GPU acceleration. MediaPipe provides cross-platform, customizable ML solutions for real-time and streaming media. The core framework of MediaPipe is implemented in C++ and supports Java and Objective C. The main concepts of MediaPipe include Packet, Stream, Calculator, Graph, and Subgraph.

Features of MediaPipe:

End-to-end acceleration: Built-in fast ML inference and processing acceleration even on ordinary hardware.
Build once, deploy anywhere: Unified solutions for Android, iOS, desktop/cloud, web, and IoT.
Ready-to-use solutions: Cutting-edge ML solutions showcasing the full capabilities of the framework.
Free and open source: Framework and solutions under Apache 2.0, fully extensible and customizable.

3、MediaPipe Hands

MediaPipe Hands is a high-fidelity hand and finger tracking solution. It uses machine learning (ML) to infer 21 3D hand coordinates from a single frame.

After palm detection over the entire image, the hand landmark model performs precise keypoint localization of 21 3D hand joint coordinates within the detected hand region via regression, i.e., direct coordinate prediction. The model learns a consistent internal hand pose representation and is robust even to partially visible hands and self-occlusions.

To obtain ground truth data, approximately 30K real-world images were manually annotated with 21 3D coordinates as shown below (Z values obtained from image depth maps where available). To better cover possible hand poses and provide additional supervision on hand geometry, high-quality synthetic hand models were rendered on various backgrounds and mapped to corresponding 3D coordinates.

Tracking game

Posture detection