Welcome to DOFBOT-Ultra repository

Gesture Recognition

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Detect hand keypoints in real-time via the camera, recognize gestures based on the extension/bending state of each finger and fingertip distances, and display the recognition result in the lower left corner of the screen. Designed for the right hand, supporting a total of 14 types of gestures.

1、Recognizable Gestures Overview

2、Startup

2.1、Startup

• Enter the following command to start the program

ros2 run dofbot_mediapipe 10_GestureRecognition.py

Press the q key in the image window or Ctrl+C in the terminal to exit the program.

2.2、Source Code

Raspberry Pi source path:

xxxxxxxxxx
~/dofbot_ultra_ws/src/dofbot_mediapipe/scripts/10_GestureRecognition.py

RDK X5 source path:

xxxxxxxxxx
~/dofbot_ultra_ws/src/2d/dofbot_mediapipe/scripts/10_GestureRecognition.py

​x
#!/usr/bin/env python3
# encoding: utf-8
import cv2 as cv
from dofbot_utils.camera import create_camera
from dofbot_utils.fps import FPS
from dofbot_utils.GestureRecognition import handDetector


def main():
    detector = handDetector(detectorCon=0.75)
    fps = FPS()

    cap = create_camera()
    if not cap.isOpened():
        raise RuntimeError("Failed to open camera")

    try:
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                continue

            frame, lmList = detector.findHands(frame, draw=False)
            if len(lmList) != 0:
                gesture = detector.get_gesture()
                cv.rectangle(frame, (0, 430), (230, 480),
                             (0, 255, 0), cv.FILLED)
                cv.putText(frame, str(gesture), (10, 470),
                           cv.FONT_HERSHEY_PLAIN, 2, (255, 0, 0), 2)

            fps.update_fps()
            fps.show_fps(frame)
            combined = detector.frame_combine(frame, detector.draw_img)
            cv.imshow('GestureRecognition', combined)

            if cv.waitKey(1) & 0xFF == ord('q'):
                break
    finally:
        cap.release()
        cv.destroyAllWindows()


if __name__ == '__main__':
    main()

2、Introduction

MediaPipe is a data flow processing machine learning application development framework developed and open-sourced by Google. It is a graph-based data processing pipeline for building applications that use various forms of data sources such as video, audio, sensor data, and any time series data.

MediaPipe is cross-platform, running on embedded platforms (Jetson Nano, etc.), mobile devices (iOS and Android), workstations, and servers, and supports mobile GPU acceleration. MediaPipe provides cross-platform, customizable ML solutions for real-time and streaming media.

The core framework of MediaPipe is implemented in C++ and supports Java and Objective C. The main concepts of MediaPipe include Packet, Stream, Calculator, Graph, and Subgraph.

Features of MediaPipe:

• End-to-end acceleration: Built-in fast ML inference and processing acceleration even on ordinary hardware.

• Build once, deploy anywhere: Unified solutions for Android, iOS, desktop/cloud, web, and IoT.

• Ready-to-use solutions: Cutting-edge ML solutions showcasing the full capabilities of the framework.

• Free and open source: Framework and solutions under Apache 2.0, fully extensible and customizable.

4、Recognition Principle

A gesture recognition algorithm designed for the right hand. First, MediaPipe Hands detects 21 hand keypoints, then:

1. Finger Extension Determination: Determine whether each finger is extended based on the relative position (y-coordinate comparison) of the fingertip and finger base. The thumb uses joint angle determination

2. Feature Combination: Match predefined gestures based on the extension/bend combination of the 5 fingers

3. Special Gestures: Gestures like OK, Heart_single also detect distances between fingertips. Thumb_up/Thumb_down use the y-coordinate height relationship between the thumb and other four fingers for determination