Welcome to DOFBOT-Ultra repository

2. Posture detection

Posture Detection

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Use MediaPipe Pose to detect human body posture in real-time, identifying 33 body keypoints (including face, torso, and limbs), and drawing the pose skeleton on both the original image and a black background.

1、Startup

1.1、Startup

Enter the following command to start the program


ros2 run dofbot_mediapipe 02_PoseDetector.py

Press the q key in the image window or Ctrl+C in the terminal to exit the program.

1.2、Source Code

Raspberry Pi source path:


xxxxxxxxxx
~/dofbot_ultra_ws/src/dofbot_mediapipe/scripts/02_PoseDetector.py

RDK X5 source path:


xxxxxxxxxx
~/dofbot_ultra_ws/src/2d/dofbot_mediapipe/scripts/02_PoseDetector.py


x
#!/usr/bin/env python3
# encoding: utf-8
import cv2 as cv
import numpy as np
import mediapipe as mp
from dofbot_utils.camera import create_camera
from dofbot_utils.fps import FPS


class PoseDetector:
    def __init__(self):
        self.mpPose = mp.solutions.pose
        self.mpDraw = mp.solutions.drawing_utils
        self.pose = self.mpPose.Pose(
            static_image_mode=False,
            smooth_landmarks=True,
            min_detection_confidence=0.5,
            min_tracking_confidence=0.5)
        self.lmDrawSpec = mp.solutions.drawing_utils.DrawingSpec(
            color=(0, 0, 255), thickness=-1, circle_radius=6)
        self.drawSpec = mp.solutions.drawing_utils.DrawingSpec(
            color=(0, 255, 0), thickness=2, circle_radius=2)

    def detect(self, frame, draw=True):
        img = np.zeros(frame.shape, np.uint8)
        img_RGB = cv.cvtColor(frame, cv.COLOR_BGR2RGB)
        results = self.pose.process(img_RGB)
        if results.pose_landmarks:
            if draw:
                self.mpDraw.draw_landmarks(
                    frame, results.pose_landmarks,
                    self.mpPose.POSE_CONNECTIONS,
                    self.lmDrawSpec, self.drawSpec)
            self.mpDraw.draw_landmarks(
                img, results.pose_landmarks,
                self.mpPose.POSE_CONNECTIONS,
                self.lmDrawSpec, self.drawSpec)
        return frame, img


def frame_combine(frame, src):
    if len(frame.shape) == 3:
        frameH, frameW = frame.shape[:2]
        srcH, srcW = src.shape[:2]
        dst = np.zeros((max(frameH, srcH), frameW + srcW, 3), np.uint8)
        dst[:frameH, :frameW] = frame
        dst[:srcH, frameW:frameW + srcW] = src
    else:
        src = cv.cvtColor(src, cv.COLOR_BGR2GRAY)
        frameH, frameW = frame.shape[:2]
        srcH, srcW = src.shape[:2]
        dst = np.zeros((max(frameH, srcH), frameW + srcW), np.uint8)
        dst[:frameH, :frameW] = frame
        dst[:srcH, frameW:frameW + srcW] = src
    return dst


def main():
    detector = PoseDetector()
    fps = FPS()

    cap = create_camera()
    if not cap.isOpened():
        raise RuntimeError("Failed to open camera")

    try:
        while cap.isOpened():
            ret, frame = cap.read()
            if not ret:
                continue

            frame, img = detector.detect(frame)
            fps.update_fps()
            fps.show_fps(frame)

            combined = frame_combine(frame, img)
            cv.imshow('PoseDetector', combined)

            if cv.waitKey(1) & 0xFF == ord('q'):
                break
    finally:
        cap.release()
        cv.destroyAllWindows()


if __name__ == '__main__':
    main()

2、Introduction

MediaPipe is a data flow processing machine learning application development framework developed and open-sourced by Google. It is a graph-based data processing pipeline for building applications that use various forms of data sources such as video, audio, sensor data, and any time series data. MediaPipe is cross-platform, running on embedded platforms (Jetson Nano, etc.), mobile devices (iOS and Android), workstations, and servers, and supports mobile GPU acceleration. MediaPipe provides cross-platform, customizable ML solutions for real-time and streaming media.

The core framework of MediaPipe is implemented in C++ and supports Java and Objective C. The main concepts of MediaPipe include Packet, Stream, Calculator, Graph, and Subgraph.

Features of MediaPipe:

End-to-end acceleration: Built-in fast ML inference and processing acceleration even on ordinary hardware.
Build once, deploy anywhere: Unified solutions for Android, iOS, desktop/cloud, web, and IoT.
Ready-to-use solutions: Cutting-edge ML solutions showcasing the full capabilities of the framework.
Free and open source: Framework and solutions under Apache 2.0, fully extensible and customizable.

3、MediaPipe Pose

MediaPipe Pose is an ML solution for high-fidelity body pose tracking, utilizing BlazePose research to infer 33 3D coordinates and a full-body background segmentation mask from RGB video frames, which also powers the ML Kit pose detection API.

The landmark model in MediaPipe Pose predicts the positions of 33 pose coordinates (see figure below).

Hand detection

Overall detection