Mediapipe development

1. Introduction

MediaPipe is a data stream processing machine learning application development framework developed and open sourced by Google. It is a graph-based data processing pipeline for building applications that use a variety of data sources, such as video, audio, sensor data, and any time series data. MediaPipe is cross-platform and can run on embedded platforms (Raspberry Pi, etc.), mobile devices (iOS and Android), workstations, and servers, and supports mobile GPU acceleration. MediaPipe provides cross-platform, customizable ML solutions for real-time and streaming media.

The core framework of MediaPipe is implemented in C++ and provides support for languages such as Java and Objective C. The main concepts of MediaPipe include packets, streams, calculators, graphs, and subgraphs.

Features of MediaPipe:

• End-to-end acceleration: built-in fast ML inference and processing can be accelerated even on ordinary hardware.
• Build once, deploy anywhere: unified solution for Android, iOS, desktop/cloud, web, and IoT.
• Ready-to-use solutions: cutting-edge ML solutions that demonstrate the full capabilities of the framework.
• Free and open source: Framework and solution under Apache2.0, fully extensible and customizable.

Deep learning solution in MediaPipe

Face Detection	Face Mesh	Iris	Hands	Pose	Holistic
Hair Segmentation	Object Detection	Box Tracking	Instant Motion Tracking	Objectron	KNIFT

2. Use

You need to enter Docker to run the case. The ROS1 environment is located in the Docker image.

sh ~/ros_melodic.sh

2.1. Hand detection

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
roscore
rosrun yahboomcar_mediapipe 01_HandDetector.py

2.2, Posture detection

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
roscore
rosrun yahboomcar_mediapipe 02_PoseDetector.py

2.3. Overall detection

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
roscore
rosrun yahboomcar_mediapipe 03_Holistic.py

2.4, Face Detection

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
roscore
rosrun yahboomcar_mediapipe 04_FaceMesh.py

2.5, Face Recognition

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
roscore
rosrun yahboomcar_mediapipe 05_FaceEyeDetection.py

2.6, Face effects

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
cd ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts
python3 06_FaceLandmarks.py

2.7, 3D object recognition

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
cd ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts
python3 08_Objectron.py

Click the camera preview screen and press the f key to switch the recognized model.

2.8, Brush

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
cd ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts
python3 09_VirtualPaint.py

2.9, Finger control

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
cd ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts
python3 10_HandCtrl.py

2.10, Gesture Recognition

Source code location: ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts

xxxxxxxxxx
cd ~/yahboomcar_ws/src/yahboomcar_mediapipe/scripts
python3 11_GestureRecognition.py