11. Mediapipe gesture recognition to control car movement

11.1. Use

After the function is turned on, the camera captures images and recognizes gestures to control the movement of the car. Multi-machine communication needs to be configured

Note: [R2] on the remote control handle has the [pause/start] function for this gameplay.

jetson motherboard/Raspberry Pi 4B

Start command (robot side)

roslaunch arm_mediapipe mediaArm.launch

Start command (virtual machine)

xxxxxxxxxx
rosrun arm_mediapipe HandCtrl.py

Raspberry Pi 5

Before running, please confirm that the large program has been permanently closed

Enter docker

Note: If there is a terminal that automatically starts docker, or there is a docker terminal that has been opened, you can directly enter the docker terminal to run the command, and there is no need to manually start docker

Start docker manually

xxxxxxxxxx
./run_docker.sh

Start command (robot side)

xxxxxxxxxx
roslaunch arm_mediapipe mediaArm.launch

Start command (virtual machine)

xxxxxxxxxx
rosrun arm_mediapipe HandCtrl.py

After the program is started, press the R2 button on the handle to turn on the function, and then put your hand in front of the camera. The shape of the finger will be drawn on the screen. After the program recognizes the gesture, it will send the speed to the chassis to control the movement of the car.

Gesture number 1: the car moves left

Gesture number 2: the car moves right

Gesture number 3: the camera rotates left

Gesture number 4: the camera rotates right

Gesture number 5: the car moves forward

Gesture fist: the car moves back

Gesture rock (the index finger and the little finger are straight, the others are bent): the buzzer sounds

MediaPipe Hands infers 3D coordinates of 21 hand-valued joints from one frame

11.2. Core code analysis HandCtrl.py

Code reference path：~/transbot_ws/src/arm_mediapipe/scripts

• Import critical libraries

  xxxxxxxxxx
  from media_library import * #This library contains functions to detect hands, get gestures, etc.
  

• Get finger data

  xxxxxxxxxx
  frame, lmList, _ = self.hand_detector.findHands(frame)
  fingers = self.hand_detector.fingersUp(lmList)
  sum(fingers)
  fingers[]
  

  It can be seen that the hand is detected first, the value of lmList is obtained, and then the fingersUp function is passed in. The fingersUp function is used to detect which fingers are straight. The value of the straight finger is 1. The specific code here can also be seen in the media_library, py function, which has a detailed explanation, which is actually to judge the finger joints xy value to judge when it is straightened. The sum(fingers) function is used to calculate the number of straight fingers, fingers[] can be used to enumerate fingers, such as the index finger, we use fingers[1] to represent.

• Post speed to chassis

  xxxxxxxxxx
  self.media_ros.pub_vel(x,y,z)   #This function is also in media_library,py
  

11.3.Flowchart