Gesture control

Note: This case needs to be run as root user, and administrator privileges are required to call the MIPI camera!

Switch to root user

su root

Password: yahboom

1. Program function description

After the program is started, stand in front of the robot's MIPI camera and control the robot's movement through different gestures.

You can view the recognition results in the PC browser.

2. Principle introduction

The gesture control function is to control the robot's movement through gestures, including left and right rotation and forward and backward translation. It consists of MIPI image acquisition, human detection and tracking, human hand key point detection, gesture recognition, gesture control strategy, image encoding and decoding, and WEB display. The process is as follows:

The hand key point detection algorithm subscribes to pictures and the hand frame detection results published by the human detection and tracking package. After inference, the algorithm msg containing the hand key point information is published. The websocket package is used to render and display the pictures published by the camera and the corresponding algorithm results on the PC browser.

The index of the key points of the hand is as follows:

3. Program reference path

After SSH connects to the car, the source code of this function is located at,

xxxxxxxxxx
/home/sunrise/yahboomcar_ws/src/yahboomcar_deeplearning/gesture_control/launch/hobot_gesture_control.launch.py

4. Program startup

After SSH connects to the car, the terminal runs,

​x
# Copy the configuration files needed to run the example from the TogetherROS installation path
cp -r /opt/tros/lib/mono2d_body_detection/config/ .
cp -r /opt/tros/lib/hand_lmk_detection/config/ .
cp -r /opt/tros/lib/hand_gesture_detection/config/ .

# Launch the launch file
ros2 launch gesture_control hobot_gesture_control.launch.py

The output log shows that the program runs successfully. The input and output frame rates of the algorithm during inference are 30fps, and the statistical frame rate is refreshed every second.

Open the browser on the PC (note that the computer and the Xuripai network must be in the same LAN), enter the URL: car IP:8000, for example, my car IP is 192.168.2.67, enter the URL in the browser on the virtual machine,

xxxxxxxxxx
192.168.2.67:8000

Click to enter the Web display terminal, the display screen is as follows,

The following gestures can be used to control the movement of the car: