12. Gesture control

1. Program function description

After the program starts, standing in front of the MIPI camera of the car, the robot can be controlled through different gestures.

You can view the identification result in the browser on the PC.

2. Introduction to principle

The gesture control function is to control the movement of the robot car through gestures, including left and right rotation and forward and backward translation. It is composed of MIPI image acquisition, human body detection and tracking, human key point detection, gesture recognition, gesture control strategy, image codec and WEB display terminal. The process is as follows:

Manual key point detection algorithm Subscribe pictures and human body detection and track the manual frame detection results released by the package. After reasoning, the algorithm msg containing manual key point information is released. Through websocket package, the pictures released by the camera and the corresponding algorithm results are rendered and displayed on the browser on the PC side.

The key point index of the human hand is as follows:

3. Program reference path

After SSH connection car, the function source code is located in,

xxxxxxxxxx
/userdata/yahboomcar_ws/src/yahboomcar_deeplearning/gesture_control/launch/hobot_gesture_control.launch.py

4. The program starts

After SSH connects to the car, the terminal runs,

xxxxxxxxxx
# 从TogetherROS的安装路径中拷贝出运行示例需要的配置文件
# Copy the configuration files needed to run the example from TogetherROS
cp -r /opt/tros/lib/mono2d_body_detection/config/ .
cp -r /opt/tros/lib/hand_lmk_detection/config/ .
cp -r /opt/tros/lib/hand_gesture_detection/config/ .
​
#启动launch文件
# Start the launch file
ros2 launch gesture_control hobot_gesture_control.launch.py

The output log shows that the program runs successfully, the algorithm input and output frame rate is 30fps when reasoning, and the statistical frame rate is refreshed once every second.

Open the browser on the PC side (note that the computer and the Rising Sun network must be in the same LAN), enter the URL: car IP:8000, for example, my car IP is 192.168.2.67, enter the URL in the browser on the virtual machine side,

xxxxxxxxxx
192.168.2.67:8000

Click to enter the Web display terminal, the display screen is as follows:

The car movement can be controlled by the following gestures: