Welcome to JetCobot repository

Gesture Recognition

1. Introduction

The gesture recognition gameplay is to detect the position information of the finger joints through the MediaPipe framework function, first divide the number of fingers in the open state according to the characteristics of gestures 1-8, and then calculate the 1-8 gestures.

2. Core content analysis

Import related libraries.

​x
import threading
import cv2 
import time
import math
from time import sleep
import ipywidgets.widgets as widgets

from gesture_action import handDetector

Initialize the USB camera. The default device is /dev/video0. You can modify the device number of the USB camera according to the device number found in the system.

xxxxxxxxxx
g_camera = cv2.VideoCapture(0)
g_camera.set(3, 640)
g_camera.set(4, 480)
g_camera.set(5, 30)  #设置帧率
g_camera.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M', 'J', 'P', 'G'))
g_camera.set(cv2.CAP_PROP_BRIGHTNESS, 40) #设置亮度 -64 - 64  0.0
g_camera.set(cv2.CAP_PROP_CONTRAST, 50) #设置对比度 -64 - 64  2.0
g_camera.set(cv2.CAP_PROP_EXPOSURE, 156) #设置曝光值 1.0 - 5000  156.0

Initialize the gesture detection object. detectorCon represents the detection value, which can be modified according to the actual effect. The recommended range is 0.5-1.0.

xxxxxxxxxx
hand_detector = handDetector(detectorCon=0.75)
image_original = widgets.Image(format='jpeg', width=640, height=480)
image_result = widgets.Image(format='jpeg', width=640, height=480)
image_widget = widgets.HBox([image_original, image_result])

Read the camera image and transmit it to the findHands function to calculate and output the detection result.

xxxxxxxxxx
display(image_widget)   
try:
    while True:
        ret, frame = g_camera.read()
        frame, img = hand_detector.findHands(frame, draw=False)
        if len(hand_detector.lmList) != 0:
            finger_number = hand_detector.get_gesture()
            cv2.rectangle(frame, (0, 430), (230, 480), (0, 255, 0), cv2.FILLED)
            cv2.putText(frame, str(finger_number), (10, 470), cv2.FONT_HERSHEY_PLAIN, 2, (255, 0, 0), 2)
            print("Number:", finger_number)
        image_original.value = bgr8_to_jpeg(frame)
        image_result.value = bgr8_to_jpeg(img)
except:
    print(" Program closed! ")
    pass

3. Program operation and operation

Code path:

xxxxxxxxxx
~/jetcobot_ws/src/jetcobot_ai_basic/scripts/2.Gesture_recognition.ipynb

Open the jupyterlab webpage, run all the program blocks, and then pull to the bottom of the webpage.

At this time, the camera screen is displayed. Please put your hand into the camera detection range and make gestures 1-8. After the robot recognizes the gesture features, it will print out the recognized gestures.

The corresponding actions of the gesture features are shown in the figure below:

If you need to exit, please press the stop button on the jupyterlab toolbar.