Welcome to JetCobot repository

Gesture recognition

1. Function Introduction

The gesture recognition gameplay is to detect the position information of the finger joints through the MediaPipe framework function, and first divide the number of fingers in the open state according to the characteristics of gestures 1-8, and then calculate the 1-8 gestures.

2. About code

Import related libraries.

​x
import threading
import cv2 
import time
import math
from time import sleep
import ipywidgets.widgets as widgets

from gesture_action import handDetector

Initialize the USB camera. The default device is /dev/video0.

You can modify the device number of the USB camera according to the device number found in the system.

xxxxxxxxxx
g_camera = cv2.VideoCapture(0)
g_camera.set(3, 640)
g_camera.set(4, 480)
g_camera.set(5, 30)  #Setting the frame rate
g_camera.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M', 'J', 'P', 'G'))
g_camera.set(cv2.CAP_PROP_BRIGHTNESS, 40) #Setting Brightness -64 - 64  0.0
g_camera.set(cv2.CAP_PROP_CONTRAST, 50) #Setting the contrast -64 - 64  2.0
g_camera.set(cv2.CAP_PROP_EXPOSURE, 156) #Setting the exposure value 1.0 - 5000  156.0

Initialize the gesture detection object. detectorCon represents the detection value, which can be modified according to the actual effect.

The recommended range is 0.5-1.0.

xxxxxxxxxx
hand_detector = handDetector(detectorCon=0.75)
image_original = widgets.Image(format='jpeg', width=640, height=480)
image_result = widgets.Image(format='jpeg', width=640, height=480)
image_widget = widgets.HBox([image_original, image_result])

Read the camera image and transmit it to the findHands function to calculate and output the detection result.

xxxxxxxxxx
display(image_widget)   
try:
    while True:
        ret, frame = g_camera.read()
        frame, img = hand_detector.findHands(frame, draw=False)
        if len(hand_detector.lmList) != 0:
            finger_number = hand_detector.get_gesture()
            cv2.rectangle(frame, (0, 430), (230, 480), (0, 255, 0), cv2.FILLED)
            cv2.putText(frame, str(finger_number), (10, 470), cv2.FONT_HERSHEY_PLAIN, 2, (255, 0, 0), 2)
            print("Number:", finger_number)
        image_original.value = bgr8_to_jpeg(frame)
        image_result.value = bgr8_to_jpeg(img)
except:
    print(" Program closed! ")
    pass

3. Run program

Code path:

xxxxxxxxxx
~/jetcobot_ws/src/jetcobot_ai_basic/scripts/2.Gesture_recognition.ipynb

Open the jupyterlab webpage, run all the program blocks, and then pull to the bottom of the webpage.

The camera screen is displayed at this time. Please put your hand into the camera detection range and make gestures 1-8. After the robot recognizes the gesture features, it will print out the recognized gestures.

The corresponding actions of the gesture features are shown in the figure below.

If you need to exit, please click the stop button on the Jupyterlab toolbar.