Gesture Recognition Grasping Blocks

If using a depth camera (DABAI_DCW2), make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

For all functions involving grasping of square blocks, please use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo No. 6 stall and damage.

The gesture recognition grasping blocks function combines Mediapipe gesture recognition, the robotic arm, and the map. It recognizes five gestures. Gestures 1-4 correspond to grab positions 1-4 on the map and place the block on the cross. Gesture 5 performs the gripper rotation action.

Note: Before starting the program, please follow the [Assembly Tutorial] -> [Install the Map] tutorial to correctly install the map before operation.

1. Startup and Operation

Experiment Setup

Place the blocks at positions numbered 1-4.

Start the Program

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Raspberry Pi code path:

xxxxxxxxxx
/home/yahboom/dofbot_ultra_ws/src/dofbot_gesture/scripts/Gesture_Grasp.ipynb

RDK X5 code path:

xxxxxxxxxx
/home/sunrise/dofbot_ultra_ws/src/2d/dofbot_gesture/scripts/Gesture_Grasp.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

Experiment Effect

After the program runs, scroll to the bottom. The JupyterLab page will display the camera view and related button functions.

If a preset gesture is recognized, the arm will perform the corresponding action. When gesture 1 is recognized, the arm goes to position 1 to grab the block and place it on the center cross. After placement, remove the block from the cross to avoid conflicts for the next placement. When gestures 2, 3, or 4 are recognized, the arm goes to the corresponding position to grab the block and place it on the cross. When gesture 5 is recognized, the arm performs the gripper rotation action.

The gesture and action mapping for this example:

As shown below:

If there is a block on the center cross, please remove it before starting gesture recognition.

To exit the program, click the [Exit] button.

2. Technical Overview

The core workflow of this function:

1. Gesture Detection: Use Google Mediapipe gesture recognition library to detect hand keypoints in real-time via the camera and determine the number of extended fingers (1-5)

2. Gesture Mapping: Map the detected gesture number to the corresponding map position (gestures 1-4) or special action (gesture 5)

3. Position Planning: Determine the target grab position of the arm based on the map number corresponding to the gesture

4. Grab Execution: The arm moves to the target position, performs the grab action, and places the block on the cross

5. Special Action: Gesture 5 triggers the gripper rotation action, demonstrating the arm's flexibility

3. Core Code Analysis

• Import header files

xxxxxxxxxx
import cv2 as cv
import threading
import time
import ipywidgets as widgets
from IPython.display import display
from gesture_grasp import Gesture_Grasp
from dofbot_utils.fps import FPS
from dofbot_utils.robot_controller import Robot_Controller

• Create instances and initialize parameters

​x
robot = Robot_Controller()
robot.move_init_pose()
fps = FPS()

gesture = Gesture_Grasp()
model = 'General'

• Create widgets

xxxxxxxxxx
def exit_button_Callback(value):
    global model
    model = 'Exit'
    with output:
        print(model)

exit_button.on_click(exit_button_Callback)

• Main program

xxxxxxxxxx
def camera():
    global model, gesture
    from dofbot_utils.camera import create_camera
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            fps.update_fps()
            gesture.process(img)
            if model == 'Exit':
                capture.release()
                del gesture
                break
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
            time.sleep(0.002)
        except Exception as e:
            print("program end")
            print(e)
            capture.release()

• Startup

display(controls_box,output)
threading.Thread(target=camera, ).start()