Gesture Recognition Stacking Blocks

If using a depth camera (DABAI_DCW2), make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The gesture recognition stacking blocks function is built upon gesture recognition grasping, modifying the placement method to stacking. Gesture 5 is changed to knocking over the blocks and clearing the record.

For all functions involving grasping of square blocks, please use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo No. 6 stall and damage.

Note: Before starting the program, please follow the [Assembly Tutorial] -> [Install the Map] tutorial to correctly install the map before operation.

1. Startup and Operation

Experiment Setup

Place the blocks at positions numbered 1-4.

Start the Program

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Raspberry Pi code path:

xxxxxxxxxx
/home/yahboom/dofbot_ultra_ws/src/dofbot_gesture/scripts/Gesture_Stacking.ipynb

RDK X5 code path:

xxxxxxxxxx
/home/sunrise/dofbot_ultra_ws/src/2d/dofbot_gesture/scripts/Gesture_Stacking.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

Experiment Effect

After the program runs, scroll to the bottom. The JupyterLab page will display the camera view and related button functions.

If a preset gesture is recognized, the arm will perform the corresponding action. When the arm recognizes a number for the first time, it goes to the corresponding position to grab the block and place it on the first layer. The second time it recognizes a gesture number, it places on the second layer, and so on up to the fourth layer. Each number can only be recognized once; repeated recognition will not trigger any action. When number five is recognized, the arm knocks over the blocks, clears the record, and restarts.

The gesture and action mapping for this example:

As shown below:

To exit the program, click the [Exit] button.

2. Technical Overview

This function shares the same principle as gesture recognition grasping, with differences in the placement method and fist function:

1. Gesture Detection: Use Mediapipe to recognize gesture numbers (1-5)

2. Gesture Mapping: Gestures 1-4 correspond to the 4 grab positions on the map

3. Stacking Placement: Each grabbed block is stacked at the same position with increasing height per layer

4. Deduplication Mechanism: Each gesture number can only trigger one grab to avoid repeated operations

5. Knock Over Reset: Gesture 5 triggers the block knock-over action and clears the recognition record, allowing a fresh start

3. Core Code Analysis

• Import header files

xxxxxxxxxx
import cv2 as cv
import threading
import time
import ipywidgets as widgets
from IPython.display import display
from gesture_stacking import Gesture_Stacking
from dofbot_utils.fps import FPS
from dofbot_utils.robot_controller import Robot_Controller

• Create instances and initialize parameters

​x
robot = Robot_Controller()
robot.move_init_pose()
fps = FPS()

gesture = Gesture_Stacking()
model = 'General'

• Main program

xxxxxxxxxx
def camera():
    global model, gesture
    from dofbot_utils.camera import create_camera
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            fps.update_fps()
            gesture.process(img)
            if model == 'Exit':
                capture.release()
                del gesture
                break
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
            time.sleep(0.002)
        except Exception as e:
            print("program end")
            print(e)
            capture.release()

• Start display

display(controls_box,output)
threading.Thread(target=camera, ).start()