Welcome to DOFBOT-Ultra repository
11. Gesture control robotic arm action group
Gesture Control Robotic Arm Action Group
If using a depth camera (DABAI_DCW2), make sure to execute
ros2 run dofbot_ultra_driver calibrate_6pointfor a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.
Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!
Recognize gestures via the camera. When the same gesture is stably recognized for 10 consecutive frames, the corresponding arm action group is triggered. Supports 5 gestures, each corresponding to a different arm performance action.
1、Gesture and Action Mapping Table
| Gesture | Hand Shape Description | Arm Action |
|---|---|---|
| One (number 1) | Only index finger extended | Arm extends, resets after 1.5 seconds |
| Two (number 2) | Index and middle fingers extended (peace sign) | Arm bends upward, then gripper rapidly opens and closes 5 times (waving effect), resets after completion |
| Three (number 3) | Index, middle, and ring fingers extended | Arm applauds 3 times, then resets |
| Four (number 4) | Four fingers extended (thumb tucked) | Performs nodding action, resets after completion |
| Five (number 5) | All five fingers spread open | Joint 5 rotates and swings 5 times (head-shaking effect), resets after completion |
Tip: Hold the same gesture in front of the camera for about 1-2 seconds (10 frames) to trigger the arm action. The arm will not respond to new gestures during action execution and will automatically return to the observation pose after completion.
2、Startup
Open the desktop terminal and enter the following command to start the program
xxxxxxxxxxros2 run dofbot_mediapipe 11_GestureAction.py
Press the q key in the image window or Ctrl+C in the terminal to exit the program.
2.1、Source Code
Raspberry Pi code path:
xxxxxxxxxx~/dofbot_ultra_ws/src/dofbot_mediapipe/scripts/11_GestureAction.py
RDK X5 code path:
xxxxxxxxxx~/dofbot_ultra_ws/src/2d/dofbot_mediapipe/scripts/11_GestureAction.py
3、Principle Description
The program detects hand keypoints and recognizes gesture names through GestureRecognition.handDetector, using a counter to track consecutive frames recognizing the same gesture. When the count reaches the threshold (10 frames), the corresponding action group is executed in a separate thread to avoid blocking camera capture. The action groups control the arm movement through predefined joint angle arrays (P_ACTION_1 ~ P_ACTION_5) in robot_controller.Robot_Controller.
x#!/usr/bin/env python3# encoding: utf-8import cv2 as cvimport timeimport threadingfrom dofbot_utils.camera import create_camerafrom dofbot_utils.fps import FPSfrom dofbot_utils.robot_controller import Robot_Controllerfrom dofbot_utils.GestureRecognition import handDetectorfrom Arm_Lib import Arm_Device
class GestureAction: def __init__(self): self.hand_detector = handDetector(detectorCon=0.75) self.fps = FPS() self.arm = Arm_Device() self.robot = Robot_Controller() self.move_state = False
self.gesture_counts = { 'One': 0, 'Two': 0, 'Three': 0, 'Four': 0, 'Five': 0}
self.arm.Arm_serial_servo_write6_array(self.robot.P_LOOK_AT, 1000)
def reset_counts(self, keep): for k in self.gesture_counts: if k != keep: self.gesture_counts[k] = 0
def process(self, frame): frame, lmList = self.hand_detector.findHands(frame, draw=False) if len(lmList) != 0: gesture = self.hand_detector.get_gesture() if gesture in self.gesture_counts: cv.putText(frame, gesture, (250, 30), cv.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 1) self.gesture_counts[gesture] += 1 self.reset_counts(gesture) if self.gesture_counts[gesture] >= 10 and not self.move_state: self.move_state = True self.gesture_counts[gesture] = 0 task = threading.Thread( target=self.arm_ctrl, args=(gesture,), daemon=True) task.start() self.fps.update_fps() self.fps.show_fps(frame) return frame
def arm_ctrl(self, gesture): if gesture == 'One': self.arm.Arm_serial_servo_write6_array( self.robot.P_ACTION_1, 1000) time.sleep(1.5) self.arm.Arm_serial_servo_write6_array( self.robot.P_LOOK_AT, 1000) time.sleep(1) elif gesture == 'Two': self.arm.Arm_serial_servo_write6_array( self.robot.P_ACTION_2, 1000) time.sleep(1.5) for _ in range(5): self.arm.Arm_serial_servo_write(6, 180, 100) time.sleep(0.15) self.arm.Arm_serial_servo_write(6, 30, 100) time.sleep(0.15) self.arm.Arm_serial_servo_write6_array( self.robot.P_LOOK_AT, 1000) time.sleep(1) elif gesture == 'Three': for _ in range(3): self.arm.Arm_serial_servo_write6_array( self.robot.P_ACTION_3, 1200) time.sleep(1.2) self.arm.Arm_serial_servo_write6_array( self.robot.P_LOOK_AT, 1000) time.sleep(1) elif gesture == 'Four': self.arm.Arm_serial_servo_write6_array( self.robot.P_ACTION_4, 1500) time.sleep(1.4) for _ in range(3): self.arm.Arm_serial_servo_write(4, -15, 300) time.sleep(0.4) self.arm.Arm_serial_servo_write(4, 20, 300) time.sleep(0.4) self.arm.Arm_serial_servo_write6_array( self.robot.P_LOOK_AT, 1000) time.sleep(1) elif gesture == 'Five': for _ in range(5): self.arm.Arm_serial_servo_write(5, 60, 300) time.sleep(0.4) self.arm.Arm_serial_servo_write(5, 120, 300) time.sleep(0.4) self.arm.Arm_serial_servo_write(5, 90, 300) time.sleep(0.4) self.arm.Arm_serial_servo_write6_array( self.robot.P_LOOK_AT, 1000) time.sleep(1) self.move_state = False
def main(): gesture_action = GestureAction()
cap = create_camera() if not cap.isOpened(): raise RuntimeError("Failed to open camera")
try: while cap.isOpened(): ret, frame = cap.read() if not ret: continue frame = gesture_action.process(frame) cv.imshow('GestureAction', frame) if cv.waitKey(1) & 0xFF == ord('q'): break finally: cap.release() cv.destroyAllWindows()
if __name__ == '__main__': main()