yahboom
HomeLearn
Solution
Blog
Online Store
About Us
EN/中文
DOFBOT-JetsonNANO📖
00.Must Read Before Use!!
  • 00.Must Read Before Use!!
01.First Trial
  • 1.First Trial
  • 2.Analysis of common problems
  • 3.Calibration offset
  • 4.Precautions for DIY Robotic Arm
  • 5.Jetson Nano Version – Must Read
  • 6.Robotic arm gripping anomaly solutions
02.Assemble Course
  • 1.Installation video
  • 2.Calibrate the Robotic Arm
  • 3.Install the map
03.Remote control course
  • 1.Mobile APP control
  • 2.USB handle control
  • 3.PC web page programming
04.Preparation
  • 1.Manage APP control service
  • 2.Manage JupyterLab service
  • 3.Manage OLED service
  • 4.Modifying ROS_DOMAIN_ID for Multi-Machine
  • 5.Jetpack old system upgraded to SUPER system
05.Basic control course
  • 1.Control RGB light
  • 2.Control buzzer
  • 3.Control single servo
  • 4.Read servo angle
  • 5.Control all servo
  • 6.DOFBOT up down left right
  • 7.DOFBOT dancing
  • 8.DOFBOT memory action
  • 9.DOFBOT grab block
  • 10.Porter
  • 11.Stacked Arhat
  • 12.Synchronous action
  • 13.USB camera display
06.AI vision basic course
  • 1.Gesture Recognition
  • 2.Color threshold adjustment&Color calibration
  • 3.Color recognition
  • 4.Face recognition
  • 5.YOLOv11 garbage recognition
  • 6.YOLOv11 object recognition
07.2D Visual Tracking Course
  • 1.PID algorithm basics
  • 2.Color block positioning
  • 3.Color block tracking
  • 4.Face positioning
  • 5.Face tracking
  • 6.Gesture recognition fixed action
08.2D Visual Grasping Course
  • 1.Gesture recognition stack blocks
  • 2.Color recognition grab blocks
  • 3.Catch game
  • 4.Color sorting and stacking
  • 5.Garbage sorting
  • 6.Color interaction
09.Mediapipe course
  • 1.Hand detection
  • 02.Posture detection
  • 3.Overall detection
  • 4.Facial Detection
  • 5.Face Detection
  • 6.Face effects
  • 7.3D object recognition
  • 8.Paintbrush
  • 9.Finger control
  • 10.Gesture Recognition
  • 11.Gesture control robotic arm action group
  • 12.Arm posture control robotic arm
  • 13.Palm Targeting
  • 14.Robotic arm tracks the palm
  • 15.Fingertip trajectory recognition
  • 16.Fingertip trajectory control robotic arm
  • 17.Gesture Grasp and Release Objects
  • 18.Finger Control Robot Arm
10.AI Large Model Voice Module (Must Read)new
  • 1. Introduction to AI Large Model Voice Module
  • 2. Flash the firmware (Must read)
  • 3. Modify the wake-up word and command word
11.2D Voice Control Coursenew
  • 01.Simple Voice Control
  • 02.Smart Voice Control
  • 03.Voice Action Control
  • 04.Memory Learning Mode
  • 05.Color Recognition with Voice Broadcast
  • 06.Color Recognition and Grasping with Voice Broadcast
  • 07.Color Recognition and Sorting with Voice Broadcast
  • 08.Voice Recognition Target Tracking
  • 09.Voice Control Stacking
  • 10.Voice Control Color Sorting and Stacking
  • 11.Garbage Recognition with Voice Broadcast
  • 12.Garbage Sorting with Voice Broadcast
12.Robotic Arm Control and Inverse Kinematics
  • 1. Robotic arm ROS control
  • 2. Inverse solution
13.3D Sorting and Grasping Coursenew
  • 1.AprilTag ID sorting
  • 2.MediaPipe Gesture-AprilTag ID sorting
  • 3.MediaPipe Gesture-AprilTag distance sorting
  • 4.Block color sorting
  • 5.Block different shape sorting
  • 6.Yolov11 garbage sorting
14.3D Tracking Coursenew
  • 1.AprilTag Tracking
  • 2.Color Block Tracking
  • 3.KCF Object Tracking
15.3D Voice Control for Sorting & Tracking Coursenew
  • 1.Voice Control AprilTag ID Sorting
  • 2.Voice Control Color Block Sorting
  • 3.Voice Control Yolov11 Garbage Recognition Sorting
  • 4.Voice Control AprilTag Tracking
  • 5.Voice Control Color Block Tracking
  • 6.Voice Control Object Tracking
16.AI Model Basicsnew
  • 1.AI large model types and principles
  • 2.RAG retrieval enhancement and model training samples
  • 3.Embodied intelligent robot system architecture
  • 4.Embodied intelligent functions core source code
17.AI Model Developmentnew
  • 1.Register a model service provider account
  • 2. Configure API-KEY
  • 3. Introduction to Dify
  • 4. Basic Dify Features
  • 5. Deploy the RAG knowledge base
  • 6. AI Large Model Development - chatbot
  • 7. RAG knowledge base + chatbot
  • 8. AI agent workflow
18.AI Model – Text Versionnew
  • 1. Semantic understand instruction following
  • 2. Multimodal visual understand
  • 3. Text Large Model + robotic arm control
  • 4. Multimodal Large Model+video understanding
  • 5. Multimodal Large Model+robotic arm gripping
  • 6. Multimodal Large Model+robotic arm tracking
  • 7. Multimodal Large Model+robotic arm sorting
  • 8. Multimodal Large Model+robotic arm handling
  • 9. Multimodal Large Model+Color Blocks Back In Place
  • 10. Multimodal Large Model + Intent Prediction + Robotic Arm Gripping
  • 11. Text Large Model + YOLOv11 + Intelligent Garbage Classification
19.AI Model – Voice Versionnew
  • 1. Semantic understand instruction following
  • 2. Multimodal visual understand
  • 3. Text Large Model + robotic arm control
  • 4. Multimodal Large Model+video understanding
  • 5. Multimodal Large Model+robotic arm gripping
  • 6. Multimodal Large Model+robotic arm tracking
  • 7. Multimodal Large Model+robotic arm sorting
  • 8. Multimodal Large Model+robotic arm handling
  • 9. Multimodal Large Model+Color Blocks Back In Place
  • 10. Multimodal Large Model + Intent Prediction + Robotic Arm Gripping
  • 11. Text Large Model + YOLOv11 + Intelligent Garbage Classification
20.AI Model – Fun Applicationsnew
  • 1.AI Butler-Intent Inference
  • 2.AI Encyclopedia-Arithmetic Master
  • 3.AI Encyclopedia-You asked I Pointed
  • 4.AI Encyclopedia-Nutritional Analysis
  • 5.AI Encyclopedia-Problem Solving Expert
  • 6.AI Encyclopedia-English Spelling
  • 7.AI Butler-Garbage Sorting
  • 8.AI Top Chef
21.MoveIt2 Course
  • 1.MoveIt configuration
  • 2. MoveIt Usage
  • 3. MoveIt Drive Real Robot
  • 4. MoveIt Random Movement
  • 5. MoveIt Forward Kinematics Design
  • 6. MoveIt Inverse Kinematics Design
  • 7. MoveIt Cartesian Path
  • 8. MoveIt Scene Design
  • 9. MoveIt Trajectory Planning
  • 10. MoveIt Collision Detection
22.YOLOv11 Development Course
  • 1. YOLO Environment Setup
  • 2. CLI Usage
  • 3. Object Detection
  • 4. Instance Segmentation
  • 5. Pose Estimation
  • 6. Image Classification
  • 7. Oriented Bounding Boxes Object Detection
  • 8. Model Conversion
  • 9. Model Prediction
23.Linux Operating System
  • 1.Introduction to Linux system
  • 2.Ubuntu file system
  • 3.Ubuntu common commands
  • 4.Ubuntu common editors
  • 5.Ubuntu software operation commands
  • 6.Virtual machine installation
  • 7.SSH remote control
  • 8.VNC remote control
  • 9.Transfer files remotely
  • 10.Driver library and communication
  • 11.Capacity expansion and resource allocation
  • 12.Bind device ID
  • 13.Capacity expansion and resource allocation
  • 14.Update system software sources
  • 15.Set root user password
  • 16.sudo free password
  • 17.Connect to WiFi network
  • 18.View system version
  • 19.Customized service management
  • 20.Back up system image
24.GPIO Hardware Control Course
  • 1.GPIO description
  • 2.GPIO library installation
  • 3.GPIO reading
  • 4.GPIO output
  • 5.Serial communication
  • 6.I2C communication
25.OpenCV course
  • 0. Must Read!
  • 1. Introduction to Open Source CV
  • 2. Image reading and display
  • 3. Image writing
  • 4. Image quality
  • 5. Pixel Operation
  • 6.Image zoom
  • 7.Picture cut
  • 8.Picture pan
  • 9.Picture mirroring
  • 10.Affine transformation
  • 11.Picture rotation
  • 12.perspective transformation
  • 13.Grayscale processing
  • 14.Binary image
  • 15.Edge green detection
  • 16.Line segment drawing
  • 17.Rectangular circle drawing
  • 18.Text and picture drawing
26.ROS2 Basic Course
  • 1.Introduction to ROS2
  • 2.ROS2 install Humble
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
27. Advanced Visuals Course
  • 0. Must Read! Porting and Compilation
  • 1. QR Code
  • 2. Human Pose Estimation
  • 3. Object Detection
  • 4. AR Vision
  • 5. ROS + OpenCV Basics
28.Jupyter Lab Usage Course
  • 0.Read the document before operation
  • 1.Jupyter Lab environment construction
  • 2.Basic use of Jupyter Lab
  • 3.Jupyter Lab plug-in installation
  • 4.Basic use of Jupyter Widgets
29.Docker Course
  • 1.Docker overview and docker installation
  • 2.Common commands for docker image containers
  • 3.Docker images deeply understand and publish images
  • 4.Docker hardware interaction and data processing
  • 5.Start the Dofbot container
30.Python Programming Basics
  • 1.Python usage
  • 2.Use APT to install Python packages
  • 3.Python virtual environment
  • 4.GPIO Zero
  • 5.Pin output high and low levels
  • 6.Pin reads high and low levels
  • 7.Pin output PWM
Download
  • APP(iOS/Android)
  • Code
  • SystemFile
  • VM File
  • PC software
  • Instruction manual
  • Jetson_Boot_USB.tar.gz
  • More Annex
  • Tools
Download ZIP
FAQ
  • 1、Precautions for DIY Robotic Arm
  • 2、Why i can't robotic arm start normally
  • 3、When you control the No.1 servo, all servos will turn.
  • 4、How open VNC service on DOFBOT system
  • 5、Where can I get the URDF file?
  • 6、Gesture recognition cannot work normally.
  • 7、Write system image into U disk
  • 8、Write EMMC boot file

Welcome to DOFBOT-JetsonNANO repository

6.Gesture recognition fixed action

Gesture Recognition Fixed Action

For applying for a Baidu account, you can refer to Chapter 8 for the gesture recognition process. This tutorial is based on successful gesture recognition.

1. Main Code Content

  • Start Docker

After entering the main board desktop, open the terminal and run the following command to start the corresponding Dofbot container:

./Docker_Ros.sh

Then, in the Docker terminal, input the following command to start jupyter-lab,

xxxxxxxxxx
cd
jupyter-lab --allow-root

Then, in the browser, input the main board's IP address plus :9999,

xxxxxxxxxx
IP:9999         // Example: 192.168.1.11:9999

Code path: /root/Dofbot/6.AI_Visual/1.gesture_action.ipynb

xxxxxxxxxx
#bgr8 to jpeg format
import enum
import cv2
def bgr8_to_jpeg(value, quality=75):
 return bytes(cv2.imencode('.jpg', value)[1])
xxxxxxxxxx
#Import related modules
import threading
import time
from Arm_Lib import Arm_Device
# Create robotic arm object
Arm = Arm_Device()
time.sleep(.1)
xxxxxxxxxx
import cv2 
import time
import demjson
import pygame 
from aip import AipBodyAnalysis
from aip import AipSpeech
from PIL import Image, ImageDraw, ImageFont
import numpy
import ipywidgets.widgets as widgets
# For specific gestures, please refer to the official documentation https://ai.baidu.com/ai-doc/BODY/4k3cpywrv
hand={'One':'Number 1','Two':'Number 2','Three':'Number 3','Four':'Number 4',
 'Five':'Number 5', 'Six':'Number 6','Seven':'Number 7',
 'Eight':'Number 8','Nine':'Number 9','Fist':'Fist','Ok':'OK',
 'Prayer':'Prayer','Congratulation':'Bow','Honour':'Farewell',
 'Heart_single':'Single Heart','Thumb_up':'Thumbs Up','Thumb_down':'Thumbs Down',
 'ILY':'I Love You','Palm_up':'Palm Up','Heart_1':'Double Heart 1',
 'Heart_2':'Double Heart 2','Heart_3':'Double Heart 3','Rock':'Rock',
 'Insult':'Middle Finger','Face':'Face'}
# The key below should be replaced with your own
""" Body Analysis APPID AK SK """
APP_ID = '18550528'
API_KEY = 'K6PWqtiUTKYK1fYaz13O8E3i'
SECRET_KEY = 'IDBUII1j6srF1XVNDX32I2WpuwBWczzK'
client = AipBodyAnalysis(APP_ID, API_KEY, SECRET_KEY)
xxxxxxxxxx
g_camera = cv2.VideoCapture(0)
g_camera.set(3, 640)
g_camera.set(4, 480)
g_camera.set(5, 30) #Set frame rate
g_camera.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M', 'J', 'P', 'G'))
g_camera.set(cv2.CAP_PROP_BRIGHTNESS, 40) #Set brightness -64 - 64 0.0
g_camera.set(cv2.CAP_PROP_CONTRAST, 50) #Set contrast -64 - 64 2.0
g_camera.set(cv2.CAP_PROP_EXPOSURE, 156) #Set exposure value 1.0 - 5000 156.0
ret, frame = g_camera.read()
xxxxxxxxxx
# Define camera display component
image_widget = widgets.Image(format='jpeg', width=600, height=500) #Set camera display component
display(image_widget) 
image_widget.value = bgr8_to_jpeg(frame)
xxxxxxxxxx
# Define function to convert and display Chinese text
def cv2ImgAddText(img, text, left, top, textColor=(0, 255, 0), textSize=20):
 if (isinstance(img, numpy.ndarray)): # Determine if it's OpenCV image type
 img = Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
 # Create an object that can draw on the given image
 draw = ImageDraw.Draw(img)
 # Font format
 fontStyle = ImageFont.truetype(
 "simhei.ttf", textSize, encoding="utf-8")
 # Draw text
 draw.text((left, top), text, textColor, font=fontStyle)
 # Convert back to OpenCV format
 return cv2.cvtColor(numpy.asarray(img), cv2.COLOR_RGB2BGR)
xxxxxxxxxx
#Define servo angles for different positions
look_at = [90, 164, 18, 0, 90, 90]
p_Prayer = [90, 90, 0, 180, 90, 180] #Prayer
p_Thumb_up = [90, 90, 90, 90, 90, 180] #Thumbs Up
p_Heart_single = [90, 0, 180, 0, 90, 30] #Single Heart
p_Eight = [90, 180, 18, 0, 90, 90] #Eight
p_Congratulation = [90, 131, 52, 0, 90, 180] #Bow
p_Rock = [90, 0, 90, 180, 90, 0] #rock
p_fist = [90, 90, 0, 0, 90, 0] #Fist
p_horse_1 = [90, 7, 153, 19, 0, 126] #
p_horse_2 = [90, 5, 176, 0, 0, 180]
p_horse_3 = [90, 62, 158, 0, 0, 0]
global running
running = 0
xxxxxxxxxx
# Define function to move robotic arm, control 1-6 servo movement simultaneously, p=[S1,S2,S3,S4,S5,S6]
def arm_move_6(p, s_time = 500):
 for i in range(6):
 id = i + 1
 Arm.Arm_serial_servo_write(id, p[i], s_time)
 time.sleep(.01)
 time.sleep(s_time/1000)
 
# Define horse movement
def horse_running():
 Arm.Arm_serial_servo_write(6, 150, 300)
 time.sleep(.3)
 Arm.Arm_serial_servo_write(6, 180, 300)
 time.sleep(.3)
xxxxxxxxxx
global g_state_arm
g_state_arm = 0
def ctrl_arm_move(index):
 global running
 if index == "Prayer":
 arm_move_6(p_Prayer, 1000)
 time.sleep(1.5)
 arm_move_6(look_at, 1000)
 time.sleep(1)
 elif index == "Thumb_up":
 s_time = 500
 Arm.Arm_serial_servo_write(6, 180, s_time)
 time.sleep(s_time/1000)
 Arm.Arm_serial_servo_write(6, 90, s_time)
 time.sleep(s_time/1000)
 Arm.Arm_serial_servo_write(6, 180, s_time)
 time.sleep(s_time/1000)
 Arm.Arm_serial_servo_write(6, 90, s_time)
 time.sleep(s_time/1000)
 elif index == "Ok":
 s_time = 300
 Arm.Arm_serial_servo_write(4, 10, s_time)
 time.sleep(s_time/1000)
 Arm.Arm_serial_servo_write(4, 0, s_time)
 time.sleep(s_time/1000)
 Arm.Arm_serial_servo_write(4, 10, s_time)
 time.sleep(s_time/1000)
Arm.Arm_serial_servo_write(4, 0, s_time)
 time.sleep(s_time/1000)
 elif index == "Heart_single":
 arm_move_6([90, 90, 90, 90, 90, 90], 800)
 time.sleep(.1)
 arm_move_6(p_Heart_single, 1000)
 time.sleep(1)
 elif index == "Five":
 arm_move_6(look_at, 1000)
 time.sleep(.5)
 elif index == "Eight":
 s_time = 300
 arm_move_6(p_Eight, 0)
 time.sleep(1)
 Arm.Arm_serial_servo_write(2, 165, s_time)
 time.sleep(s_time/1000)
 elif index == "Rock": #rock
 Arm.Arm_serial_servo_write6_array(p_Rock, 1300)
 time.sleep(3)
 Arm.Arm_serial_servo_write6_array(look_at, 1000)
 time.sleep(1)
 elif index == "Thumb_down": #拇指向下
 Arm.Arm_serial_servo_write6_array(p_horse_1, 1300)
 time.sleep(1) 
 elif index == "Congratulation": #作揖
 Arm.Arm_serial_servo_write6_array(p_horse_2, 1000)
 time.sleep(1)
 running = 1
 while running == 1:
 horse_running()
 elif index == "Seven": #数字 7
 Arm.Arm_Buzzer_On(8) #蜂鸣器自动响 0.5 秒
 Arm.Arm_serial_servo_write6_array(p_horse_3, 1000)
 time.sleep(2)
 Arm.Arm_serial_servo_write6_array(look_at, 1000)
 time.sleep(1)
 
 global g_state_arm
 g_state_arm = 0
xxxxxxxxxx
# Move robotic arm to position looking forward with camera
arm_move_6(look_at, 1000)
time.sleep(1)
xxxxxxxxxx
def start_move_arm(index):
 # Enable robotic arm control thread
 global g_state_arm
 global running
 if g_state_arm == 0:
 closeTid = threading.Thread(target = ctrl_arm_move, args = [index])
 closeTid.setDaemon(True)
 closeTid.start()
 g_state_arm = 1
 
 if running == 1 and index == "Seven":
 running = 0
​x
# Main process
try:
     Arm.Arm_Buzzer_On(1)
     s_time = 300
     Arm.Arm_serial_servo_write(4, 10, s_time)
     time.sleep(s_time/1000)
     Arm.Arm_serial_servo_write(4, 0, s_time)
     time.sleep(s_time/1000)
     Arm.Arm_serial_servo_write(4, 10, s_time)
     time.sleep(s_time/1000)
     Arm.Arm_serial_servo_write(4, 0, s_time)
     time.sleep(s_time/1000)
​
     while True:
         """1. Take photo """
         ret, frame = g_camera.read()
         #image = get_file_content('./image.jpg')
         """ 2. Call gesture recognition """
         raw = str(client.gesture(image_widget.value))
         text = demjson.decode(raw)
         try:
         res = text['result'][0]['classname']
         except:
             # print('识别结果：什么也没识别到哦~' )
             # img = cv2ImgAddText(frame, "未识别", 250, 30, (0, 0 , 255), 30)
             img = frame
         else:
             # print('识别结果：' + hand[res])
             # img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
            www.yahboom.com
         if res == 'Prayer': # 1 Prayer
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res) 
         elif res == 'Thumb_up':# 2 Thumbs Up
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == 'Ok': # 3 OK
             print('Recognition result:' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == 'Heart_single': # 4 Single Heart
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == 'Five': # 5 数字 5
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == "Eight": # 数字 8
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
​
         elif res == "Rock": # rock
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == "Congratulation": # Bow
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == "Seven": # 数字 7
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res)
         elif res == "Thumb_down": # Thumbs Down
             print('识别结果：' + hand[res])
             img = cv2ImgAddText(frame, hand[res], 250, 30, (0, 255 , 0), 30)
             start_move_arm(res) 
 
         else:
            img = frame
 image_widget.value = bgr8_to_jpeg(img)
except KeyboardInterrupt:
     print(" Program closed! ")
     pass
​

If a recognized gesture action is detected, the robotic arm will perform the corresponding action. Currently, the gesture and action correspondence is as follows:

 

 

 

 

 

 

 

 

 

Face tracking
Gesture recognition stack blocks

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文