Tracking Game

If using a depth camera (DABAI_DCW2), make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration before running this example. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

The Tracking Game function is based on color recognition, combined with forward and inverse kinematics to control the arm movement. After selecting a color, place the block of that color in front of the arm camera. The closer the block gets to the arm, the further it retreats; the further the block moves away, the arm follows forward. When the maximum distance is reached, the arm grabs the block and places it in the corresponding color area.

For all functions involving grasping of square blocks, please use 30x30x30 mm wooden blocks. Non-standard sized objects may cause servo No. 6 stall and damage.

Note: Before starting the program, please follow the [Assembly Tutorial] -> [Install the Map] tutorial to correctly install the map before operation.

1. Startup and Operation

Start ROS Node Service

Open the system terminal and enter the following command. If already running, there is no need to start again.

xxxxxxxxxx
ros2 run dofbot_ultra_info kinematics_dofbotultra

Start the Program

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Raspberry Pi code path:

xxxxxxxxxx
/home/yahboom/dofbot_ultra_ws/src/dofbot_snake_follow/scripts/Snake_Follow.ipynb

RDK X5 code path:

xxxxxxxxxx
/home/sunrise/dofbot_ultra_ws/src/2d/dofbot_snake_follow/scripts/Snake_Follow.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

Experiment Effect

After the program runs, the interface is displayed as shown. Click any color button to select a color. The selected color is displayed in the center above the image. Here, yellow is used as an example.

When the selected color appears in view, the arm estimates the block's position based on its area in the image.

When the area decreases, the arm moves forward; when the area increases, the arm retreats.

When the arm extends to the maximum distance, the gripper opens. At this point, place the block into the gripper, grab it, and place it in the corresponding color area.

Before performing the next recognition operation, please remove the block to avoid conflicts during placement.

To exit the program, click the [Exit] button.

Raspberry Pi HSV Calibration Note: The HSV color calibration file is saved at /home/yahboom/dofbot_ultra_ws/src/dofbot_snake_follow/scripts/HSV_config.txt. If color recognition is not accurate enough, open the file /home/yahboom/dofbot_ultra_ws/src/dofbot_snake_follow/scripts/HSV_calibration.ipynb to recalibrate the block color HSV values.

RDK X5 HSV Calibration Note: The HSV color calibration file is saved at /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_snake_follow/scripts/HSV_config.txt. If color recognition is not accurate enough, open the file /home/sunrise/dofbot_ultra_ws/src/2d/dofbot_snake_follow/scripts/HSV_calibration.ipynb to recalibrate the block color HSV values.

2. Technical Overview

The Tracking Game is an interactive function. Its core principles:

1. Color Selection: The user selects a target color (red/green/blue/yellow) through the interface

2. Color Detection and Area Estimation: Identify the target block through HSV color space and calculate the block's area in the image. The area size reflects the object's distance

3. Forward Kinematics (FK): Calculate the end effector's position in space based on the current joint angles of the arm

4. Distance Control:

   • Block area increases (closer) → Arm retreats via inverse kinematics (IK)

   • Block area decreases (farther) → Arm extends forward via inverse kinematics (IK)

5. Grab Trigger: When the arm extends to the maximum distance, the gripper automatically opens, waiting for the user to place the block

6. Place and Return: After grabbing the block, place it in the corresponding color area

3. Core Code Analysis

• Import header files

​x
#!/usr/bin/env python
# coding: utf-8
import cv2 as cv
import threading
from time import sleep

import ipywidgets as widgets
from IPython.display import display
from snake_target import snake_target
from snake_ctrl import snake_ctrl
from dofbot_utils.dofbot_config import *

• Create instances and initialize parameters

xxxxxxxxxx
import Arm_Lib
Arm = Arm_Lib.Arm_Device()
joints_0 = [90, 135, 0, 45, 0, 180]
Arm.Arm_serial_servo_write6_array(joints_0, 1000)

xxxxxxxxxx
snake_target = snake_target()
snake_ctrl = snake_ctrl()
model = 'General'
color = [[random.randint(0, 255) for _ in range(3)] for _ in range(255)]
color_hsv  = {"red"   : ((0, 43, 46), (10, 255, 255)),
              "green" : ((35, 43, 46), (77, 255, 255)),
              "blue"  : ((100, 43, 46), (124, 255, 255)),
              "yellow": ((26, 43, 46), (34, 255, 255))}
HSV_path="/home/yahboom/dofbot_ultra_ws/src/dofbot_snake_follow/scripts/HSV_config.txt"
try: read_HSV(HSV_path,color_hsv)
except Exception: print("Read HSV_config Error!!!")

• Create widgets

xxxxxxxxxx
button_layout      = widgets.Layout(width='150px', height='27px', align_self='center')
output = widgets.Output()
choose_color=widgets.ToggleButtons( options=['red', 'green', 'blue','yellow'], button_style='success', 
    tooltips=['Description of slow', 'Description of regular', 'Description of fast'])
exit_button = widgets.Button(description='Exit', button_style='danger', layout=button_layout)
imgbox = widgets.Image(format='jpg', height=480, width=640, layout=widgets.Layout(align_self='center'))
down_box = widgets.HBox([choose_color,exit_button], layout=widgets.Layout(align_self='center'));
controls_box = widgets.VBox([imgbox, down_box], layout=widgets.Layout(align_self='center'))

• Main program

xxxxxxxxxx
def camera():
    from dofbot_utils.camera import create_camera
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            img, snake_msg = snake_target.target_run(img, color_hsv)
            if len(snake_msg) == 1:
                threading.Thread(target=snake_ctrl.snake_main, args=(choose_color.value, snake_msg,)).start()
            if model == 'Exit':
                capture.release()
                break
            cv.putText(img, choose_color.value, (int(img.shape[0] / 2), 50), cv.FONT_HERSHEY_SIMPLEX, 2, color[random.randint(0, 254)], 2)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
        except KeyboardInterrupt:capture.release()

• Start display

display(controls_box,output)
threading.Thread(target=camera, ).start()