Welcome to DOFBOT-Ultra repository

Label Code Tracking Experiment

Before running this example, make sure to execute ros2 run dofbot_ultra_driver calibrate_6point for a six-point calibration. Otherwise, the arm cannot accurately locate the target.

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Based on the label code positioning function, combined with the arm to achieve label code tracking. After the program starts, the arm automatically follows the AprilTag in the frame.

1. Startup and Operation

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Code path:

xxxxxxxxxx
dofbot_apriltag/scripts/Apriltag_Follow.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

The camera feed will be displayed. Click the [Start] button to enable tracking. Place the label code block in the camera view. The label code will be correctly located and its ID number displayed. Move the block, and the arm follows it.

Do not move the block too quickly, or the arm may not keep up.

To stop tracking, click the [Cancel] button.

To end the program, click the [Exit] button to avoid affecting resource usage by other programs.

2. Technical Overview

Label code tracking is an advanced function built on label code positioning, adding arm联动:

1. Tag Detection: Same as the label code positioning experiment, detect the AprilTag in the frame and get its center coordinates and ID.

2. Deviation Calculation: Calculate the X and Y direction deviations between the tag center and the image center.

3. PID Control: Convert the deviation into smooth servo control values using a PID algorithm.

4. Arm Motion: Drive the arm to keep the tag centered in the frame.

3. Core Code Analysis

• Import header files.

xxxxxxxxxx
import cv2 as cv
import threading
import random
from time import sleep
import ipywidgets as widgets
from IPython.display import display
from apriltag_identify import ApriltagIdentify
from apriltag_follow import Apriltag_Follow
from dofbot_utils.fps import FPS
from dofbot_utils.robot_controller import Robot_Controller
from dofbot_utils.camera import create_camera

• Create instances and initialize parameters.

xxxxxxxxxx
apriltag_Identify = ApriltagIdentify()
follow = Apriltag_Follow()
model = 'General'
robot = Robot_Controller()
robot.move_init_pose()
fps = FPS()

• Create widgets.

xxxxxxxxxx
button_layout = widgets.Layout(width='320px', height='60px', align_self='center')
output = widgets.Output()
start_button = widgets.Button(description='Start', button_style='success', layout=button_layout)
cancel_button = widgets.Button(description='Cancel', button_style='danger', layout=button_layout)
exit_button = widgets.Button(description='Exit', button_style='danger', layout=button_layout)
box_button = widgets.VBox([start_button, cancel_button, exit_button], layout=widgets.Layout(align_self='center'))
imgbox = widgets.Image(format='jpg', height=480, width=640)
display_box = widgets.HBox([imgbox, box_button])

• Mode switching.

​x
def exit_button_Callback(value):
    global model
    model = 'Exit'
    
def start_button_Callback(value):
    global model
    model = 'Start'
    
def cancel_button_Callback(value):
    global model
    model = 'General'

exit_button.on_click(exit_button_Callback)
start_button.on_click(start_button_Callback)
cancel_button.on_click(cancel_button_Callback)

• Main program.

xxxxxxxxxx
def camera():
    global model
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            img = cv.convertScaleAbs(img, beta=35)
            fps.update_fps()
            img, msg = apriltag_Identify.getApriltagPosition(img)
            if model == 'Start':
                follow.follow_function(msg)
            if model == 'Exit':
                capture.release()
                break
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
        except Exception as e:
            print("program end")
            print(e)
            capture.release()

• Startup.

display(controls_box,output)
threading.Thread(target=camera, ).start()