Welcome to DOFBOT-Ultra repository

Face Tracking Experiment

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Based on the face positioning function, combined with the arm to achieve face tracking. After the program starts, the arm automatically follows the face in the frame.

1. Startup and Operation

Open a browser (any browser on any host in the same local network) and enter the IP address:8888 in the address bar to access JupyterLab.

Code path:

xxxxxxxxxx
dofbot_face_follow/scripts/Face_follow.ipynb

Click the Run All button on the JupyterLab toolbar, then scroll to the bottom.

The camera feed will be displayed. Place your face in the camera view, and the arm will follow it. Do not move your face too quickly, or the arm may not keep up.

To end the program, click [Exit] to avoid affecting resource usage by other programs.

2. Technical Overview

Face tracking is an advanced function built on face positioning, adding arm联动:

1. Face Detection: Same as the face positioning experiment, detect the face in the frame and get the center point coordinates.

2. Deviation Calculation: Calculate the X and Y direction deviations between the face center and the image center.

3. PID Control: Convert the deviation into smooth servo control values using a PID algorithm to prevent arm jitter.

4. Arm Motion: Drive the arm base (left/right rotation) and lift joint (up/down pitch) to keep the face centered in the frame.

3. Core Code Analysis

• Import header files.

xxxxxxxxxx
import cv2 as cv
import threading
from time import sleep
import ipywidgets as widgets
from IPython.display import display
from face_follow import Face_Follow
from dofbot_utils.fps import FPS
from dofbot_utils.robot_controller import Robot_Controller
from dofbot_utils.dofbot_config import *
from dofbot_utils.camera import create_camera

• Create instances and initialize parameters.

xxxxxxxxxx
robot = Robot_Controller()
robot.move_init_pose()
fps = FPS()
follow = Face_Follow()
model = 'General'

• Create widgets.

xxxxxxxxxx
button_layout = widgets.Layout(width='250px', height='50px', align_self='center')
output = widgets.Output()
exit_button = widgets.Button(description='Exit', button_style='danger', layout=button_layout)
imgbox = widgets.Image(format='jpg', height=480, width=640, layout=widgets.Layout(align_self='center'))
controls_box = widgets.VBox([imgbox, exit_button], layout=widgets.Layout(align_self='center'))

• Exit program.

xxxxxxxxxx
def exit_button_Callback(value):
    global model
    model = 'Exit'
exit_button.on_click(exit_button_Callback)

• Main program.

xxxxxxxxxx
def camera():
    global model
    capture = create_camera()
    while capture.isOpened():
        try:
            _, img = capture.read()
            fps.update_fps()
            img, pos = follow.follow_function(img)
            if model == 'Exit':
                cv.destroyAllWindows()
                capture.release()
                break
            fps.show_fps(img)
            imgbox.value = cv.imencode('.jpg', img)[1].tobytes()
        except KeyboardInterrupt:capture.release()

• Startup.

display(controls_box,output)
threading.Thread(target=camera, ).start()