1. Mobile Web Control

Table of Contents

• 1. Startup Methods
• 2. Interface Overview
• 3. Settings - System & Calibration
• 4. Remote Control - Basic Control
• 5. Motion - Fixed Motion Groups & Custom Motions
• 6. Gesture - Gesture Interaction
• 7. Color - Color Interaction
• 8. Tracking - Tracking Modes
• 9. Sorting - Waste Recognition
• 10. Sorting - Color Sorting

1. Startup Methods

By default, the web control backend process starts automatically at boot. No manual startup is needed. If you hear a startup tone after powering on DofbotUltra and the arm performs an initializing motion, the web control backend process is running normally.

All functions involving grasping cubic objects must use 30x30x30 mm blocks. Non-standard sized objects may cause servo 6 to stall and be damaged.

1.1 Start the Service

For Raspberry Pi, execute the startup script in the Docker terminal:

x
cd /home/yahboom/dofbot_ultra/app/APP_DOFBOT_ULTRA
bash start_app.sh

For RDK X5, run the script directly on the host:

xxxxxxxxxx
cd /home/sunrise/dofbot_ultra_ws/src/app/APP_DOFBOT_ULTRA
bash start_app.sh

This script will automatically do the following:

1. Load the ROS2 Humble environment
2. Start the ROS forward kinematics service (kinematics_dofbotultra, required by the Snake Attraction feature)
3. Start the web main program (YahboomArm.py)

Note: If you see the error "Camera failed to open. Attempting to force release /dev/video0...", the camera is occupied. In this case, try restarting the device.

1.2 Access the Control Interface via Mobile Browser

Here I am using the Via browser on my phone. Generally speaking, most browsers should work fine. You can even use it on a computer browser for an even better experience!

When using on a phone, the browser needs to be in landscape mode. Different phones/browsers have different settings for this, which will not be explained here.

After successful startup, connect your phone to the same local network as the device, open your mobile browser, and visit:

xxxxxxxxxx
http://<Device IP>:6500

For example: http://192.168.1.100:6500

The device IP address can be found in the terminal output, through the router management interface, or on the DofbotUltra OLED screen.

Tip: Landscape mode on your phone provides a better experience. The left side of the interface shows the camera feed, and the right side shows the control panel.

1.3 Stop the Service

Press Ctrl + C in the terminal to stop all related processes.

2. Interface Overview

After opening the control page with your mobile browser, you will see the following layout:

Click the tab icons at the top to switch between different function modules. Swipe left or right if there are many tabs.

3. Settings - System & Calibration

The Settings tab is the home page displayed after startup. It includes system settings, servo center calibration, color calibration, etc. It is recommended to check each setting before first use.

3.1 System Settings

• Language Switch: Supports Chinese and English bilingual switching.

3.2 Servo Center (Check)

Click to return all servos to the standard center position. Used to quickly check if the arm is correctly assembled.

3.3 Servo Center Calibration (Offset)

When a servo is installed with an offset, use this function to calibrate:

1. Click Start Calibration - Servos are released.
2. Manually adjust each servo to the standard pose (align the scale marks).
3. Click the corresponding ID button to save the current position as the new center.
4. Click End Calibration - Servos lock.

To restore factory center, click Restore Default Center.

3.4 Color Calibration (Color Learning)

When environmental lighting changes cause inaccurate color recognition, recalibrate the HSV values.

Quick Calibration (Color Learning)

1. Select the color to calibrate (Red / Green / Blue / Yellow).
2. Place the object of the corresponding color in the rectangular frame at the center of the screen.
3. The system automatically learns the HSV range of that color.
4. After confirmation, click Save Result.

Precision HSV Calibration

It is recommended to click Restore Default Values below.

Enter the precision calibration interface:

1. Select the color to calibrate.
2. The left side displays an HSV binarization preview (black and white mask).
3. Adjust the 6 sliders: Hmin, Hmax, Smin, Smax, Vmin, Vmax.
4. Goal: The target color block should appear completely white in the preview, and the background should appear completely black.
5. Once satisfied, click Confirm Save.

Note: In OpenCV, the H channel range is 0-179 (not 0-255). Setting all sliders to maximum does not mean selecting all colors.

4. Remote Control - Basic Control

Switch to the Remote tab for basic arm control.

4.1 Basic Buttons

4.2 Servo Slider Control (ID 1-6)

The interface provides 6 sliders to control the angles of servos ID1 to ID6 (0° to 180°).

Click Read Current Angle to synchronize the current actual angles of the arm to the sliders.

5. Motion - Fixed Motion Groups & Custom Motions

Switch to the Motion tab.

5.1 Fixed Motion Groups

6 preset motions are available. Click to execute:

5.2 Custom Motions (Learning Mode)

Custom motions allow you to manually teach the arm a series of poses and replay them. Steps:

1. Start Learning: Click Learning Mode button. The arm servos release, allowing free manual adjustment.
2. Record Motion: Manually pose the arm to the target position, then click Record Motion to save the current pose. Repeat to record multiple key poses.
3. Finish Learning: Click Finish Learning. The arm locks and saves all recorded poses.
4. Playback: Click Single Run to replay the recorded motion sequence once.
5. Stop Playback: Click Stop during playback to interrupt.
6. Clear: Click Clear Motions to delete all recorded motion data.

Note: Follow the sequence: Start Learning -> Record Motion -> Finish Learning -> Single Run.

6. Gesture - Gesture Interaction

Switch to the Gesture tab. Based on MediaPipe gesture recognition, the arm responds to your gestures.

6.1 Gesture Recognition

Turn on the Gesture Recognition switch and place your hand in front of the camera. The arm recognizes these 5 gestures:

The top-left corner of the screen displays the recognized gesture and a confirmation progress bar. The gesture must be held stable for about 20 frames to trigger the action.

6.2 Gesture Stacking

Turn on the Gesture Stacking switch to command the arm to pick up different colored blocks in sequence for stacking:

Stacking supports up to 4 layers. After pushing over, stacking can be restarted.

Note: If grasping or placement is inaccurate during stacking, modify the grasping/placement coordinates in the code file dofbot_utils/robot_controller.py.

Default stacking poses:

self.P_CENTER_HEAP_L1 = [90, 35, 65, 0, 90, self.gripper_clamp]

self.P_CENTER_HEAP_L2 = [90, 50, 55, 0, 90, self.gripper_clamp]

self.P_CENTER_HEAP_L3 = [89, 68, 44, 2, 90, 133]

self.P_CENTER_HEAP_L4 = [90, 90, 28, 5, 90, self.gripper_clamp]

7. Color - Color Interaction

Switch to the Color tab. Three color-based interactive modes are available.

7.1 You Place, I Grasp

Turn on the You Place, I Grasp switch:

1. The arm moves to a pose overlooking the workspace.
2. A rectangular recognition area appears at the center of the screen.
3. Place a red / green / blue / yellow block in the recognition area.
4. The arm automatically recognizes the color, confirms for 10 consecutive frames, then grasps and places it at the designated position.

Tip: Place only one colored object at a time. Wait for the arm to complete the grasp and return to the observation pose before placing the next one.

7.2 Color Grasp

Turn on the Color Grasp switch:

1. The arm automatically initializes to the observation pose and emits a beep.
2. A detection frame appears at the center of the screen.
3. Place a color block in the detection frame. The system shows the detected color and confirmation progress in real time.
4. After 20 consecutive frames of confirmation, the arm automatically grasps the block and places it in the center area.

7.3 Snake Attraction

Select a color and turn on the switch:

1. The arm moves to a forward-reaching observation pose.
2. Place an object of the selected color in front of the arm.
3. The arm extends forward like a "snake" based on the target area, gradually approaching the object.
4. If the target suddenly becomes too large (too close), the arm "startles" and retreats while clicking the gripper.
5. When the "snake" is fully extended, it automatically closes the gripper, grabs the object, and places it at the corresponding color position.

Supports Red, Green, Blue, Yellow as target colors.

Note: Turn off the current switch before switching colors. This feature depends on the ROS forward kinematics service. If the service is not started, it automatically degrades to interpolation mode.

8. Tracking - Tracking Modes

Switch to the Tracking tab. Two modes are available: face tracking and color tracking.

8.1 Face Tracking

Turn on the Face Tracking switch:

1. The arm automatically adjusts to the tracking initial pose.
2. The camera detects faces in real time.
3. When a face is detected, a green rectangle highlights the face position on screen.
4. The arm follows the face movement in real time using PID control.

Tip: Keep your face facing the camera at a distance of 30-80 cm for best results.

8.2 Color Tracking

Click the corresponding color button to start color tracking:

When enabled, the arm follows the largest color block of the corresponding color in real time. The RGB LED light synchronizes with the tracked color.

9. Sorting - Waste Recognition

Auto Waste (Recognition)

Turn on the Auto Sorting switch:

1. The system loads the YOLOv11 object detection model (the first load may take a while).
2. After loading, it automatically enters real-time recognition mode.

Note:

• Model loading may take 10-30 seconds. Please be patient.
• The top of the screen shows the current status (Model Loading / Recognizing / Sorting).
• This feature requires additional YOLO model files.

10. Sorting - Color Sorting

Switch to the Sorting tab. This is a complete color sorting workflow in three steps.

Step 1: Calibration

Calibration determines the mapping between the camera view and the arm coordinates. Complete visual calibration on the settings page.

Step 2: Set Color Order

Set the colors for each of the 4 slots. The arm sorts blocks sequentially according to the slot order.

Step 3: Detection and Grasping

1. Click Start Detection. Detected color blocks on screen will be highlighted.
2. Confirm detection is working, then click Start Grasping. The arm automatically performs sorting.
3. Click Stop to interrupt at any time, or Reset List to reconfigure.

Tip: Ensure calibration is completed correctly before sorting, otherwise grasping accuracy will be affected.