App Control

1. Dofbot-SE wiring and launching the main program

• The camera is connected to the computer via a USB extension cable
• The micro USB end connects to the Dofbot-SE expansion board, USB end to the computer USB port

Insert the STM32 core board into the Dofbot-SE expansion board, and turn on the expansion board's power. If the LED on the STM32 core board blinks every second, it indicates the factory firmware loaded successfully. If the LED indicator does not light up, please try reflashing the STM32 core board firmware. The indicators are shown below.

Open a terminal, and run the following command to start the main program; when you hear three beeps from the buzzer, the program started successfully

python3 ~/rosmaster/YahboomArm.pyc

Later, during the app connection steps, you can enter the IP shown in the terminal to connect. Note that for the application to connect successfully, the VM and the phone must be on the same network

2. Mobile App Connection and Control

2.1 When you first open the YahboomRobot app, you need to select the corresponding product. In the home page, choose 【Kit】 → 【Dofbot】 from the left sidebar.

2.2 Follow prompts to choose the network connection method; choose wired connection. The Dofbot-SE version does not support wireless connections.

2.3 The phone automatically searches for devices; after a few seconds a dialog will pop up; simply click Refresh to connect. If nothing appears for a long time, click Search Device and try again.

2.4 If no devices are found for a long time, check the robot's network status. You can also try IP connection. IP: enter the robot's IP address, "IPort": 6000, "Video": 6500. For example 192.168.1.68; this IP address must be the VM's IP address

2.5. Arm calibration guide: Before using the arm, the arm's center position is crucial; the offset affects later gameplay. Press the 【Middle】 button to determine whether the arm is upright, center horizontally and that the gripper is tightened. If normal, click 【Skip】; if not, click 【Calibrate】.

2.6 After clicking 【Calibrate】, you will enter the calibration state. Carefully check that the arm is upright, horizontal centering is normal, and the gripper is pressed tightly. When the adjustment is complete, click 【OK】. When all the indicator rings change from blue to green, the setup is successful; click 【Next】.

2.7. Color calibration guide: Place the red block within the camera's field of view so that the detection box fully covers the color; release your hand, wait for the HSV values to stabilize in real time, and when OK appears, click 【Color Calibration】 to finish calibrating that color. Then calibrate green, blue, and yellow blocks in the same way. Finally click 【Finish】 to end the guide.

Arm App Features Overview

The Dofbot arm app main interface is divided into eight modules, each with different functions.

1) Remote Control

On the main interface, click the 【Remote Control】 icon to view the following screen.

On the left is the camera view; at the top is a robotic arm with numbers 1-6 representing the six joints. When an ID is highlighted in red, that joint is selected; you can drag the bottom slider or use the left and right buttons to adjust the joint position.

【Angle】: Each time you click, the app reads the current angle of the selected joint and updates the above slider.

【Middle】: Center the arm vertically.

【Stop】: The first click disables torque; joints will not accept commands and cannot hold the current position, allowing you to manually move the joints. The second click re-enables torque and all joints hold their current position and can receive commands.

2) Action Groups

In the main interface, click the 【Action Groups】 icon to view the following screen.

There are two play modes for action groups: the left is for custom actions, the right is for fixed actions. The middle area shows the action group previews, run actions, and stop actions. Click 【Run】 to execute the current action group; click 【Stop】 to stop the action group.

【Customize action groups】: You can train the arm to learn to execute action groups. Click 【Study Mode】; a dialog will appear, and the RGB LED on the arm's expansion board will show a blue breathing light.

Click the 【Record X Action】 button, and the arm will record the current posture as an action group. The RGB LED on the expansion board will breathe and change color to indicate completion of this action. After recording multiple actions, to exit click 【Completed】; the expansion board LED will turn off. If a red breathing LED appears, learning failed or the action group storage is full (up to 20 groups); click 【Completed】 to exit.

【Fixed action groups】: Click different number buttons; from the preview window you can see the function of the corresponding action group. When you click Run, the fixed robotic arm action group is executed.

3) Gesture Interaction

In the main interface, click the 【Gesture Interaction】 icon to view the following screen.

Gesture Interaction includes two modes: Gesture action and Gesture stack. After selecting the function, click the 【Switch】 to start; the function preview window shows the gestures that can be recognized; clicking the Play switch again can turn it off.

【Gesture action】: Recognizes multiple gestures and executes corresponding actions.

【Gesture stack】: Recognize gestures 1-2-3-4, grabbing yellow, red, green, and blue blocks respectively and stacking them in order. Recognize a fist to push all blocks down and reset recognition data.

4) Color Recognition

On the main interface, click 【Color Recognition】 to view the following screen.

Color Interaction includes three modes: Catch game, Color capture, and Color interaction. After selecting the function, click the Play development slider to enable; click again to disable.

【Catch game】: Place blocks within the camera-recognition area; the arm will automatically identify the color and grab the block, placing it into the region corresponding to that color.

【Color capture】: Place the block in front of the camera; after the arm identifies the color, grab the block from the color region and move it to the middle area.

【Color interaction】: After selecting a color, enable the mode and place blocks of that color in front of the arm's camera. The arm mimics a snake to lure it toward the longest path, opens its jaws to grab the block and place it down. If the grabbed block is a different color or the arm approaches too closely, the 'snake' will shake in fear. When switching colors, first turn off the mode.

5) Tracking Game

On the main interface, click the 【Tracking Game】 icon to view the following screen.

Tracking Play includes three modes: Color tracking, Custom color tracking, and Face tracking. Clik the 【switch】 to enable; clik again to disable.

【Color tracking】: Select a color, turn on the Play switch, then place the corresponding color blocks in front of the camera; move the blocks and the arm will follow.

【Custom color tracking】: Click Take Color Tracking; a square will appear in the camera view. Fill the color to track into the box; after accurate color capture, toggle the Play switch and the system will fully detect the color contour; the arm will follow.

【Face tracking】: The arm will detect whether there is a face in the camera view; if so, it will mark and follow the face.

6) Garbage Sorting

On the main interface, click 【Garbage Sorting】 to view the following screen.

Click the 【Switch】 to load the model; wait until the red prompt "Model-Loading…" disappears; place the block with trash image within the camera view. The sorting function identifies the type of trash on the block; after 10 recognitions, sort the trash to the corresponding map location by type.

7) Developer Play (Beta)

On the main interface, click 【Developer Play (Beta)】 to view the following screen.

Adjust the 【Up/Down/Left/Right】 buttons to move the arm so the box fully appears in view; slide the 【Threshold】 slider to adjust edge detection thresholds until all four edges are detected, as shown above. Click 【OK】 to proceed to the 【Sorting Game】 interface shown below.

【Color Sorting】: Click 1, 2, 3, 4 boxes to change colors (black means not selected). Click 【Color Sorting】 to open the color recognition channel; when recognition is correct, toggle the Play switch to sort and grab in the chosen color order.

【Garbage Sorting】: Click 【Garbage Sorting】, load the model, please wait; when the red prompt "Model-Loading …" disappears, place the block with trash image in the camera view; the system automatically recognizes and click the 【switch】 to sort and grab.

8) DOFBOT Setting

On the main interface, click 【DOFBOT Setting】 to view the following screen.

DOFBOT Settings include three functions: 1) Servo Calibration, 2) Color Calibration, 3) Restore Default Configuration.

【Servo Calibration】: The function is the same as in the guidance.

【Color Calibration】: Color calibration has two functions: the first is 【Basic color calibration】, the second is 【Advanced color calibration】.

【Basic color calibration】: See 2.7. Color calibration guide.

【Advanced color calibration】: First place the four colors' blocks into the view; select the color to calibrate; you can slide the [Display switch] to view a black-and-white image; adjust with HSV sliders until interfering colors are not detected; click 【Color Calibration】 to complete calibration for that color, then calibrate the other colors similarly. Verify results by freely moving the blocks; under color view, the blocks should be recognized accurately. Click 【Back】 to end the guide. For details, see the 【06.AI Vision Basic Course】 -> 【2.Color threshold adjustment&Color calibration】.

【Restore Default Configuration】: Clears the APP configuration data; you will need to re-run the arm setup before you can use it again.