Welcome to DOFBOT-Ultra repository

Model Conversion

[!NOTE]

Note: This uses D-Robotics' OpenExplore package and BPU algorithm toolchain. For environment setup, please refer to the D-Robotics official GitHub documentation. This article will not cover the environment setup process.

This example uses YOLO's official model yolo11n.pt, quantized to a .bin file.

• The provided virtual machine Rosmaster-M1-Ubuntu22.04 already has the OE package horizon_x5_open_explorer_v1.2.8-py310_20240926 and docker(ubuntu 20.04 cpu version)open_explorer_v1.2.8-py310_20240926 and environment configured.
• If you need to use a GPU docker image, please set it up yourself according to the official documentation.

xxxxxxxxxx
D-Robotics OpenExplore(RDK X5, Bayes-e BPU) Version: >= 1.2.8
Ultralytics YOLO Version: 8.3.193
Ubuntu 22.04
Python 3.10

1. Introduction

This article is written based on the D-Robotics official GitHub documentation. The prediction program is developed based on rdk_model_zoo. If you need to study it further, you can search for it yourself.

• Ultralytics YOLO: You Only Look Once
• Introduction to RDK Model Zoo

• Object Detection

2. Preparation

2.1. Environment Setup and Model Training

Note: This operation needs to be performed on your own x86 machine. It is recommended to use a machine with hardware acceleration, such as a GPU supporting CUDA, where torch.cuda.is_available() is True. Ubuntu 22.04 with Python 3.10 is recommended.

Download the ultralytics/ultralytics repository and configure the environment according to the ultralytics official documentation.

For model training, please refer to the ultralytics official documentation. This documentation is maintained by ultralytics and is of very high quality. There are also many reference materials available online. Obtaining a model with pre-trained weights similar to the official one is not difficult. Please note, no program modifications or forward method changes are required during training.

Ultralytics YOLO Official Documentation:

• Quick Start: https://docs.ultralytics.com/quickstart/
• Model Training: https://docs.ultralytics.com/modes/train/

Prepare the Rosmaster-M1-Ubuntu22.04 virtual machine from the learning materials!

2.2. Prepare Calibration Images and pt Model

Download val2017 from the official COCO dataset. Select about 100 images from the validation set. Do not use too many or too few.

[!TIP]

All preparation work is done inside the virtual machine.

If using your own pt model, use your own validation set images!

• Create a folder named coco and put the images in it.

• Place the yolo11n.pt model file in this path.

2.3. Convert to ONNX Model

• Open a terminal and enter the algorithm toolchain docker container.

xxxxxxxxxx
~/rdk.sh

• Navigate to the previous path.

xxxxxxxxxx
cd ~/data/ultralytics/

xxxxxxxxxx
python3 export_monkey_patch.py --pt yolo11n.pt

The output shows that yolo11n.onnx model was successfully exported.

• Check the model with the algorithm toolchain.

xxxxxxxxxx
hb_mapper checker --model-type onnx --march bayes-e --model yolo11n.onnx

[!TIP]

According to the official introduction of this command, we run it to check our model. The system will produce a long output. From the output, we can see that the X5's BPU supports all operators of YOLO, meaning all model computations can be run on the X5's BPU. This check shows six output heads, so we can proceed with model conversion.

2.4. Quantize the Model

• Start model conversion.

xxxxxxxxxx
python3 mapper.py --onnx yolo11n.onnx --cal-images ~/data/test_data/coco

[!TIP]

Fill in the path to your own dataset here.

The conversion time depends on the model size. The CPU version takes longer, please be patient.

The output shows that yolo11n_bayese_640x640_nv12.bin model was successfully exported.

• Check the model with the algorithm toolchain.

xxxxxxxxxx
hb_perf yolo11n_bayese_640x640_nv12.bin

Then you will get a hb_perf_result folder containing the inspection results. We mainly focus on the model structure diagram inside. The image below is for reference.

3. Model Prediction

• (On the robot host) Enter the ultralytics project.

xxxxxxxxxx
cd /home/sunrise/ultralytics/ultralytics

[!TIP]

Place the yolo11n_bayese_640x640_nv12.bin obtained from the virtual machine into the robot host's project directory.

Before running the program, we need to modify the label names and model path in the program.

xxxxxxxxxx
 # Need to modify
 /home/sunrise/ultralytics/ultralytics/rdk_roscv.py

xxxxxxxxxx
coco_names = [
    "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", 
    "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", 
    "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", 
    "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", 
    "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", 
    "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", 
    "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", 
    "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush"
    ]

Start the following program according to your camera model, input in the terminal:

xxxxxxxxxx
# USB camera
 Model Prediction
Image Classification