YOLO Object Recognition and Tracking (No-PTZ)

1. Experiment Objective

Click any object detected by YOLO with the mouse, and the car rotates to keep tracking it.

2. Model and Configuration Check

The YOLO ONNX follow node reads the configuration file by default:

~/microros_ws/src/yolo_onnx_follow_pkg/config/yolo_onnx_follow_config.yaml

Pay special attention to the following parameters:

xxxxxxxxxx
YOLO_ONNX_FOLLOW:
  model_path: "/home/yahboom/MODELS/yolo/yolo11n.onnx"
  image_topic: "/camera/image_raw"
  input_width: 640
  input_height: 640
  conf_threshold: 0.2
  nms_threshold: 0.45
  max_detections: 3

If the model path on your machine is different (you can replace it with your own trained model), modify model_path, or override it at launch time with model_path:=:

xxxxxxxxxx
ros2 launch yolo_onnx_follow_pkg yolo_onnx_follow.launch.py model_path:=/home/yahboom/MODELS/yolo/yolo11n.onnx

Replacing It with a Custom Model: What Needs to Be Changed in the Program

In general, when switching to a standard YOLOv8/YOLOv11 ONNX model, you do not need to modify the Python source code — only the configuration file:

• model_path: change it to the path of your own .onnx model.
• class_names_path: change it to the path of your own class-name file.
• input_width, input_height: change them to the input size used when exporting the ONNX model.
• target_class or target_class_id: if you want to automatically follow a certain class of targets, set it here.
• conf_threshold: lower it when your self-trained model misses detections; raise it when there are many false detections.

There are mainly two cases where you need to modify the source code.

First case: you want to hardcode the class names directly in the program instead of using class_names_path. You can modify COCO_CLASS_NAMES at the top of yolo_onnx_follow_pkg/yolo_onnx_follow_node.py:

xxxxxxxxxx
COCO_CLASS_NAMES = [
    "person",
    "cup",
    "bottle",
]

Using class_names_path is recommended, so you do not have to modify the source code when switching models.

3. Experiment Steps

1. Hardware Preparation

You need to change the vehicle model to the corresponding one in the .bashrc file:

xxxxxxxxxx
gedit .bashrc

After modifying, save and exit, then run the following command to make it take effect:

xxxxxxxxxx
source ~/.bashrc

2. Start the Agent + Camera + Joint Model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

3. Start YOLO Object Recognition and Tracking (Terminal 2)

xxxxxxxxxx
ros2 launch yolo_onnx_follow_pkg yolo_onnx_follow.launch.py

Click the object to follow with the mouse once it is detected:

Operation Method

After the program starts, an OpenCV window pops up showing the detection boxes, class names, tracking IDs, and confidence scores.

• Left-click a target box: select it and start following that target.
• r: clear the current target, stop chassis steering, and re-select a target.
• q or Esc: exit the program; on exit it publishes a stop velocity and returns the PTZ gimbal to its initial angle.

Normal behavior:

• Green detection boxes appear in the window.
• The target box turns yellow after being clicked.
• When the target deviates from the frame center, the PTZ gimbal rotates first to correct it.
• When the PTZ horizontal angle deviation is large, the chassis publishes angular.z to turn in place.
• When the target has been lost for longer than the threshold, the chassis stops steering; if a target of the same class reappears near its previous position within a short time, the program attempts to reacquire it.

At this point, slowly move the detected object, and the PTZ gimbal and the car chassis will track it.

4. Source Code Explanation

4.1 File Structure

xxxxxxxxxx
yolo_onnx_follow_pkg/
├── launch/
│   └── yolo_onnx_follow.launch.py
├── config/
│   └── yolo_onnx_follow_config.yaml
├── yolo_onnx_follow_pkg/
│   ├── yolo_onnx_follow_node.py
│   └── ptz_controller.py
├── setup.py
└── package.xml

4.2 Configuration File

config/yolo_onnx_follow_config.yaml is divided into three main sections:

• SYSTEM: configures the chassis velocity topic, default /cmd_vel.
• PTZ: configures the PTZ angle topic, servo limits, initial angle, publish count, and whether to return to center after stopping.
• YOLO_ONNX_FOLLOW: configures the model path, image topic, detection thresholds, window display, target selection, tracking reacquisition, and control parameters.

Commonly tuned parameters:

4.3 Core Source Code Explanation

The main program file is:

xxxxxxxxxx
yolo_onnx_follow_pkg/yolo_onnx_follow_node.py

Its core workflow can be summarized as:

xxxxxxxxxx
Read configuration -> initialize ROS topics -> load ONNX model -> receive camera images
-> YOLO inference -> NMS filtering -> assign tracking IDs -> select/reacquire target
-> compute PTZ angles -> compute chassis angular velocity -> publish control commands

4.3.1 Detection Data Structure

Detection is used to store the information of each detection box:

xxxxxxxxxx
@dataclass
class Detection:
    bbox: tuple[float, float, float, float]
    score: float
    class_id: int
    class_name: str
    track_id: int = -1

Where:

• bbox: detection box coordinates in the format (x1, y1, x2, y2).
• score: confidence score.
• class_id: class ID.
• class_name: class name.
• track_id: a temporary tracking ID assigned to the target internally by the program.

cx, cy, width, and height are property functions used to quickly get the target center point and box size; the subsequent PTZ control mainly uses cx and cy.

4.3.2 SimpleIouTracker Tracking ID Assignment

SimpleIouTracker is a lightweight local tracker that does not depend on additional tracking libraries. It ensures that the same object in consecutive frames keeps the same ID as much as possible.

The core logic is in update():

xxxxxxxxxx
tracked = self.tracker.update(
    detections, image_width=frame.shape[1], image_height=frame.shape[0]
)

Matching order:

1. First compute the IoU for targets of the same class; when the IoU is greater than track_iou_threshold, the old ID is inherited.
2. If IoU matching fails, fall back to matching by the center-point distance of targets of the same class.
3. If it still cannot be matched, assign a new track_id.

The purpose is to reduce tracking jitter when objects of the same class move quickly, when detection boxes suddenly grow or shrink, or when IDs temporarily jump.

Related configuration:

xxxxxxxxxx
track_iou_threshold: 0.2
track_center_threshold_ratio: 0.12
track_max_lost_frames: 30

The larger track_center_threshold_ratio is, the easier it is to merge nearby targets of the same class into the same ID; however, if several same-class targets are very close together in the frame, a value that is too large may cause incorrect matching.

4.3.3 Model Loading _load_model()

_load_model() is responsible for checking the model path and loading the ONNX model:

xxxxxxxxxx
model_path = str(self.demo_config.get("model_path", "") or "")
if not model_path or not os.path.exists(model_path):
    raise FileNotFoundError(...)
self.net = cv2.dnn.readNetFromONNX(model_path)

If use_cuda is true, the program tries to use the CUDA backend of OpenCV DNN:

xxxxxxxxxx
self.net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
self.net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA_FP16)

If the current OpenCV has no CUDA support, keep this in the configuration:

xxxxxxxxxx
use_cuda: false

4.3.4 Image Callback _image_callback()

After a camera image enters the program, it is first converted to an OpenCV image via cv_bridge:

xxxxxxxxxx
frame = self.bridge.imgmsg_to_cv2(msg, "bgr8")

Then max_inference_rate_hz is used to limit the maximum inference rate, avoiding running inference on every frame and saturating the CPU:

xxxxxxxxxx
if now - self.last_inference_time < 1.0 / max_rate:
    return

The actual recognition and tracking workflow is:

xxxxxxxxxx
detections = self._run_yolo(frame)
tracked = self.tracker.update(detections, image_width=frame.shape[1], image_height=frame.shape[0])
self._update_selected_target(tracked, frame.shape[1], frame.shape[0])

That is, first YOLO detection, then ID assignment, and finally updating the target currently being followed.

4.3.5 YOLO Inference _run_yolo() and Result Parsing _decode_outputs()

_run_yolo() first letterboxes the original image to the model input size:

xxxxxxxxxx
letterboxed, scale, pad_x, pad_y = self._letterbox(frame, input_w, input_h)

Then it constructs a blob and runs the ONNX forward inference:

xxxxxxxxxx
blob = cv2.dnn.blobFromImage(letterboxed, 1.0 / 255.0, (input_w, input_h), swapRB=True, crop=False)
self.net.setInput(blob)
outputs = self.net.forward()

_decode_outputs() does four things:

1. Calls _parse_yolo_row() to parse each output row.
2. Uses conf_threshold to filter out low-confidence results.
3. Uses cv2.dnn.NMSBoxes() to remove duplicate boxes.
4. Sorts by confidence and keeps only max_detections targets.

The current default configuration is:

xxxxxxxxxx
conf_threshold: 0.2
max_detections: 3

Therefore, the window displays at most the 3 targets with the highest confidence, reducing the interference of cluttered detection boxes on selection and tracking.

4.3.6 Target Selection and Reacquisition

There are mainly two entry points for target selection:

• _on_mouse(): after clicking a detection box in the window with the mouse, records the target's track_id and class.
• _select_detection(): finds the target that should currently be followed from the detection results each frame.

After clicking a target, the program saves:

xxxxxxxxxx
self.selected_track_id = int(detection.track_id)
self.selected_class_id = int(detection.class_id)
self.selected_last_bbox = detection.bbox

If the same track_id can still be found in the next frame, tracking continues directly. If the ID jumps, _try_reacquire_selected_detection() is entered to re-find the target based on the same class and the previous frame's position.

The reacquisition score considers three factors:

• The IoU with the previous frame's detection box.
• The distance to the previous frame's target center point.
• The confidence of the current detection box.

Related configuration:

xxxxxxxxxx
reacquire_enable: true
reacquire_iou_threshold: 0.05
 QR Code Tracking
Mediapipe Palm Tracking