Object Detection

1. Experiment Objective

Use YOLO for object detection — recognize objects in the frame and annotate bounding boxes and classes. It supports three input sources: images, videos, and a real-time camera feed.

2. Experiment Principle

1. Glossary

2. Technical Architecture

YOLO vision recognition starts inference nodes through ROS2 launch files, supporting three input sources:

• Image/Video mode: The node reads local files for offline inference; it does not require micro_ros_agent or the state estimation layer. You can start the YOLO launch directly.
• Real-time camera mode: The node subscribes to the /camera/image_raw topic to get real-time camera frames. It requires three processes to work together: micro_ros_agent (firmware communication) → state estimation layer (camera data publishing) → YOLO detection node.

Real-time mode data flow:

ESP32 camera → micro_ros_agent → /camera/image_raw → YOLO detection node → /yolo/detect/image

YOLO inference depends on the ultralytics + PyTorch libraries installed in the current Python environment; the ROS2 and workspace environment must be sourced before starting.

3. Core Topic Quick Reference

ROS2 Internal Topics

3. Experiment Steps

1. Hardware Requirements

2. Start the YOLO Environment

All YOLO ROS2 nodes depend on the ultralytics + PyTorch packages in the current Python environment. Before starting, the ROS2 and workspace environment must be sourced:

xxxxxxxxxx
yolo_env

3. Image Mode (Single Image Inference)

Image mode does not depend on the robot hardware or micro_ros_agent, so inference can be started directly:

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_image_detect.launch.py

4. Video Mode (Frame-by-Frame Inference on a Video File)

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_video_detect.launch.py

5. Real-time Camera Mode

Launch the proxy + camera + joint model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

Terminal 2 — Launch YOLO real-time detection:

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_realtime_detect.launch.py

6. Operating Instructions

After startup, the YOLO model runs inference on the input frames, and the detection results are annotated on the frame as bounding boxes. Each box is accompanied by a class name and confidence score.

• Image mode: The program reads a single image and shows the annotated results after inference. Press q or ESC to close the window and exit.
• Video mode: The program reads a video file frame by frame, runs inference independently on each frame, and shows the annotated results. Press q or ESC to stop playback and exit.
• Real-time camera mode: The program subscribes to the /camera/image_raw topic to receive real-time camera frames for inference. Press q or ESC to exit.
• Common interaction: All modes support pressing q to exit; the first model load takes a few seconds, and subsequent inference runs in real time.

7. How to Stop

Press Ctrl+C in each terminal, or press q or ESC to exit.

4. Experimental Phenomena

1. Image Mode

• After a single image is fed in, the objects in the frame are annotated with colored rectangular boxes.
• Above each box, the class name (e.g. person, car, dog) and confidence (e.g. 0.85) are displayed.
• The result window stays open; press q to close it.

2. Video Mode

• Frame-by-frame inference; the detection results are displayed in real time on each frame.
• Detection boxes keep updating as objects move, which is suitable for observing dynamic scenes.

3. Real-time Camera Mode

• The real-time camera feed is annotated; smoothness depends on CPU performance and model size.
• Tracking mode is supported (use_tracking:=true), which assigns a stable ID to the same object.

4. Edge Cases

• When there are no objects in the scene, the frame stays as-is without any boxes.
• Low-confidence objects can be shown by lowering the conf threshold, but this also increases false detections.

5. Program Analysis

Source code paths:

• Launch (real-time): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_realtime_detect.launch.py
• Launch (image): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_image_detect.launch.py
• Launch (video): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_video_detect.launch.py
• Node source: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_realtime_detect_node.py

1. Core Node Analysis

The real-time detection node YoloRealtimeDetectNode is the ROS2 entry point for the object detection feature. It subscribes to the raw camera image topic, calls the YOLO model for inference, and publishes the annotated results.

Constructor (__init__): Declares all parameters, initializes CvBridge, loads the model, and sets up ROS2 communication.

​x
# source file: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_realtime_detect_node.py
​
class YoloRealtimeDetectNode(Node):
    def __init__(self):
        super().__init__("yolo_realtime_detect_node")
​
        self.declare_parameter("model_name", "yolo11n")
        self.declare_parameter("model_path", "")
        self.declare_parameter("image_topic", "/camera/image_raw")
        self.declare_parameter("output_topic", "/yolo/detect/image")
        self.declare_parameter("conf", 0.05)
        self.declare_parameter("iou", 0.7)
        self.declare_parameter("imgsz", 960)
        self.declare_parameter("device", "cpu")
        self.declare_parameter("max_det", 1000)
        self.declare_parameter("use_tracking", True)
        self.declare_parameter("tracker", "bytetrack.yaml")
        self.declare_parameter("persist", True)
        self.declare_parameter("process_every_n_frames", 1)
        self.declare_parameter("show_window", True)
        self.declare_parameter("window_name", "YOLO Realtime Tracking")
        self.declare_parameter("display_width", 960)
        self.declare_parameter("display_height", 540)
        self.declare_parameter("save_video", False)
        self.declare_parameter("output_dir", str(OUTPUT_VIDEO_DIR))
        self.declare_parameter("output_video_name", "realtime_yolo_track.mp4")
        self.declare_parameter("save_video_fps", 15.0)
        self.declare_parameter("fourcc", "mp4v")
​
        self.bridge = CvBridge()
        self.model = None
        self.frame_index = 0
        self.processed_count = 0
        self.start_time = time.time()
        self.video_writer = None
        self.video_output_path = None
​
        self._load_parameters()
        self._load_model()
        self._setup_ros_io()

Image callback (image_callback): Receives the raw image, runs YOLO inference, and publishes the annotated result. Core flow: Image → cv2 image → YOLO inference → annotation drawing → publish Image message. Supports frame-skipping (process_every_n_frames) and optional video saving.

xxxxxxxxxx
# source file: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_realtime_detect_node.py
​
    def image_callback(self, msg: Image):
        self.frame_index += 1
​
        if self.frame_index % self.process_every_n_frames != 0:
            return
​
        try:
            frame = self.bridge.imgmsg_to_cv2(msg, desired_encoding="bgr8")
        except Exception as exc:
            self.get_logger().error(f"cv_bridge convert failed: {exc}")
            return
​
        try:
            results = self._run_yolo(frame)
            if results:
                annotated = results[0].plot()
            else:
                annotated = frame
        except Exception as exc:
            self.get_logger().error(f"YOLO inference failed: {exc}")
            return
​
        try:
            out_msg = self.bridge.cv2_to_imgmsg(annotated, encoding="bgr8")
            out_msg.header = msg.header
            self.publisher.publish(out_msg)
        except Exception as exc:
            self.get_logger().error(f"publish result image failed: {exc}")
​
        self._init_video_writer_if_needed(annotated)
        if self.video_writer is not None:
            self.video_writer.write(annotated)
​
        if self.show_window:
            if self.display_width > 0 and self.display_height > 0:
                show_img = cv2.resize(annotated, (self.display_width, self.display_height))
            else:
                show_img = annotated
            cv2.imshow(self.window_name, show_img)
            key = cv2.waitKey(1) & 0xFF
            if key in (ord("q"), ord("Q"), 27):
                self.get_logger().info("q/ESC pressed, shutting down.")
                os._exit(0)
​
        self.processed_count += 1

YOLO inference (_run_yolo): Selects model.track() (object tracking) or model.predict() (object detection) based on the use_tracking parameter. In tracking mode, ByteTrack is used to keep cross-frame ID consistency.

xxxxxxxxxx
# source file: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_realtime_detect_node.py
​
    def _run_yolo(self, frame):
        if self.use_tracking:
            return self.model.track(
                source=frame,
                persist=self.persist,
                tracker=self.tracker,
                conf=self.conf,
                iou=self.iou,
                imgsz=self.imgsz,
                device=self.device,
                max_det=self.max_det,
                verbose=False,
            )
        return self.model.predict(
            source=frame,
            conf=self.conf,
            iou=self.iou,
            imgsz=self.imgsz,
            device=self.device,
            max_det=self.max_det,
            verbose=False,
        )

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_realtime_detect.launch.py
• Node: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_realtime_detect_node.py

Note: The launch file defaults (e.g. conf:=0.12) override the node source defaults (e.g. conf=0.05). At runtime, the launch defaults or the values overridden on the command line take effect.

3. Node Description

6. Common Problems