Instance Segmentation

1. Experiment Objective

Use YOLO for instance segmentation — segment object contours at the pixel level and distinguish different instances of the same class. It supports three input sources: images, videos, and a real-time camera feed.

2. Experiment Principle

1. Glossary

2. Technical Architecture

YOLO instance segmentation starts inference nodes through ROS2 launch files, supporting three input sources:

• Image/Video mode: The node reads local files for offline inference; it does not require micro_ros_agent or the state estimation layer.
• Real-time camera mode: The node subscribes to the /camera/image_raw topic to get real-time camera frames. It requires three processes to work together: micro_ros_agent → state estimation layer → YOLO segmentation node.

Real-time mode data flow:

ESP32 camera → micro_ros_agent → /camera/image_raw → YOLO segmentation node → /yolo_seg/annotated_image + /yolo_seg/status

YOLO inference depends on the ultralytics + PyTorch libraries installed in the current Python environment; the ROS2 and workspace environment must be sourced before starting.

3. Core Topic Quick Reference

ROS2 Internal Topics

3. Experiment Steps

1. Hardware Requirements

2. Start the YOLO Environment

All YOLO ROS2 nodes depend on the ultralytics + PyTorch packages; the environment must be activated before starting:

xxxxxxxxxx
yolo_env

3. Image Mode (Single Image Inference)

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_seg_image.launch.py

4. Video Mode (Frame-by-Frame Inference on a Video File)

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_seg_video.launch.py

5. Real-time Camera Mode

Launch the proxy + camera + joint model (Terminal 1)

xxxxxxxxxx
ros2 launch microros_v2_bringup car_base.launch.py

Terminal 2 — Launch YOLO real-time segmentation:

xxxxxxxxxx
ros2 launch microros_v2_yolo_pkg yolo_seg_realtime.launch.py

6. Operating Instructions

• Image mode: The program reads a single image and overlays a colored semi-transparent mask on the object contours after inference. Each mask is accompanied by a bounding box, class name, and confidence. Press q or ESC to close the window and exit.
• Video mode: Frame-by-frame inference with mask annotation; the masked regions update in real time as objects move. Different instances of different classes are distinguished by different colors. Press q or ESC to stop playback and exit.
• Real-time camera mode: Subscribes to the camera topic for real-time inference; the model name and live FPS are shown at the top of the frame. ByteTrack object tracking is enabled by default (use_track=true) to keep the same target ID across frames. Press q or ESC to exit.

Instance segmentation is finer-grained than object detection — object detection only outputs rectangular boxes, while instance segmentation outputs the precise contour of each object (the edge hugs the object shape). In scenes where objects occlude each other, segmentation clearly outperforms detection boxes.

7. How to Stop

Press Ctrl+C in each terminal, or press q or ESC to exit.

4. Experimental Phenomena

1. Image Mode

• Each object in the frame is precisely covered by a colored semi-transparent mask whose contour hugs the object shape.
• The top-left corner of the frame shows the model name, real-time FPS, detection box count, and mask count.
• Different classes are distinguished by different mask colors.

2. Video Mode

• Every video frame outputs precise object-contour masks, and the covered regions update in near-real time.
• For fast-moving objects, masks may show slight lag or jitter (a CPU-inference limitation).

3. Real-time Camera Mode

• Objects in the camera feed are covered by colored masks; tracking is enabled by default, supporting stable cross-frame IDs.
• When objects occlude each other (e.g. several stacked bottles), segmentation clearly outperforms plain detection boxes.

4. Edge Cases

• Segmentation accuracy for small objects is limited (smaller than 32x32 pixels); valid masks may not be produced.
• Transparent/reflective objects may be segmented poorly, limited by the training data distribution.

5. Program Analysis

Source code paths:

• Launch (real-time): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_seg_realtime.launch.py
• Launch (image): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_seg_image.launch.py
• Launch (video): ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_seg_video.launch.py
• Node source: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_seg_realtime_node.py

1. Core Node Analysis

The real-time segmentation node yolo_seg_realtime_node has an architecture similar to the real-time detection node. The key differences are that it uses a segmentation model (-seg.pt) and publishes segmentation-specific topics (/yolo_seg/annotated_image, /yolo_seg/status). The node subscribes to the raw image topic, calls the YOLO segmentation model for inference, and publishes the annotated results and status.

ROS2 communication setup: It subscribes to the raw image and publishes the annotated image and status; the QoS can be configured through launch parameters (default best_effort). The node uses image_callback to receive sensor_msgs/Image and drops new frames while CPU inference is busy to keep latency low.

​x
# source file: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_seg_realtime_node.py
​
    def __init__(self) -> None:
        super().__init__("yolo_seg_realtime_node")
​
        self.declare_parameter("model_path", DEFAULT_MODEL)
        self.declare_parameter("image_topic", "/camera/image_raw")
        self.declare_parameter("annotated_topic", "/yolo_seg/annotated_image")
        self.declare_parameter("status_topic", "/yolo_seg/status")
        self.declare_parameter("imgsz", 960)
        self.declare_parameter("conf", 0.05)
        self.declare_parameter("iou", 0.7)
        self.declare_parameter("max_det", 1000)
        self.declare_parameter("device", "cpu")
        self.declare_parameter("tracker", "bytetrack.yaml")
        self.declare_parameter("show", True)
        self.declare_parameter("save_video", True)
        self.declare_parameter("output_fps", 20.0)
        self.declare_parameter("window_width", 1280)
        self.declare_parameter("window_height", 720)
        self.declare_parameter("qos_reliability", "best_effort")
        self.declare_parameter("qos_depth", 1)
        self.declare_parameter("process_every_n_frames", 1)
        self.declare_parameter("line_width", 2)
        self.declare_parameter("retina_masks", False)
        self.declare_parameter("publish_status", True)
​
        self.model_path = str(self.get_parameter("model_path").value).strip()
        self.image_topic = str(self.get_parameter("image_topic").value).strip()
        self.annotated_topic = str(self.get_parameter("annotated_topic").value).strip()
        self.status_topic = str(self.get_parameter("status_topic").value).strip()
        self.output_dir = str(self.get_parameter("output_dir").value).strip()
        self.imgsz = int(self.get_parameter("imgsz").value)
        self.conf = float(self.get_parameter("conf").value)
        self.iou = float(self.get_parameter("iou").value)
        self.max_det = int(self.get_parameter("max_det").value)
        self.device = str(self.get_parameter("device").value).strip()
        self.tracker = str(self.get_parameter("tracker").value).strip()
        self.show = _as_bool(self.get_parameter("show").value)
        self.save_video = _as_bool(self.get_parameter("save_video").value)
        self.output_fps = float(self.get_parameter("output_fps").value)
        self.window_width = int(self.get_parameter("window_width").value)
        self.window_height = int(self.get_parameter("window_height").value)
        self.qos_reliability = str(self.get_parameter("qos_reliability").value).strip()
        self.qos_depth = int(self.get_parameter("qos_depth").value)
        self.process_every_n_frames = max(1, int(self.get_parameter("process_every_n_frames").value))
        self.line_width = int(self.get_parameter("line_width").value)
        self.retina_masks = _as_bool(self.get_parameter("retina_masks").value)
        self.publish_status = _as_bool(self.get_parameter("publish_status").value)
​
        self.annotated_pub = self.create_publisher(Image, self.annotated_topic, 10)
        self.status_pub = self.create_publisher(String, self.status_topic, 10) if self.publish_status else None
​
        qos = _make_qos(self.qos_reliability, self.qos_depth)
        self.sub = self.create_subscription(Image, self.image_topic, self.image_callback, qos)

Image callback (image_callback): Receives raw image frames, calls model.track() for inference, obtains masks, draws the annotations on the frame, and publishes the Image result and the optional String status. The processing flag prevents callbacks from stacking up during CPU inference.

xxxxxxxxxx
# source file: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_seg_realtime_node.py
​
    def image_callback(self, msg: Image) -> None:
        self.frame_index += 1
        if self.frame_index % self.process_every_n_frames != 0:
            return
        if self.processing:
            return
​
        self.processing = True
        try:
            frame = self.bridge.imgmsg_to_cv2(msg, desired_encoding="bgr8")
            results = self.model.track(
                source=frame, persist=True, tracker=self.tracker,
                imgsz=self.imgsz, conf=self.conf, iou=self.iou,
                max_det=self.max_det, device=self.device,
                line_width=self.line_width, retina_masks=self.retina_masks,
                verbose=False,
            )
            if not results:
                return
            annotated = results[0].plot()
            out_msg = self.bridge.cv2_to_imgmsg(annotated, encoding="bgr8")
            out_msg.header = msg.header
            self.annotated_pub.publish(out_msg)
        finally:
            self.processing = False

2. Key Parameters

Parameter definition files:

• Launch: ~/microros_ws/src/microros_v2_yolo_pkg/launch/yolo_seg_realtime.launch.py
• Node: ~/microros_ws/src/microros_v2_yolo_pkg/microros_v2_yolo_pkg/yolo_seg_realtime_node.py

3. Node Description

6. Common Problems