Binocular Depth Algorithm

1. Camera Introduction

The D-ROBOTICS binocular IMU camera comes with complete calibration parameters, including binocular calibration parameters, IMU intrinsic parameters, and extrinsic parameters between the binocular camera and IMU. Users can use it directly without additional calibration. Using these parameters, high-precision depth maps can be computed through binocular stereo matching algorithms, enabling real-time 3D environment perception. Meanwhile, the camera data can be applied to open-source Visual-Inertial Odometry (VIO) algorithms such as OpenVINS, which can compute camera pose and trajectory information. Suitable for robot navigation, obstacle avoidance and other applications, providing developers with plug-and-play depth perception and visual-inertial fusion capabilities.

MIPI camera code repository: https://github.com/D-Robotics/hobot_mipi_cam

2. Supported Platforms

3. Preparation

3.1. RDK Platform

1. RDK has been flashed with RDK OS system
2. RDK has successfully installed TogetheROS.Bot
3. Confirm that the PC can access RDK through the network

3.2. System and Package Versions

• If the system image version doesn't meet requirements, please refer to the corresponding section for image flashing
• If the package version doesn't meet requirements, execute the following commands to update:

xxxxxxxxxx
sudo apt update
sudo apt install --only-upgrade tros-humble-hobot-stereonet
sudo apt install --only-upgrade tros-humble-mipi-cam

• If the above commands cannot update to the latest version, you need to modify the apt source file to beta source:

xxxxxxxxxx
# Modify to beta source, execute the following command:
sudo echo 'deb [signed-by=/usr/share/keyrings/sunrise.gpg] http://archive.d-robotics.cc/ubuntu-rdk-x5-beta  jammy main' | sudo tee /etc/apt/sources.list.d/sunrise.list
apt update
​
# If you need to switch back to the official source, execute the following command:
sudo echo 'deb [signed-by=/usr/share/keyrings/sunrise.gpg] http://archive.d-robotics.cc/ubuntu-rdk-x5  jammy main' | sudo tee /etc/apt/sources.list.d/sunrise.list
apt update

4. Camera Connection and Preparation

4.1. Binocular IMU Camera

• Note that the camera has two versions: black metal shell and acrylic shell, both with the same functionality (the cable may vary based on what you receive)

4.2. Hardware Connection

1. Set the switch behind the module to EXT mode, not LPWM mode

2. No other pins need to be connected except the black Dupont wire connecting to RDK X5's 37 pin, which is needed for external trigger control of camera and IMU time synchronization

3. Camera 3Pin pin definition

   

   

4. For the motherboard pin definition, please refer to 3.1.1 Pin Definition and Application

4.3. RDK X5 Configuration

1. RDK X5 needs to be configured to read IMU data. First check if the system version meets the requirements, need 3.4.1 or above:

xxxxxxxxxx
cat /etc/version

2. Run the following command on RDK X5 for configuration:

xxxxxxxxxx
srpi-config

3. After reboot, confirm that the directory /sys/bus/iio/devices/ has iio:device1 and iio:device2, which means the configuration is successful:

xxxxxxxxxx
ll /sys/bus/iio/devices/

5. Binocular Depth Algorithm

5.1 Start Binocular Depth Algorithm

Switch to root user, username: root, password: root

Confirm that the tros-humble-hobot-stereonet package is installed, then you can copy directly, otherwise you need to go back to the environment installation step and reinstall:

xxxxxxxxxx
cp -rv /opt/tros/humble/share/hobot_stereonet/script/run_stereo.sh ./

Execute the following command to start:

xxxxxxxxxx
bash run_stereo.sh --mipi_rotation 0.0

5.2 View Depth Image

• View RGB image and depth image through the web page, enter in browser http://ip:8000 (in the figure, RDK ip is 192.168.1.100):

• View point cloud through rviz2, RDK can directly install rviz2 for viewing. Note that in rviz2, you need to do the following configuration:

xxxxxxxxxx
# Install rviz2
sudo apt install ros-humble-rviz2

xxxxxxxxxx
# Start rviz2
source /opt/tros/humble/setup.bash
rviz2

For the introduction of binocular algorithm and startup command introduction, please refer to Binocular Depth Algorithm | RDK DOC