yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

05.Multimodal table scanning application

Multimodal Table Scanning Application

Multimodal Table Scanning Application1. Concept Introduction1.1 What Is "Multimodal Table Scanning"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The table content is not fully recognized, or typos appear.Problem 2: The message "Table file not found" is displayed.

Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.


1. Concept Introduction

1.1 What Is "Multimodal Table Scanning"?

Multimodal table scanning is a technology that uses image processing and artificial intelligence techniques to recognize and extract table information from images or PDF documents. It focuses not only on visual table structure recognition, but also combines multimodal data such as text content and layout information to enhance the understanding of the table. Large language models (LLMs) then provide powerful semantic analysis capability in understanding this extracted information; the two complement each other and jointly improve the intelligence level of document processing.

1.2 Implementation Principles Overview

  1. Table Detection and Content Recognition

    • Computer vision technology is used to locate the table in the document, and OCR technology is used to convert the text inside the table into an editable format.

    • Deep learning methods are used to parse the table structure (row and column division, merged cells, and so on) and generate a structured data representation.

  2. Multimodal Fusion

    • Visual information (such as the table layout), text (OCR results), and possibly existing metadata (such as file type and source) are integrated to form a comprehensive data view.

    • A specially designed multimodal model (such as LayoutLM) is used to process these different types of data simultaneously, so as to understand the table content and its contextual relationships more accurately.

2. Code Analysis

Key Code

1. Tool Layer Entry (largemodel/utils/tools_manager.py)

The scan_table function in this file defines the execution flow of the tool, in particular how it builds a prompt requiring Markdown format output.

​x
# From largemodel/utils/tools_manager.py
class ToolsManager:
    # ...
    def scan_table(self, args):
        """
        Scan a table from an image and save the content as a Markdown file.

        :param args: Arguments containing the image path.
        :return: Dictionary with file path and content.
        """
        self.node.get_logger().info(f"Executing scan_table() tool with args: {args}")
        try:
            image_path = args.get("image_path")
            # ... (path check and fallback)

            # Construct a prompt asking the large model to recognize the table and return it in Markdown format.
            if self.node.language == 'zh':
                prompt = "Please carefully analyze this image, identify the table in it, and return its content in Markdown format."
            else:
                prompt = "Please carefully analyze this image, identify the table within it, and return its content in Markdown format."

            result = self.node.model_client.infer_with_image(image_path, prompt)
            
            # ... (extract the Markdown text from the result)

            # Save the recognized content to a Markdown file.
            md_file_path = os.path.join(self.node.pkg_path, "resources_file", "scanned_tables", f"table_{timestamp}.md")
            with open(md_file_path, 'w', encoding='utf-8') as f:
                f.write(table_content)

            return {
                "file_path": md_file_path,
                "table_content": table_content
            }
        # ... (error handling)

2. Model Interface Layer (largemodel/utils/large_model_interface.py)

The infer_with_image function in this file is the unified entry point for all image-related tasks.

xxxxxxxxxx
# From largemodel/utils/large_model_interface.py

class model_interface:
    # ...
    def infer_with_image(self, image_path, text=None, message=None):
        """Unified image inference interface."""
        # ... (prepare messages)
        try:
            # Decide which specific implementation to call based on the value of self.llm_platform
            if self.llm_platform == 'ollama':
                response_content = self.ollama_infer(self.messages, image_path=image_path)
            elif self.llm_platform == 'tongyi':
                # ... logic for calling the Tongyi model
                pass
            # ... (logic for other platforms)
        # ...
        return {'response': response_content, 'messages': self.messages.copy()}

Code Analysis

The table scanning feature is a typical application of converting unstructured image data into structured text data. Its core technology is likewise guiding model behavior through prompt engineering.

  1. Tool Layer (tools_manager.py):

    • The scan_table function is the business process controller of this feature. It receives an image containing a table as input.

    • The most critical operation of this function is to build a prompt with a clear objective. This prompt directly instructs the large model to perform two tasks: 1. Identify the table in the image. 2. Return the recognized content in Markdown format. This mandatory requirement on the output format is the key to realizing the conversion from unstructured to structured data.

    • After building the prompt, it calls the infer_with_image method of the model interface layer, passing the image together with this formatting instruction.

    • After obtaining the returned Markdown text from the model interface layer, it performs a file operation: writing this text content into a new .md file.

    • Finally, it returns structured data containing the new file path and the table content.

  2. Model Interface Layer (large_model_interface.py):

    • The infer_with_image function continues to serve as the unified "dispatching center". It receives the image and prompt from scan_table and dispatches the task to the correct backend model implementation according to the current system configuration (self.llm_platform).

    • Regardless of the backend model, the task of this layer is to handle the communication details with the specific platform, ensure that the image and text data are sent correctly, and then return the plain text returned by the model (in this case, text in Markdown format) to the tool layer.

In summary, the general flow of table scanning is: ToolsManager receives the image and builds an instruction to "convert the table in this image to Markdown" -> ToolsManager calls the model interface -> model_interface packages the image and the instruction and sends them to the corresponding model platform according to the configuration -> the model returns text in Markdown format -> model_interface returns the text to ToolsManager -> ToolsManager saves the text as a .md file and returns the result. This flow demonstrates how to use the format-following capability of a large model to use it as a powerful OCR (optical character recognition) and data structuring tool.

3. Practical Operations

3.1 Configuring the Large Model

This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.

3.1.1 Configuring the LLM Platform (yahboom.yaml)

Open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

Set llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:

xxxxxxxxxx
model_service:
  ros__parameters:
    language: 'zh'
    llm_platform: 'ollama'  # Offline large model; change to 'tongyi' when using an online model
    regional_setting: "China"
  • Offline Ollama: set to llm_platform: 'ollama'.

  • Online Tongyi Qianwen: set to llm_platform: 'tongyi'.

If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.

3.1.2 Configuring the Model Interface (large_model_interface.yaml)

Open the model interface configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

According to the backend selected in the previous step, fill in only the corresponding group of parameters.

Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:

xxxxxxxxxx
qianwen_api_key: "Replace with your API-Key"
qianwen_model: "qwen-vl-max-latest"

Replace the placeholder with your own API Key. Table scanning requires the model to be able to read the text and table structure in the image and organize them into Markdown as prompted. Do not disclose real API Keys.

Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "llava"  # Replace with the model name in ollama list

The model name must be consistent with the output of ollama list, and the model needs to support image input.

After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.

The table recognition speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed is also related to the model size and the image clarity.

3.2 Starting and Testing the Feature (Text Mode)

  1. Prepare the table image file:

    Place a table image file to be tested in the following path: /home/jetson/yahboom_ws/src/largemodel/resources_file/scan_table

    Then name the image test_table.jpg

  2. Start the largemodel main program (text mode):

    Open a terminal and run the following command:

    xxxxxxxxxx
    ros2 launch largemodel largemodel_control.launch.py text_chat_mode:=true
  3. Send a text command: Open another terminal again and run the following command,

    xxxxxxxxxx
    ros2 run text_chat text_chat

    Then start typing the text: "Analyze the table".

  4. Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the scan_table tool, that scan_table has finished executing, and that the scanned information was saved to a document.

    We can find this document in the path ~/yahboom_ws/src/largemodel/resources_file/scan_table.

3.3 Voice Interaction

Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:

xxxxxxxxxx
model_service:
  ros__parameters:
    use_oline_asr: False  # True: online ASR; False: offline ASR
    useolinetts: False    # True: online TTS; False: offline TTS

Complete the corresponding configuration according to the ASR/TTS tutorials. Confirm that the table image is in the directory specified above and named test_table.jpg, then start:

xxxxxxxxxx
ros2 launch largemodel largemodel_control.launch.py

After waking the device, say "Analyze the table". Check the ASR recognized text, the scan_table tool call log, and the saved Markdown file, and confirm the speaker announcement. Tool calling depends on the current program version and the model capability; if the tool is not executed, first verify in text mode.

4. Common Problems and Solutions

Problem 1: The table content is not fully recognized, or typos appear.

Solution:

  1. Image quality: Make sure the input table image is clear, not distorted, and evenly lit. Low-quality images are the most common cause of recognition errors.

  2. Model selection: Models with more parameters generally achieve better recognition results; try switching to a more powerful recognition model.

Problem 2: The message "Table file not found" is displayed.

Solution:

  1. Check the path: Make sure you have placed the table image file in the specified path, that it is named test_table.jpg, and that you have permission to read the file.

  2. Image format: Make sure the image file is not damaged and is in .jpg format.

Multi module visual position application
Multimodal autonomous agent application

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文