Welcome to Jetson Orin NANO Developer Kit repository
05.Multimodal table scanning application
Multimodal Table Scanning Application
Multimodal Table Scanning Application1. Concept Introduction1.1 What Is "Multimodal Table Scanning"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Tool Layer Entry (largemodel/utils/tools_manager.py)2. Model Interface Layer (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring the Large Model3.1.1 Configuring the LLM Platform (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature (Text Mode)3.3 Voice Interaction4. Common Problems and SolutionsProblem 1: The table content is not fully recognized, or typos appear.Problem 2: The message "Table file not found" is displayed.
Hardware requirement: a 4GB Jetson cannot run this case; use a Jetson with 8GB or more.
1. Concept Introduction
1.1 What Is "Multimodal Table Scanning"?
Multimodal table scanning is a technology that uses image processing and artificial intelligence techniques to recognize and extract table information from images or PDF documents. It focuses not only on visual table structure recognition, but also combines multimodal data such as text content and layout information to enhance the understanding of the table. Large language models (LLMs) then provide powerful semantic analysis capability in understanding this extracted information; the two complement each other and jointly improve the intelligence level of document processing.
1.2 Implementation Principles Overview
Table Detection and Content Recognition
Computer vision technology is used to locate the table in the document, and OCR technology is used to convert the text inside the table into an editable format.
Deep learning methods are used to parse the table structure (row and column division, merged cells, and so on) and generate a structured data representation.
Multimodal Fusion
Visual information (such as the table layout), text (OCR results), and possibly existing metadata (such as file type and source) are integrated to form a comprehensive data view.
A specially designed multimodal model (such as LayoutLM) is used to process these different types of data simultaneously, so as to understand the table content and its contextual relationships more accurately.
2. Code Analysis
Key Code
1. Tool Layer Entry (largemodel/utils/tools_manager.py)
The scan_table function in this file defines the execution flow of the tool, in particular how it builds a prompt requiring Markdown format output.
x# From largemodel/utils/tools_manager.pyclass ToolsManager: # ... def scan_table(self, args): """ Scan a table from an image and save the content as a Markdown file.
:param args: Arguments containing the image path. :return: Dictionary with file path and content. """ self.node.get_logger().info(f"Executing scan_table() tool with args: {args}") try: image_path = args.get("image_path") # ... (path check and fallback)
# Construct a prompt asking the large model to recognize the table and return it in Markdown format. if self.node.language == 'zh': prompt = "Please carefully analyze this image, identify the table in it, and return its content in Markdown format." else: prompt = "Please carefully analyze this image, identify the table within it, and return its content in Markdown format."
result = self.node.model_client.infer_with_image(image_path, prompt) # ... (extract the Markdown text from the result)
# Save the recognized content to a Markdown file. md_file_path = os.path.join(self.node.pkg_path, "resources_file", "scanned_tables", f"table_{timestamp}.md") with open(md_file_path, 'w', encoding='utf-8') as f: f.write(table_content)
return { "file_path": md_file_path, "table_content": table_content } # ... (error handling)2. Model Interface Layer (largemodel/utils/large_model_interface.py)
The infer_with_image function in this file is the unified entry point for all image-related tasks.
xxxxxxxxxx# From largemodel/utils/large_model_interface.py
class model_interface: # ... def infer_with_image(self, image_path, text=None, message=None): """Unified image inference interface.""" # ... (prepare messages) try: # Decide which specific implementation to call based on the value of self.llm_platform if self.llm_platform == 'ollama': response_content = self.ollama_infer(self.messages, image_path=image_path) elif self.llm_platform == 'tongyi': # ... logic for calling the Tongyi model pass # ... (logic for other platforms) # ... return {'response': response_content, 'messages': self.messages.copy()}Code Analysis
The table scanning feature is a typical application of converting unstructured image data into structured text data. Its core technology is likewise guiding model behavior through prompt engineering.
Tool Layer (
tools_manager.py):The
scan_tablefunction is the business process controller of this feature. It receives an image containing a table as input.The most critical operation of this function is to build a prompt with a clear objective. This prompt directly instructs the large model to perform two tasks: 1. Identify the table in the image. 2. Return the recognized content in Markdown format. This mandatory requirement on the output format is the key to realizing the conversion from unstructured to structured data.
After building the prompt, it calls the
infer_with_imagemethod of the model interface layer, passing the image together with this formatting instruction.After obtaining the returned Markdown text from the model interface layer, it performs a file operation: writing this text content into a new
.mdfile.Finally, it returns structured data containing the new file path and the table content.
Model Interface Layer (
large_model_interface.py):The
infer_with_imagefunction continues to serve as the unified "dispatching center". It receives the image and prompt fromscan_tableand dispatches the task to the correct backend model implementation according to the current system configuration (self.llm_platform).Regardless of the backend model, the task of this layer is to handle the communication details with the specific platform, ensure that the image and text data are sent correctly, and then return the plain text returned by the model (in this case, text in Markdown format) to the tool layer.
In summary, the general flow of table scanning is: ToolsManager receives the image and builds an instruction to "convert the table in this image to Markdown" -> ToolsManager calls the model interface -> model_interface packages the image and the instruction and sends them to the corresponding model platform according to the configuration -> the model returns text in Markdown format -> model_interface returns the text to ToolsManager -> ToolsManager saves the text as a .md file and returns the result. This flow demonstrates how to use the format-following capability of a large model to use it as a powerful OCR (optical character recognition) and data structuring tool.
3. Practical Operations
3.1 Configuring the Large Model
This section selects the backend through the platform option in yahboom.yaml and fills in the corresponding model parameters in large_model_interface.yaml. The online example uses Tongyi Qianwen; the offline example uses the local Ollama.
3.1.1 Configuring the LLM Platform (yahboom.yaml)
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlSet llm_platform in the existing model_service.ros__parameters parameter block. Only one of the two backends needs to be selected:
xxxxxxxxxxmodel_service ros__parameters language'zh' llm_platform'ollama' # Offline large model; change to 'tongyi' when using an online model regional_setting"China"Offline Ollama: set to
llm_platform: 'ollama'.Online Tongyi Qianwen: set to
llm_platform: 'tongyi'.
If these parameters already exist in the file, only modify the corresponding values and keep the other configurations; do not add duplicate YAML keys with the same name.
3.1.2 Configuring the Model Interface (large_model_interface.yaml)
Open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlAccording to the backend selected in the previous step, fill in only the corresponding group of parameters.
Online Tongyi Qianwen: First apply for an API Key according to the online large model basic tutorial and confirm that the account has available quota; then fill in the Key and a model that supports image input:
xxxxxxxxxxqianwen_api_key"Replace with your API-Key"qianwen_model"qwen-vl-max-latest"Replace the placeholder with your own API Key. Table scanning requires the model to be able to read the text and table structure in the image and organize them into Markdown as prompted. Do not disclose real API Keys.
Offline Ollama: Make sure the Ollama service has started and that a model supporting image input has been downloaded; then fill in the service address and the local model name:
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"llava" # Replace with the model name in ollama listThe model name must be consistent with the output of ollama list, and the model needs to support image input.
After saving both configuration files, restart largemodel for the settings to take effect. Online mode requires that the Jetson can access the Tongyi service; offline mode requires that the local Ollama service is available.
The table recognition speed of an offline large model is affected by the computing power of the Jetson mainboard and is usually slower than an online large model; the actual speed is also related to the model size and the image clarity.
3.2 Starting and Testing the Feature (Text Mode)
Prepare the table image file:
Place a table image file to be tested in the following path:
/home/jetson/yahboom_ws/src/largemodel/resources_file/scan_tableThen name the image
test_table.jpgStart the
largemodelmain program (text mode):Open a terminal and run the following command:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.py text_chat_mode:=trueSend a text command: Open another terminal again and run the following command,
xxxxxxxxxxros2 run text_chat text_chatThen start typing the text: "Analyze the table".
Observe the result: In the first terminal running the main program, you will see log output indicating that the system received the command, called the
scan_tabletool, that scan_table has finished executing, and that the scanned information was saved to a document.We can find this document in the path ~/yahboom_ws/src/largemodel/resources_file/scan_table.
3.3 Voice Interaction
Voice mode can use an online or offline LLM. In yahboom.yaml, set llm_platform to tongyi for online and ollama for offline; ASR/TTS are selected independently:
xxxxxxxxxxmodel_service ros__parameters use_oline_asrFalse # True: online ASR; False: offline ASR useolinettsFalse # True: online TTS; False: offline TTSComplete the corresponding configuration according to the ASR/TTS tutorials. Confirm that the table image is in the directory specified above and named test_table.jpg, then start:
xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyAfter waking the device, say "Analyze the table". Check the ASR recognized text, the scan_table tool call log, and the saved Markdown file, and confirm the speaker announcement. Tool calling depends on the current program version and the model capability; if the tool is not executed, first verify in text mode.
4. Common Problems and Solutions
Problem 1: The table content is not fully recognized, or typos appear.
Solution:
Image quality: Make sure the input table image is clear, not distorted, and evenly lit. Low-quality images are the most common cause of recognition errors.
Model selection: Models with more parameters generally achieve better recognition results; try switching to a more powerful recognition model.
Problem 2: The message "Table file not found" is displayed.
Solution:
Check the path: Make sure you have placed the table image file in the specified path, that it is named
test_table.jpg, and that you have permission to read the file.Image format: Make sure the image file is not damaged and is in .jpg format.