yahboom
HomeTutorialsProduct
Solution
Blog
Online Store
About Us
EN/中文
Store
Jetson Orin NANO Developer Kit📖
01.Get started
  • 01. Learning route
  • 02. Get started quickly
  • 03. System Update Notes
  • 04. Disabling automatic updates and troubleshooting issues
  • 05. Jetson Orin Nano official kit screwdriver instructions
02.Jetson basic course
  • 01.Jetson board introduction
  • 02.Write Jetson original system
  • 03.Installing Jetson environment
  • 04.Backup SSD system
  • 05.Restore initial image system
  • 06.Upgrade to SUPER version (Optional)
  • 07.Write SUPER boot (Must read)
  • 08.SSD expansion
  • 09.Jetpack old system upgraded to Yahboom SUPER factory system
  • 10.Write the system onto your own SSD
03.Basic setting course
  • 01.Basic knowledge
  • 02.Network configuration
  • 03.SSH remote login
  • 04.VNC remote control
  • 05.RDP remote control
  • 06.Remote file transfer
  • 07.Jtop tool
  • 08.Exchange space expansion
  • 09.Use VSCode
  • 10.Use VMware
  • 11.Modify screen resolution
04.GPIO control course
  • 01.GPIO description
  • 02.GPIO library installation
  • 03.GPIO reading
  • 04.GPIO output
  • 05.Serial communication
  • 06.I2C communication
05.Vision basic course
  • 01.CSI camera preview
  • 02.USB camera preview
  • 03.Use Jupyter Lab
  • 04.Use JetCam
06.OpenCV basic course
  • 01.Image reading
  • 02.Image saving
  • 03.Image modification
  • 04.Image scaling
  • 05.Image cropping
  • 06.Image translation
  • 07.Image mirroring
  • 08.Image grayscale conversion
  • 09.Image binarization
  • 10.Image edge detection
  • 11.Add line segments
  • 12.Add rectangles
  • 13.Add circular shape
  • 14.Add ellipses
  • 15.Add polygons
  • 16.Add Text
  • 17.Preview USB camera image
  • 18.Preview CSI camera image
07.Vision advanced course
01.Environment construction
  • 01.TensorFlow
  • 02.PyTorch
  • 03.Torchvision
02.DeepStream development
  • 01.DeepStream environment
  • 02.License plate detection and recognition
  • 03.Parallel multi-model inference
  • 04.Object embedding vector generation
  • 05.Pose recognition
  • 06.NV optical flow anomaly detection
  • 07.Single-view 3D tracking
  • 08.Multi-view 3D tracking
03.NVIDIA TAO toolkit application
  • 01.Traffic scene object detection
  • 02.Person object detection
  • 03.Vehicle brand classification
  • 04.Vehicle model classification
  • 05.License plate detection and recognition
  • 06.Person semantic segmentation
  • 07.Retail product detection and recognition
  • 08.Person feature extraction and matching (ReID)
  • 09.Person action and pose classification
  • 10.Transformer-based person detection
  • 11.3D human skeleton detection
  • 12.LiDAR 3D point cloud detection
04.YOLO11 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
05.Mediapipe development
  • 01.Mediapipe environment
  • 02.Hand detection
  • 03.Face detection
  • 04.Face effects
  • 05.3D object recognition
  • 06.Virtual brush
  • 07.Gesture recognition
  • 08.Finger control
  • 09.Posture estimation
  • 10.Face mask
  • 11.Overall detection
  • 12.Green scene removal(Human)
06.YOLO26 development
  • 01.YOLO environment construction
  • 02.CLI use
  • 03.Object detection
  • 04.Instance segmentation
  • 05.Pose estimation
  • 06.Image classification
  • 07.Oriented object detection
  • 08.Model conversion
  • 09.Dataset annotation
  • 10.Model training conversion
  • 11.Model prediction
07.QR code recognition
  • QR code recognition
08.Docker basic course
  • 01.Docker introduction
  • 02.Docker installation
  • 03.Docker command
  • 04.Docker usage
  • 05.Docker interaction
09.ROS2-Jazzy
01.ROS2 basic course
  • 1.Introduction to ROS2
  • 2.ROS2 install Jazzy
  • 3.ROS2 development environment
  • 4.ROS2 workspace
  • 5.ROS2 function package
  • 6.ROS2 node
  • 7.ROS2 topic communication
  • 8.ROS2 service communication
  • 9.ROS2 action communication
  • 10.ROS2 custom interface message
  • 11.ROS2 parameter service case
  • 12.ROS2 meta-function package
  • 13.ROS2 distributed communication
  • 14.ROS2 DDS
  • 15.ROS2 time related API
  • 16.ROS2 common command tools
  • 17.ROS2 rviz2 use
  • 18.ROS2 rqt toolbox
  • 19.ROS2 Launch startup file configuration
  • 20.ROS2 recording and playback tool
  • 21.ROS2 URDF model
  • 22.ROS2 Gazebo simulation platform
  • 23.ROS2 TF2 coordinate transformation
02.ROS2 vision course
  • 01.Camera preview (USB)
  • 02.Camera preview (CSI)
  • 03.Camera calibration (USB)
  • 04.Camera calibration (CSI)
  • 05.AR vision (USB)
  • 06.AR vision (CSI)
10.Large model basic course
01.Offline large model
  • 01.AI large model environment deployment
  • 02.Install large model dialogue platform
  • 03.Meta AI - Llama model
  • 04.Alibaba Cloud - Qwen model
  • 05.Microsoft - Phi-4-mini model
  • 06.DeepSeek - DeepSeek model
  • 07.Alibaba Cloud - Qwen-VL visual multimodal large model
  • 08.Google - Gemma visual multimodal large model
  • 9.Llava visual multimodal large model
  • 10.MiniCPM-V visual multimodal large model
  • 11.High-quality vision-language grounding (LocateAnything)
02.Online large model
  • 01.Tongyi Qianwen large model
  • 02.OpenRouter large model API aggregation platform
11.Large model application practice
01.Voice interaction application
  • 1.Voice interaction hardware connection (Must read)
  • 2.Offline speech to text (ASR)
  • 3.Offline text to speech (TTS)
  • 4.AI large model voice interaction
  • 5.AI large model offline voice assistant
02.Multimodal large model application (Online_Offline large model + Text_Voice interaction)
  • 01.Multimodal visual understand application
  • 02.Multimodal text image application
  • 03.Multimodal video analysis application
  • 04.Multi module visual position application
  • 05.Multimodal table scanning application
  • 06.Multimodal autonomous agent application
12.NVIDIA Isaac ROS course
  • 01.Build Isaac ROS environment
  • 02.Deep segmentation
  • 03.DNN stereoscopic depth
  • 04.Free space segmentation
  • 05.Image distortion processing
  • 06.Image segmentation
  • 07.3D scene reconstruction and mapping
  • 08.Object detection
  • 09.3D attitude estimation
  • 10.Visual SLAM
13.OpenClaw deployment and basic usage
01.OpenClaw deployment and introduction
  • 01.Quick start with the pre-configured environment (Recommended)
  • 02.OpenClaw Deployment
  • 03.OpenClaw WhatsApp Plugin Application
  • 04.OpenClaw WebChat Interaction
  • 05.OpenClaw TUI Interaction
  • 06.OpenClaw Tools Introduction
  • 07.OpenClawGate Gateway User Manual
  • 08.OpenClaw Features Overview
  • 09.OpenClawHub Introduction (Skill Installation)
02.OpenClaw basic application
  • 01.OpenClaw Application - File Management
  • 02.OpenClaw Application - Camera
  • 03.OpenClaw Application - Browser Control
  • 04.OpenClaw Application - Script Execution
  • 05.OpenClaw Application - AI Programming (Peripheral & GPIO Control)
  • 06.OpenClaw Application - Dedicated AI Assistant
  • 07.OpenClaw Web Search Capability Configuration
03.NVIDIA official Agent Skills application
  • 01.AI-assisted development with VS Code or Cursor
  • 02.Jetson device skills
  • 03.Jetson BSP custom agent skills
14.OpenClaw advanced development
01.Preparation Before Use
  • 01.Peripheral Hardware Configuration
  • 02.OpenClaw API-KEY Configuration
  • 03.OpenClaw Model Switching
  • 04.OpenClaw Prompts
  • 05.AI Voice Interaction Configuration
  • 06.Three Dialogue Configuration Tests
02.OpenClaw AI Programming Peripheral Control
  • 1. Servo Control
  • 2. RGB Strip
  • 3. OLED Display
  • 4. Temperature and Humidity Sensor
  • 5. Camera Application
  • 6. AI Script Peripheral Control
03.OpenClaw Peripheral Integration Applications
  • 1. Plant Care Assistant
  • 2. AI Calorie Estimation
  • 3. AI Rock Paper Scissors Game
  • 4. AI Pet
  • 5. AI Weather Station
  • 6. Temperature Humidity Dashboard
  • 7. Scheduled Tasks
Download
  • System_File
  • VMFile_BootFile
  • 3D Model File
  • DataSheet
  • Software_Tools
  • Code
  • Manual
Download ZIP

Welcome to Jetson Orin NANO Developer Kit repository

4.AI large model voice interaction

3. AI Large Model Voice Interaction

3. AI Large Model Voice Interaction1. Concept Introduction1.1 What is "AI Large Model Voice Interaction"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Voice Input Node (largemodel/largemodel/asr.py)2. AI Service and Voice Output Node (largemodel/largemodel/model_service.py)Code Analysis3. Practical Operations3.1 Configuring Voice Interaction3.1.1 Configuring the Platform and Voice Backends (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature

1. Concept Introduction

1.1 What is "AI Large Model Voice Interaction"?

In the largemodel project, AI large model voice interaction connects speech recognition (ASR), large language model (LLM), and text-to-speech (TTS) into a complete dialogue system that can listen, speak, and think. ASR, LLM, and TTS can each independently select an online or offline backend.

Users can hold a natural language conversation with the robot by voice; the robot understands the question, generates an answer, and announces it through the speaker. When all-offline backends are selected, no cloud access is required; when any online backend is selected, a network connection and the corresponding service credentials are required.

The core of this feature is the model_service ROS 2 node. It plays the role of the brain and nerve center: it subscribes to the ASR recognition result, calls the LLM to think, and then publishes the LLM's text reply to the TTS node for speech synthesis.

1.2 Implementation Principles Overview

The implementation principle of this feature is a classic data flow pipeline:

  1. Sound -> Text (ASR): The asr node continuously listens to ambient sound. Once it detects that the user has finished a sentence, it converts the sentence into text and publishes it to the /asr topic.

  2. Text -> Thinking -> Text (LLM): The model_service node subscribes to the /asr topic. When it receives the text sent by ASR, it passes the text as a prompt to the locally deployed large language model (such as Qwen running through Ollama). The LLM generates a text answer according to the context.

  3. Text -> Sound (TTS): After obtaining the LLM's answer, the model_service node publishes it to the /tts_topic topic.

  4. Sound Playback: The TTS node subscribes to the announcement text, calls the offline or online speech synthesis service according to the configuration, and plays it through the speaker.

This flow forms a complete closed loop of voice input -> text processing -> text output -> voice output.


2. Code Analysis

Key Code

1. Voice Input Node (largemodel/largemodel/asr.py)

​x
# From largemodel/largemodel/asr.py
class ASRNode(Node):
    def __init__(self):
        # ...
        self.asr_pub = self.create_publisher(String, "asr", 5)
        # ...

    def kws_handler(self)->None:
        if self.listen_for_speech(self.mic_index):
            asr = self.ASR_conversion(self.user_speechdir)
            if asr != 'error':
                self.asr_pub_result(asr)

    def asr_pub_result(self,asr_result:str)->None:
        msg=String(data=asr_result)
        self.asr_pub.publish(msg)

2. AI Service and Voice Output Node (largemodel/largemodel/model_service.py)

xxxxxxxxxx
# From largemodel/largemodel/model_service.py
class LargeModelService(Node):
    def __init__(self):
        # ...
        self.asrsub = self.create_subscription(String,'asr', self.asr_callback,1)
        # ...

    def asr_callback(self,msg):
        """Callback function for handling ASR messages."""
        # ...
        result = self.model_client.infer_with_text(msg.data, message=messages_to_use)
        self.process_model_result(result)

    def process_model_result(self, result, from_seewhat=False):
        """Process the result returned by the model."""
        # ...
        user_friendly_response = response_json.get("response", "I am processing...")
        # ...
        self._safe_play_audio(user_friendly_response)
        # ...
        self.execute_tools(tools_list)

    def _safe_play_audio(self, text_to_speak: str):
        """
        Synthesizes and plays all non-empty messages only in non-text chat mode.
        """
        if not self.text_chat_mode and text_to_speak:
            try:
                self.model_client.voice_synthesis(text_to_speak, self.tts_out_path)
                self.play_audio_async(self.tts_out_path)
            except Exception as e:
                self.get_logger().error(f"Safe audio playback failed: {e}")

Code Analysis

The voice interaction feature of the AI large model is implemented cooperatively by two independent ROS nodes, asr.py and model_service.py. They communicate through the ROS topic /asr, forming a complete processing loop.

  1. Voice Input and Publishing (asr.py):

    • The ASRNode node handles voice input. In the kws_handler function, it completes recording and text conversion.

    • After successful conversion, the asr_pub_result function is called. This function puts the resulting text string into a std_msgs.msg.String message, and then publishes the message to the /asr topic through the self.asr_pub publisher.

  2. Subscription and Response (model_service.py):

    • During initialization, the LargeModelService node subscribes to the /asr topic through the create_subscription method and specifies asr_callback as its callback function.

    • When ASRNode publishes a message on the /asr topic, the ROS system automatically calls the asr_callback function of LargeModelService and passes the message in as the parameter msg.

  3. Interaction Loop:

    • After receiving the message, the asr_callback function extracts the text data (msg.data) and passes it to the large model for inference (self.model_client.infer_with_text).

    • After inference completes, the process_model_result function is called. This function extracts the text to be used for the reply from the model result.

    • Next, the _safe_play_audio function is called; it uses the TTS feature (self.model_client.voice_synthesis) to synthesize the reply text into audio and play it.

    • At this point, a complete interaction loop from "receiving the user's voice" to "responding to the user by voice" is finished. The two nodes are decoupled through the ROS publish-subscribe pattern: asr.py handles input, and model_service.py handles processing and output.

3. Practical Operations

3.1 Configuring Voice Interaction

Voice interaction consists of three parts: ASR, LLM, and TTS. ASR and TTS can each select online or offline; the LLM is selected separately by llm_platform. First modify the corresponding keys in the existing configuration files; do not add duplicates to the same parameter block.

3.1.1 Configuring the Platform and Voice Backends (yahboom.yaml)

Open the main configuration file:

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/yahboom.yaml

Select as needed in the existing parameter blocks:

xxxxxxxxxx
asr:
  ros__parameters:
    use_oline_asr: False  # False: offline ASR; True: online ASR
    regional_setting: "China"

model_service:
  ros__parameters:
    language: 'zh'
    useolinetts: False    # False: offline TTS; True: online TTS
    llm_platform: 'ollama' # 'ollama': offline LLM; 'tongyi': online Tongyi
    regional_setting: "China"

For example, use False, False, and ollama for a fully offline setup; use True, True, and tongyi for a fully online setup. Mixed selections are also possible, for example an online LLM with offline ASR/TTS. Online ASR or TTS also requires configuring credentials separately as described in articles 1 and 2 of this chapter; merely switching the Boolean values will not automatically generate or fill in service provider keys.

3.1.2 Configuring the Model Interface (large_model_interface.yaml)

xxxxxxxxxx
vim ~/yahboom_ws/src/largemodel/config/large_model_interface.yaml

Online Tongyi LLM: fill in a valid API Key and an available model; if ASR/TTS are also set to online, add their respective service provider credentials and model parameters as described in articles 1 and 2.

xxxxxxxxxx
qianwen_api_key: "Replace with your Tongyi API-Key"
qianwen_model: "qwen-vl-max-latest"

Offline Ollama LLM: confirm that the service address is reachable and that the model has been downloaded; for offline ASR/TTS, confirm that the respective model file paths exist.

xxxxxxxxxx
ollama_host: "http://localhost:11434"
ollama_model: "qwen2.5:7b"  # Example; must match the model name in ollama list
local_asr_model: "/home/jetson/yahboom_ws/src/largemodel/MODELS/asr/SenseVoiceSmall"
zh_tts_model: "/home/jetson/yahboom_ws/src/largemodel/MODELS/tts/zh/zh_CN-huayan-medium.onnx"
zh_tts_json: "/home/jetson/yahboom_ws/src/largemodel/MODELS/tts/zh/zh_CN-huayan-medium.onnx.json"

After saving the configuration, restart the voice program. Online backends require that the Jetson can access the service provider and that the account is available; offline backends require that Ollama and the local voice models are available. Do not disclose real API Keys. Offline inference speed is affected by the computing power of the Jetson mainboard and is usually slower than online models; the Orin Nano 4GB may not be able to run this example smoothly.

3.2 Starting and Testing the Feature

Note: Due to performance limitations, the Jetson Orin Nano 4GB cannot run this case. To experience this feature, please refer to the corresponding section in <Online Large Model (Voice Interaction)>

  1. Start the largemodel Main Program: Run the following command to start voice interaction:

    xxxxxxxxxx
    ros2 launch largemodel largemodel_control.launch.py
  2. Test:

    • Wake-up: Say to the microphone: "Hello, Xiaoya."

    • Conversation: After the speaker responds, you can say what you want to ask.

    • Observe the logs: In the terminal running the launch file, you should see:

      1. The ASR node recognizes your question and prints it out.

      2. The model_service node receives the text, calls the LLM, and prints out the LLM's reply.

    • Listen to the answer: After a moment, you should hear the robot answer your question from the speaker using the synthesized voice.

Offline text to speech (TTS)
AI large model offline voice assistant

COMPANY

  • About us
  • Team Style
  • Blog

LEARNING & EXPLORATION

  • Tutorials
  • Edge AI hardware
  • Educational robots
  • Embedded intelligent hardware

WHERE TO BUY

  • Yahboom Online Store
  • Amazon
  • Alibaba
  • AliExpress
  • Mercado

TECHNOLOGY SUPPORT

  • Email: support@yahboom.com

BE A DISTRIBUTOR

  • Email: sales@yahboom.com
QR Code

Copyright © 2017-2026 Yahboom. All Rights Reserved.

EN/中文