Welcome to Jetson Orin NANO Developer Kit repository
4.AI large model voice interaction
3. AI Large Model Voice Interaction
3. AI Large Model Voice Interaction1. Concept Introduction1.1 What is "AI Large Model Voice Interaction"?1.2 Implementation Principles Overview2. Code AnalysisKey Code1. Voice Input Node (largemodel/largemodel/asr.py)2. AI Service and Voice Output Node (largemodel/largemodel/model_service.py)Code Analysis3. Practical Operations3.1 Configuring Voice Interaction3.1.1 Configuring the Platform and Voice Backends (yahboom.yaml)3.1.2 Configuring the Model Interface (large_model_interface.yaml)3.2 Starting and Testing the Feature
1. Concept Introduction
1.1 What is "AI Large Model Voice Interaction"?
In the largemodel project, AI large model voice interaction connects speech recognition (ASR), large language model (LLM), and text-to-speech (TTS) into a complete dialogue system that can listen, speak, and think. ASR, LLM, and TTS can each independently select an online or offline backend.
Users can hold a natural language conversation with the robot by voice; the robot understands the question, generates an answer, and announces it through the speaker. When all-offline backends are selected, no cloud access is required; when any online backend is selected, a network connection and the corresponding service credentials are required.
The core of this feature is the model_service ROS 2 node. It plays the role of the brain and nerve center: it subscribes to the ASR recognition result, calls the LLM to think, and then publishes the LLM's text reply to the TTS node for speech synthesis.
1.2 Implementation Principles Overview
The implementation principle of this feature is a classic data flow pipeline:
Sound -> Text (ASR): The
asrnode continuously listens to ambient sound. Once it detects that the user has finished a sentence, it converts the sentence into text and publishes it to the/asrtopic.Text -> Thinking -> Text (LLM): The
model_servicenode subscribes to the/asrtopic. When it receives the text sent by ASR, it passes the text as a prompt to the locally deployed large language model (such asQwenrunning throughOllama). The LLM generates a text answer according to the context.Text -> Sound (TTS): After obtaining the LLM's answer, the
model_servicenode publishes it to the/tts_topictopic.Sound Playback: The TTS node subscribes to the announcement text, calls the offline or online speech synthesis service according to the configuration, and plays it through the speaker.
This flow forms a complete closed loop of voice input -> text processing -> text output -> voice output.
2. Code Analysis
Key Code
1. Voice Input Node (largemodel/largemodel/asr.py)
x# From largemodel/largemodel/asr.pyclass ASRNode(Node): def __init__(self): # ... self.asr_pub = self.create_publisher(String, "asr", 5) # ...
def kws_handler(self)->None: if self.listen_for_speech(self.mic_index): asr = self.ASR_conversion(self.user_speechdir) if asr != 'error': self.asr_pub_result(asr)
def asr_pub_result(self,asr_result:str)->None: msg=String(data=asr_result) self.asr_pub.publish(msg)2. AI Service and Voice Output Node (largemodel/largemodel/model_service.py)
xxxxxxxxxx# From largemodel/largemodel/model_service.pyclass LargeModelService(Node): def __init__(self): # ... self.asrsub = self.create_subscription(String,'asr', self.asr_callback,1) # ...
def asr_callback(self,msg): """Callback function for handling ASR messages.""" # ... result = self.model_client.infer_with_text(msg.data, message=messages_to_use) self.process_model_result(result)
def process_model_result(self, result, from_seewhat=False): """Process the result returned by the model.""" # ... user_friendly_response = response_json.get("response", "I am processing...") # ... self._safe_play_audio(user_friendly_response) # ... self.execute_tools(tools_list)
def _safe_play_audio(self, text_to_speak: str): """ Synthesizes and plays all non-empty messages only in non-text chat mode. """ if not self.text_chat_mode and text_to_speak: try: self.model_client.voice_synthesis(text_to_speak, self.tts_out_path) self.play_audio_async(self.tts_out_path) except Exception as e: self.get_logger().error(f"Safe audio playback failed: {e}")Code Analysis
The voice interaction feature of the AI large model is implemented cooperatively by two independent ROS nodes, asr.py and model_service.py. They communicate through the ROS topic /asr, forming a complete processing loop.
Voice Input and Publishing (
asr.py):The
ASRNodenode handles voice input. In thekws_handlerfunction, it completes recording and text conversion.After successful conversion, the
asr_pub_resultfunction is called. This function puts the resulting text string into astd_msgs.msg.Stringmessage, and then publishes the message to the/asrtopic through theself.asr_pubpublisher.
Subscription and Response (
model_service.py):During initialization, the
LargeModelServicenode subscribes to the/asrtopic through thecreate_subscriptionmethod and specifiesasr_callbackas its callback function.When
ASRNodepublishes a message on the/asrtopic, the ROS system automatically calls theasr_callbackfunction ofLargeModelServiceand passes the message in as the parametermsg.
Interaction Loop:
After receiving the message, the
asr_callbackfunction extracts the text data (msg.data) and passes it to the large model for inference (self.model_client.infer_with_text).After inference completes, the
process_model_resultfunction is called. This function extracts the text to be used for the reply from the model result.Next, the
_safe_play_audiofunction is called; it uses the TTS feature (self.model_client.voice_synthesis) to synthesize the reply text into audio and play it.At this point, a complete interaction loop from "receiving the user's voice" to "responding to the user by voice" is finished. The two nodes are decoupled through the ROS publish-subscribe pattern:
asr.pyhandles input, andmodel_service.pyhandles processing and output.
3. Practical Operations
3.1 Configuring Voice Interaction
Voice interaction consists of three parts: ASR, LLM, and TTS. ASR and TTS can each select online or offline; the LLM is selected separately by llm_platform. First modify the corresponding keys in the existing configuration files; do not add duplicates to the same parameter block.
3.1.1 Configuring the Platform and Voice Backends (yahboom.yaml)
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlSelect as needed in the existing parameter blocks:
xxxxxxxxxxasr ros__parameters use_oline_asrFalse # False: offline ASR; True: online ASR regional_setting"China"
model_service ros__parameters language'zh' useolinettsFalse # False: offline TTS; True: online TTS llm_platform'ollama' # 'ollama': offline LLM; 'tongyi': online Tongyi regional_setting"China"For example, use False, False, and ollama for a fully offline setup; use True, True, and tongyi for a fully online setup. Mixed selections are also possible, for example an online LLM with offline ASR/TTS. Online ASR or TTS also requires configuring credentials separately as described in articles 1 and 2 of this chapter; merely switching the Boolean values will not automatically generate or fill in service provider keys.
3.1.2 Configuring the Model Interface (large_model_interface.yaml)
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlOnline Tongyi LLM: fill in a valid API Key and an available model; if ASR/TTS are also set to online, add their respective service provider credentials and model parameters as described in articles 1 and 2.
xxxxxxxxxxqianwen_api_key"Replace with your Tongyi API-Key"qianwen_model"qwen-vl-max-latest"Offline Ollama LLM: confirm that the service address is reachable and that the model has been downloaded; for offline ASR/TTS, confirm that the respective model file paths exist.
xxxxxxxxxxollama_host"http://localhost:11434"ollama_model"qwen2.5:7b" # Example; must match the model name in ollama listlocal_asr_model"/home/jetson/yahboom_ws/src/largemodel/MODELS/asr/SenseVoiceSmall"zh_tts_model"/home/jetson/yahboom_ws/src/largemodel/MODELS/tts/zh/zh_CN-huayan-medium.onnx"zh_tts_json"/home/jetson/yahboom_ws/src/largemodel/MODELS/tts/zh/zh_CN-huayan-medium.onnx.json"After saving the configuration, restart the voice program. Online backends require that the Jetson can access the service provider and that the account is available; offline backends require that Ollama and the local voice models are available. Do not disclose real API Keys. Offline inference speed is affected by the computing power of the Jetson mainboard and is usually slower than online models; the Orin Nano 4GB may not be able to run this example smoothly.
3.2 Starting and Testing the Feature
Note: Due to performance limitations, the Jetson Orin Nano 4GB cannot run this case. To experience this feature, please refer to the corresponding section in <Online Large Model (Voice Interaction)>
Start the
largemodelMain Program: Run the following command to start voice interaction:xxxxxxxxxxros2 launch largemodel largemodel_control.launch.pyTest:
Wake-up: Say to the microphone: "Hello, Xiaoya."
Conversation: After the speaker responds, you can say what you want to ask.
Observe the logs: In the terminal running the
launchfile, you should see:The ASR node recognizes your question and prints it out.
The
model_servicenode receives the text, calls the LLM, and prints out the LLM's reply.
Listen to the answer: After a moment, you should hear the robot answer your question from the speaker using the synthesized voice.