Welcome to Jetson Orin NANO Developer Kit repository
2.Offline speech to text (ASR)
1. Speech-to-Text (ASR, Online/Offline)
1. Speech-to-Text (ASR, Online/Offline)1. Concept Introduction1.1 What is "ASR"?1.2 Implementation Principles1. Acoustic Model2. Language Model3. Pronunciation Dictionary4. Decoder5. End-to-End ASR2. Code AnalysisKey Code1. Speech Processing and Recognition Core (largemodel/largemodel/asr.py)2. VAD Smart Recording (largemodel/largemodel/asr.py)Code Analysis3. Practical Operations3.1 Configuring ASR3.1.1 Configuring Offline ASR3.1.2 Configuring Online ASR3.2 Starting and Testing the Feature
1. Concept Introduction
1.1 What is "ASR"?
ASR (Automatic Speech Recognition) is a technology that converts human speech signals into text. It is widely used in intelligent assistants, voice command control, telephone customer service automation, real-time subtitle generation, and other fields. The goal of ASR is to let machines "understand" human speech and convert it into a form that computers can process and understand.
1.2 Implementation Principles
The implementation of an ASR system mainly relies on the following key technical components:
1. Acoustic Model
The acoustic model is responsible for converting the input audio signal into phonemes or subword units. This usually involves a feature extraction step, such as using Mel-Frequency Cepstral Coefficients (MFCCs) or filter banks to represent the audio signal.
These features are then fed into a deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), or the more advanced Transformer architecture for training, so as to learn how to map audio features to the corresponding phonemes or text.
2. Language Model
A language model is used to predict the most likely next word given the preceding context, thereby helping improve recognition accuracy. It is trained on a large amount of text data and knows which word sequences are more likely to appear.
Common language models include n-gram models, RNN-based LMs, and the more recently popular Transformer-based LMs.
3. Pronunciation Dictionary
A pronunciation dictionary provides the mapping between words and their corresponding pronunciations. This is crucial for connecting the acoustic model and the language model, because it allows the system to understand and match the sounds it hears according to known pronunciation rules.
4. Decoder
The decoder's task is to find the most likely word sequence as output given the acoustic model, language model, and pronunciation dictionary. This process usually involves complex search algorithms, such as the Viterbi algorithm or graph-based search methods, to find the optimal path.
5. End-to-End ASR
With the development of deep learning, end-to-end ASR systems have emerged. These systems attempt to learn text output directly from raw audio signals, without the explicit separate design of acoustic models, pronunciation dictionaries, and language models. Such systems are often based on sequence-to-sequence (Seq2Seq) frameworks, for example using attention mechanisms or the Transformer architecture, greatly simplifying the complexity of traditional ASR systems.
In general, modern ASR systems achieve efficient and accurate human speech-to-text capability by combining the components above and training with large-scale datasets and powerful computing resources. As technology advances, the performance of ASR systems continues to improve, and application scenarios are becoming increasingly broad.
2. Code Analysis
Key Code
1. Speech Processing and Recognition Core (largemodel/largemodel/asr.py)
x# From largemodel/largemodel/asr.pydef kws_handler(self)->None: if self.stop_event.is_set(): return
if self.listen_for_speech(self.mic_index): asr_text = self.ASR_conversion(self.user_speechdir) # Perform ASR conversion if asr_text =='error': # Check if ASR result length is less than 4 characters self.get_logger().warn("I still don't understand what you mean. Please try again") playsound(self.audio_dict[self.error_response]) # Error response else: self.get_logger().info(asr_text) self.get_logger().info("okay😀, let me think for a moment...") self.asr_pub_result(asr_text) # Publish ASR result else: return
def ASR_conversion(self, input_file:str)->str: if self.use_oline_asr: result=self.modelinterface.oline_asr(input_file) if result[0] == 'ok' and len(result[1]) > 4: return result[1] else: self.get_logger().error(f'ASR Error:{result[1]}') # ASR error. return 'error' else: result=self.modelinterface.SenseVoiceSmall_ASR(input_file) if result[0] == 'ok' and len(result[1]) > 4: return result[1] else: self.get_logger().error(f'ASR Error:{result[1]}') # ASR error. return 'error'2. VAD Smart Recording (largemodel/largemodel/asr.py)
xxxxxxxxxx# From largemodel/largemodel/asr.pydef listen_for_speech(self,mic_index=0): p = pyaudio.PyAudio() # Create PyAudio instance. audio_buffer = [] # Store audio data. silence_counter = 0 # Silence counter. MAX_SILENCE_FRAMES = 90 # Stop after 900ms of silence (30 frames * 30ms) speaking = False # Flag indicating speech activity. frame_counter = 0 # Frame counter. stream_kwargs = { 'format': pyaudio.paInt16, 'channels': 1, 'rate': self.sample_rate, 'input': True, 'frames_per_buffer': self.frame_bytes, } if mic_index != 0: stream_kwargs['input_device_index'] = mic_index
# Prompt the user to speak via the buzzer. self.pub_beep.publish(UInt16(data = 1)) time.sleep(0.5) self.pub_beep.publish(UInt16(data = 0))
try: # Open audio stream. stream = p.open(**stream_kwargs) while True: if self.stop_event.is_set(): return False frame = stream.read(self.frame_bytes, exception_on_overflow=False) # Read audio data. is_speech = self.vad.is_speech(frame, self.sample_rate) # VAD detection.
if is_speech: # Detected speech activity. speaking = True audio_buffer.append(frame) silence_counter = 0 else: if speaking: # Detect silence after speech activity. silence_counter += 1 audio_buffer.append(frame) # Continue recording buffer. # End recording when silence duration meets the threshold. if silence_counter >= MAX_SILENCE_FRAMES: break frame_counter += 1 if frame_counter % 2 == 0: self.get_logger().info('1' if is_speech else '-') # Real-time status display. finally: stream.stop_stream() stream.close() p.terminate()
# Save valid recording (remove trailing silence). if speaking and len(audio_buffer) > 0: # Trim the last silent part. clean_buffer = audio_buffer[:-MAX_SILENCE_FRAMES] if len(audio_buffer) > MAX_SILENCE_FRAMES else audio_buffer with wave.open(self.user_speechdir, 'wb') as wf: wf.setnchannels(1) wf.setsampwidth(p.get_sample_size(pyaudio.paInt16)) wf.setframerate(self.sample_rate) wf.writeframes(b''.join(clean_buffer)) return TrueCode Analysis
The ASR (speech-to-text) feature is provided by the ASRNode node (asr.py). This node is responsible for recording, converting, and publishing audio.
Audio Recording (
listen_for_speech):This function uses the
pyaudiolibrary to capture the audio stream from the microphone.It integrates the
webrtcvadlibrary for voice activity detection (VAD). The function loops through audio frames and usesvad.is_speech()to determine whether each frame contains human voice.When speech is detected, the data is written to a buffer. When sustained silence (defined by
MAX_SILENCE_FRAMES) is detected, recording stops.Finally, the audio data in the buffer is written to a
.wavfile, with the path set toself.user_speechdir.
Backend Selection and Execution (
ASR_conversion):The
kws_handlerfunction calls theASR_conversionfunction after recording succeeds.This function decides which backend implementation to call by reading the ROS parameter
use_oline_asr(a Boolean value).If it is
false, theself.modelinterface.SenseVoiceSmall_ASRmethod is called, which corresponds to local recognition.If it is
true, theself.modelinterface.oline_asrmethod is called, which corresponds to online recognition.This function passes the audio file path as a parameter to the selected method and processes the returned result.
Result Publishing (
asr_pub_result):After
ASR_conversionreturns valid text,kws_handlercalls theasr_pub_resultfunction.This function wraps the text string in a
std_msgs.msg.Stringmessage and publishes it to the/asrtopic through the ROS publisher.
3. Practical Operations
3.1 Configuring ASR
Online and offline ASR share the microphone, serial port, sample rate, and language settings, and use_oline_asr selects the recognition backend. First open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlKeep the device settings in the existing asr.ros__parameters parameter block, and set the backend as needed:
xxxxxxxxxxasr ros__parameters use_oline_asrFalse # False: offline ASR; True: online ASR mic_serial_port"/dev/ttyUSB0" mic_index0 language'zh' # 'zh': Chinese; 'en': English regional_setting"China"First run ls /dev/ttyUSB* to confirm the device number of the voice module; if it is not /dev/ttyUSB0, modify mic_serial_port according to the actual device. If these parameters already exist, simply modify the original values; do not add duplicate keys to the same YAML parameter block.
3.1.1 Configuring Offline ASR
Set use_oline_asr to False, and confirm that the local SenseVoiceSmall model directory in large_model_interface.yaml exists:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlxxxxxxxxxxlocal_asr_model"/home/jetson/yahboom_ws/src/largemodel/MODELS/asr/SenseVoiceSmall"3.1.2 Configuring Online ASR
Prepare the API Key required by the Tongyi speech recognition service, and configure the online ASR credentials and model in large_model_interface.yaml:
xxxxxxxxxxonline_asr_api_key"Replace with your online ASR API-Key"oline_asr_model'paraformer-realtime-8k-v2'oline_asr_sample_rate16000Then set asr.ros__parameters.use_oline_asr in yahboom.yaml to True. Online recognition requires that the Jetson can access the corresponding service and that the account has available permissions or quota; do not disclose real API Keys.
3.2 Starting and Testing the Feature
Startup Command:
xxxxxxxxxxros2 launch largemodel asr_only.launch.pyTest: Say "Hello, Xiaoya" into the microphone, and it will reply "I'm here"; then you can start speaking, and finally the sound it captured, converted into text, is printed on the terminal.
