Welcome to ROSMASTER M1 repository
2. Offline Text-to-Speech (TTS)
2. Offline Text-to-Speech (TTS)
2. Offline Text-to-Speech (TTS)1. Introduction1.1 What is “TTS”?1.2 Brief Description of Implementation Principles1. Text Analysis2. Language Processing3. Speech Synthesis4. Sound Waveform Generation2. Code AnalysisKey Code1. TTS Initialization and Invocation (largemodel/largemodel/model_service.py)2. TTS backend implementation (largemodel/utils/large_model_interface.py)Code Analysis3. Practical Operations3.1 Configuring Offline TTS3.2 Starting and Testing the Functionality4. Common Problems and Solutions4.1 Playback IssuesIssue 1: The program runs normally without errors, but no sound is heard.
1. Introduction
1.1 What is “TTS”?
TTS technology is a technology that converts written text into human-readable speech output. It enables computers to “read” text content and is widely used in various fields such as accessible reading, intelligent assistants, navigation systems, and educational software. Through TTS, users can hear natural and fluent human voices generated by machines, greatly improving the convenience and flexibility of information acquisition.
1.2 Brief Description of Implementation Principles
The implementation of a TTS system mainly includes the following key steps and technologies:
1. Text Analysis
- In this stage, the input text is first preprocessed, including but not limited to removing irrelevant characters, standardizing punctuation and capitalization, word segmentation, and recognizing special symbols such as numbers and converting them into their corresponding word forms.
- Simultaneously, linguistic analysis of the text is required, such as determining the pronunciation of each word (which usually requires the use of a pronunciation dictionary), stress positions, intonation patterns, and sentence structure.
2. Language Processing
- This step primarily focuses on how to correctly pronounce and adjust intonation based on context. For example, the word "read" has different pronunciations in different tenses (past tense/past participle is pronounced /red/, while in other cases it is pronounced /riːd/). Therefore, a powerful language model is needed to understand these subtle differences.
- This also involves prosodic modeling, i.e., determining which parts should be emphasized, whether the speech rate should be fast or slow, and the overall emotional tone of the sentence.
3. Speech Synthesis
- The text information processed in the previous two stages is fed into a speech synthesis engine, which is responsible for generating the actual sound waveforms.
- Traditional TTS systems use a concatenation synthesis method, which selects appropriate units from a pre-recorded speech segment database and concatenates them to form a complete sentence. While this method can produce high-quality speech, it is limited by the samples in the database.
- Modern TTS systems rely more on parametric synthesis or neural network synthesis (such as WaveNet, Tacotron, etc.). These methods can directly predict speech features from text and generate continuous sound signals. Especially deep learning-based methods can better capture subtle changes in speech, thus producing more natural and fluent sounds.
4. Sound Waveform Generation
- Finally, the generated sound waveform is further processed to ensure its quality meets expected standards, such as adjusting volume and equalizing frequency response.
- Afterward, this audio data can be played back through speakers or other audio playback devices for people to listen to.
With the development of artificial intelligence and machine learning technologies, especially the application of deep learning, TTS systems have not only significantly improved in accuracy but also made great strides in naturalness and emotional expression, making machine-generated speech increasingly closer to real human voices.
2. Code Analysis
Key Code
1. TTS Initialization and Invocation (largemodel/largemodel/model_service.py)
xxxxxxxxxx# From largemodel/largemodel/model_service.pyclass LargeModelService(Node): def __init__(self): # ... self.system_sound_init() # ... def init_param_config(self): # ... self.declare_parameter('useolinetts', False) self.useolinetts = self.get_parameter('useolinetts').get_parameter_value().bool_value if self.useolinetts: self.tts_out_path = os.path.join(self.pkg_path, "resources_file", "tts_output.mp3") else: self.tts_out_path = os.path.join(self.pkg_path, "resources_file", "tts_output.wav") def system_sound_init(self): """Initialize TTS system""" model_type = "oline" if self.useolinetts else "local" self.model_client.tts_model_init(model_type, self.language) self.get_logger().info(f'TTS initialized with {model_type} model') def _safe_play_audio(self, text_to_speak: str): """ Synthesizes and plays all non-empty messages only in non-text chat mode. """ if not self.text_chat_mode and text_to_speak: try: self.model_client.voice_synthesis(text_to_speak, self.tts_out_path) self.play_audio_async(self.tts_out_path) except Exception as e: self.get_logger().error(f"Safe audio playback failed: {e}")2. TTS backend implementation (largemodel/utils/large_model_interface.py)
xxxxxxxxxx# From largemodel/utils/large_model_interface.pyclass model_interface: # ... def tts_model_init(self,model_type='oline',language='zh'): if model_type=='oline': if self.tts_supplier=='baidu': self.token=self.fetch_token() self.model_type='oline' elif model_type=='local': self.model_type='local' if language=='zh': tts_model=self.zh_tts_model tts_json=self.zh_tts_json elif language=='en': tts_model=self.en_tts_model tts_json=self.en_tts_json self.synthesizer = piper.PiperVoice.load(tts_model, config_path=tts_json, use_cuda=False) def voice_synthesis(self,text,path): if self.model_type=='oline': if self.tts_supplier=='baidu': # ... (Baidu TTS implementation) pass elif self.tts_supplier=='aliyun': # ... (Aliyun TTS implementation) pass elif self.model_type=='local': with wave.open(path, 'wb') as wav_file: wav_file.setnchannels(1) wav_file.setsampwidth(2) wav_file.setframerate(self.synthesizer.config.sample_rate) self.synthesizer.synthesize(text, wav_file)Code Analysis
The text-to-speech (TTS) function is invoked by the LargeModelService node and implemented by the model_interface class. Its design uses parameter configuration to switch between different backend services.
Initialization Process (
model_service.py):- During
LargeModelServiceinitialization, theinit_param_configfunction reads the Boolean valueuseolinettsfrom the ROS parameter server. - Based on the value of
useolinetts, thesystem_sound_initfunction passes either the'local'or'oline'string to theself.model_client.tts_model_initmethod. - In
large_model_interface.py, thetts_model_initmethod executes the corresponding initialization logic based on the string parameter received. If the value is'local',piper.PiperVoice.loadis used to load the local model file.
- During
Synthesis and Playback Process (
model_service.py):- When voice playback is required, the
_safe_play_audiofunction is called. - This function first calls the
self.model_client.voice_synthesismethod, passing in the text to be converted and the target audio pathself.tts_out_path. - After the
voice_synthesismethod completes and generates an audio file,_safe_play_audiocallsself.play_audio_asyncto asynchronously play the file.
- When voice playback is required, the
Backend Implementation Selection (
large_model_interface.py):- The
voice_synthesismethod is the backend dispatch center for the TTS functionality. Internally, it selects the execution path by checking theself.model_typeproperty value set during initialization. - If
self.model_typeis'local', the code block uses Python'swavelibrary to open a WAV file, sets its header parameters (channels, sample bit width, and sampling rate), and then callsself.synthesizer.synthesizeto write the synthesized text audio stream directly to the file. - If
self.model_typeis'oline', the code branch will be executed for different cloud service providers (such as Baidu and Alibaba Cloud). - This structure separates the upper-level node call ("Speak this sentence") from the lower-level specific synthesis technology (which library to use, which API to call).
- The
3. Practical Operations
3.1 Configuring Offline TTS
To enable offline TTS, you need to correctly configure yahboom.yaml and large_model_interface.yaml and ensure that the local model is correctly placed.
Raspberry Pi 5 requires entering a Docker container, while RDK X5 and Orin controllers do not：
xxxxxxxxxx./ros2_docker.sh
If you need to enter the same Docker container and run other commands later, simply enter ./ros2_docker.sh again in the host terminal.
Open the main configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/yahboom.yamlModify/confirm the following key configuration::
x
model_service#Model server node parametersros__parameterslanguage'zh' #Large Model Interface LanguageuseolinettsFalse #Whether to use online speech synthesis (True to use online, False to use offline)regional_setting "China"useolinetts Make sure this is set to False to use the local model.
Select "zh" for Chinese and "en" for English.
Open the model interface configuration file:
xxxxxxxxxxvim ~/yahboom_ws/src/largemodel/config/large_model_interface.yamlConfirm the offline model path:
Raspberry Pi 5