AI Large Model Types and Principles

1. Course Content

• Understand the fundamentals of AI large models and the types of AI large models used in AI embodied intelligence features

2. Introduction to AI Large Models

AI large models, i.e., large-scale pre-trained models, are the product of the deep integration of big data, massive computing power, and powerful algorithms. In simple terms, they are like intelligent agents that have been "fed" with vast amounts of knowledge and trained repeatedly. By learning from large quantities of data, they grasp the patterns and regularities within the data and thus acquire strong generality and generalization capability. This capability means that AI large models are no longer confined to a single task; instead, like humans, they can flexibly apply their learned knowledge to solve complex problems across multiple domains. Leveraging multi-source data such as internet text and images, they learn data regularities and exhibit strong generalization ability. Through transfer learning or prompt engineering, they can adapt to a wide variety of tasks, and may even exhibit capabilities not explicitly set during training, such as logical reasoning and commonsense understanding.

• Open-source model community: ModelScope

• Alibaba Bailian large model: Bailian Large Model

3. Common Categories of AI Large Models

3.1 Text Generation Large Models

• Based on the Transformer architecture, these models learn the syntactic, semantic, and pragmatic rules of language through unsupervised or supervised learning on massive text corpora, enabling them to generate natural and fluent text according to an input prompt or context. Pre-trained through unsupervised learning, they are applicable to tasks such as text generation and dialogue systems; by converting all text tasks into a unified text-to-text problem, they provide a more flexible framework that can handle a variety of tasks such as translation, summarization, and question answering.

• In content creation, they can automatically generate practical text such as articles, news, and comments, improving content production efficiency, and can also assist authors with creative ideation and text polishing.

• In the field of intelligent interaction, they can be applied to intelligent customer service and chatbots, generating natural and fluent replies and enhancing the user experience.

• In personalized tutoring, they can analyze questions, provide exam-point explanations, solution approaches, and results, and can also help users learn languages.

• In machine translation, they can deliver automatic translation, and when combined with speech-based models, they can also achieve simultaneous interpretation and automatic daily subtitle generation.

  3.1.1 Principle Overview

  Based on the Transformer architecture (especially the self-attention mechanism), the model performs unsupervised/semi-supervised learning on massive text corpora to model the probability distribution and semantic associations of language, thereby achieving understanding and generation of natural language.

  • Pre-training logic

    • Autoregressive (AR): e.g., the GPT series, which predicts the next token through a "causal language model" (e.g., "today's weather → today's weather is fine"), learning the dependencies between preceding and following text.

    • Autoencoding (AE): e.g., BERT, which predicts masked tokens through a "masked language model" (e.g., "today [mask] is sunny" → "weather"), learning bidirectional semantics of context.

  • Key technologies

    • Attention mechanism: dynamically assigns weights to different words in the text to capture long-range dependencies (e.g., "an event mentioned earlier → its impact mentioned later").

    • Prompt tuning: activates the model's specific capabilities through templates (e.g., "Please summarize the following content: {text}") to adapt to downstream tasks.

  • Emergent capabilities As the parameter scale grows (e.g., hundreds of billions), the model may emerge with capabilities not explicitly programmed during the pre-training stage, such as logical reasoning, commonsense understanding, and few-shot learning.

3.2 Multimodal Large Models

These models are capable of processing multiple types of input data, such as text, images, audio, and video. Through cross-modal learning, they understand the relationships between data of different modalities and integrate multimodal data so as to fully utilize the information of each modality and build a unified representation space. This enables data of different modalities to understand and combine with one another, thereby performing more complex and intelligent tasks. They can be used for cross-modal retrieval, retrieving data of one modality based on data of another modality; in visual question answering, the model answers text questions based on image content; they can also perform image captioning, generating natural language text that describes image content; and they can implement multimodal dialogue, conducting conversations that involve information from multiple modalities. They have broad application prospects in complex environments such as healthcare, transportation, and security surveillance.

3.2.1 Principle Overview

Through cross-modal alignment and joint modeling, the model learns a unified representation space for data of different modalities, achieving semantic associations and collaborative processing among modalities.

• Contrastive learning: e.g., the CLIP model, which maps the feature vectors of images and text into the same space and is trained through "image-text pair matching" (e.g., "an image of a dog → the text 'dog'").

• Encoder-decoder architecture: e.g., DALL-E, in which the text encoder extracts semantic features and the image decoder generates the corresponding image (text → image).

Fusion methods

• Early fusion: merges multimodal data at the input layer (e.g., concatenating text embeddings with image pixel features).

• Late fusion: processes each modality separately and fuses the results at the decision layer (e.g., first analyzing text sentiment and image color individually, then making a comprehensive judgment).

3.3 Speech Recognition Models

• These models convert input speech signals into text information. They are typically based on deep learning algorithms: the speech signal is first subjected to feature extraction, and then the features are fed into a neural network model for training and recognition. By learning a large amount of speech data, the model identifies different speech patterns and the corresponding text content. They can assist customer service personnel in quickly recording customer needs and issues, improving service quality and facilitating subsequent inquiry; they can be applied to voice search, freeing the hands, and are suitable for a variety of search environments such as vehicle navigation and mobile phones; they can also convert meeting conversations into text, making it convenient to organize and document meeting content; and in human-computer interaction, they can use voice instructions to control intelligent facilities, including hardware facilities and software applications such as robots.

  3.3.1 Principle Overview

  The acoustic features of speech signals are converted into text sequences, achieving end-to-end modeling based on deep learning.

  • Feature extraction: preprocesses the speech waveform (e.g., frame segmentation and windowing) and extracts Mel-spectrum (MFCC) or acoustic feature vectors (e.g., extracted via CNN).

  • Sequence modeling: uses recurrent neural networks (RNN/LSTM) or a Transformer encoder to capture the temporal dependencies of speech sequences (e.g., "continuous syllables → words").

  • Decoding and mapping: through connectionist temporal classification (CTC) or attention mechanisms, maps feature sequences to text sequences (e.g., "acoustic features → 'hello'").

3.4 Speech Synthesis Models

These models convert input text into speech signals. Generally, a model is trained to learn the mapping relationship from text to speech: the model generates the corresponding speech features from the input text, and then speech synthesis technology converts these features into audible speech. They are widely applied in fields such as voice assistants, audiobooks, and intelligent customer service, providing users with voice interaction services so that devices can communicate with users in natural and fluent speech.

3.4.1 Principle Overview

Text semantics are converted into natural and fluent speech signals, simulating the prosody, intonation, and emotion of human pronunciation.

Deep learning synthesis:

• Text analysis: parses the semantics, part of speech, and emotion of the text through NLP models (e.g., "I am very happy today" → a cheerful intonation).

• Acoustic modeling: uses the Tacotron series (encoder-decoder + attention mechanism) to generate the Mel-spectrogram of speech.

• Vocoder: converts the Mel-spectrogram into a waveform signal, e.g., WaveNet and HiFi-GAN (improving speech naturalness).

4. AI Model Comparison Summary