AI Large Model Types and Principles

1. Course Content

• Learn and master the basic knowledge of AI large models and the types of AI large models used in embodied AI gameplay

[!TIP]

• This course is a pure theory course. If you want to quickly get started with hands-on features, you can skip this course for now.

2. AI Large Model Introduction

AI large models, i.e., large-scale pre-trained models, are the product of the deep integration of big data, massive computing power, and strong algorithms. In simple terms, they are like intelligent agents that have been "fed" with vast amounts of knowledge and repeatedly trained. By learning from large amounts of data, they master the patterns and regularities in the data, thereby possessing strong universality and generalization capabilities. This capability allows AI large models to no longer be limited to a single task, but to flexibly apply learned knowledge like humans to solve complex problems in multiple fields. Leveraging multi-source data such as internet text and images, they learn data patterns, possess strong generalization ability, and can adapt to various tasks through transfer learning or prompt engineering. They may also exhibit capabilities not explicitly set before training, such as logical reasoning and commonsense understanding.

• Open-source model community: ModelScope
• Alibaba Bailian large model: Bailian Large Model

3. Common AI Large Model Categories

3.1 Text Generation Large Models

• Based on the Transformer architecture, through unsupervised or supervised learning on massive text data, they learn the syntax, semantics, and pragmatic rules of language, enabling them to generate natural and fluent text based on input prompts or context. They are pre-trained through unsupervised learning and are suitable for tasks such as text generation and dialogue systems; they convert all text tasks into a unified text-to-text problem, providing a more flexible framework that can handle tasks such as translation, summarization, and question answering.

• In content creation, they can automatically generate practical text such as articles, news, and comments, improving content production efficiency, and can also assist authors with creative brainstorming and text polishing.

• In the field of intelligent interaction, they can be applied to intelligent customer service and chatbots to generate natural and fluent replies, improving user experience.

• In personalized teaching, they can analyze questions, provide exam point explanations, problem-solving approaches, and results, and can also help users with language learning.

• In machine translation, they can achieve automatic translation, and combined with speech-based models, they can also achieve simultaneous interpretation and everyday subtitle generation.

  3.1.1 Principle Introduction

  Based on the Transformer architecture (especially the self-attention mechanism), through unsupervised/semi-supervised learning on massive text data, they model the probability distribution and semantic associations of language to achieve understanding and generation of natural language.

  • Pre-training logic

    • Autoregressive (AR): such as the GPT series, which predicts the next token through a "causal language model" (e.g., "today's weather → today's weather is"), learning the forward-backward dependencies of text.
    • Autoencoding (AE): such as BERT, which predicts masked tokens through a "masked language model" (e.g., "today [mask] is sunny" → "weather"), learning bidirectional semantics of context.

  • Key technologies

    • Attention mechanism: dynamically assigns weights to different words in text, capturing long-range dependencies (e.g., "the event mentioned earlier → its impact later").
    • Prompt tuning: activates specific capabilities of the model through templates (e.g., "please summarize the following: {text}") to adapt to downstream tasks.

  • Emergent abilities As the parameter scale increases (e.g., hundreds of billions), the model may exhibit abilities not explicitly programmed during pre-training, such as logical reasoning, commonsense understanding, and few-shot learning.

3.2 Multimodal Large Models

They can handle multiple types of input data, such as text, images, audio, and video. Through cross-modal learning, they understand the relationships between data of different modalities and integrate multimodal data together to make full use of the information from each modality, building a unified representation space that enables data of different modalities to understand and combine with each other, thereby executing more complex and intelligent tasks. They can be used for cross-modal retrieval, where data of one modality is used to retrieve data of another modality; in visual question answering, the model answers text questions based on image content; they can also perform image caption generation, generating natural language text describing image content; and achieve multimodal dialogue, conducting dialogues involving information from multiple modalities, with broad application prospects in complex environments such as healthcare, transportation, and security surveillance.

3.2.1 Principle Introduction

Through cross-modal alignment and joint modeling, they learn a unified representation space for data of different modalities, achieving semantic association and collaborative processing between modalities.

• Contrastive learning: such as the CLIP model, which maps feature vectors of images and text into the same space and trains through "image-text pair matching" (e.g., "a picture of a dog → text 'dog'").
• Encoder-decoder architecture: such as DALL-E, where a text encoder extracts semantic features and an image decoder generates the corresponding image (text → image).

Fusion methods

• Early fusion: merges multimodal data at the input layer (e.g., concatenating text embeddings with image pixel features).
• Late fusion: processes data of each modality separately and fuses results at the decision layer (e.g., first analyzing text sentiment and image color separately, then making a comprehensive judgment).

3.3 Speech Recognition Models

• They convert input speech signals into text information. Usually based on deep learning algorithms, they first extract features from the speech signal, then feed the features into a neural network model for training and recognition. The model learns to recognize different speech patterns and corresponding text content by learning from large amounts of speech data. They can assist customer service staff in quickly recording customer needs and issues, improving service quality and facilitating later queries; they can be applied to voice search, freeing up hands, and are suitable for various search environments such as vehicle navigation and mobile phones; they can also convert meeting conversations into text form, making it easy to organize and record meeting content; in human-computer interaction, they use voice commands to control intelligent devices, including hardware facilities such as robots and software applications.

  3.3.1 Principle Introduction

  They convert the acoustic features of speech signals into text sequences, achieving end-to-end modeling based on deep learning.

  • Feature extraction: preprocesses the speech waveform (e.g., framing, windowing), extracting Mel-frequency features (MFCC) or acoustic feature vectors (e.g., extracted via CNN).
  • Sequence modeling: uses recurrent neural networks (RNN/LSTM) or Transformer encoders to capture the temporal dependencies of speech sequences (e.g., "continuous syllables → words").
  • Decoding and mapping: through connectionist temporal classification (CTC) or attention mechanisms, maps feature sequences to text sequences (e.g., acoustic features → "hello").

3.4 Speech Synthesis Models

They convert input text into speech signals. Generally, by training a model to learn the mapping relationship from text to speech, the model generates corresponding speech features based on the input text, and then converts the features into audible speech through speech synthesis technology. It is widely used in fields such as voice assistants, audiobooks, and intelligent customer service, providing voice interaction services for users and enabling devices to communicate with users in natural and fluent speech.

3.4.1 Principle Introduction

They convert text semantics into natural and fluent speech signals, simulating the prosody, intonation, and emotion of human pronunciation.

Deep learning synthesis:

• Text analysis: parses the semantics, part of speech, and sentiment of text through NLP models (e.g., "very happy today" → cheerful intonation).
• Acoustic modeling: uses the Tacotron series (encoder-decoder + attention mechanism) to generate Mel-frequency spectrograms of speech.
• Vocoder: converts Mel-frequency spectrograms into waveform signals, such as WaveNet and HiFi-GAN (improving speech naturalness).

4. AI Model Comparison Summary