AI Large Model Types and Principles

Please ensure that the web control program backend process is not running in the background. To close it, refer to the [Get Started Quickly - 2. Access the Terminal] section in [Disable the Auto-Start Web Control Backend Process]. After closing, please restart DofbotUltra!!!

Before starting this feature, you need to close the web control program backend process.

1. Course Content

Introduction to basic knowledge of AI large models, types and principles of large models used in embodied AI gameplay.

2. Introduction to AI Large Models

AI large models, i.e., large-scale pre-trained models, are the product of deep integration of big data, large computing power, and strong algorithms. Simply put, they are like intelligent entities that have been "fed" with massive knowledge and repeatedly trained. By learning from large amounts of data, they grasp the patterns and regularities within the data, thereby possessing strong generality and generalization capabilities. This ability allows AI large models to no longer be limited to a single task but to flexibly apply learned knowledge like humans, solving complex problems in multiple fields. Utilizing multi-source data such as internet text and images, they learn data patterns, possess strong generalization capabilities, and can adapt to various tasks through transfer learning or prompt engineering. They may also exhibit capabilities such as logical reasoning and common sense understanding that were not explicitly set during training.

3. Common AI Large Model Categories

3.1 Text Generation Large Models

• Based on the Transformer architecture, through unsupervised or supervised learning on massive text data, they learn the grammatical, semantic, and pragmatic rules of language, enabling them to generate natural and fluent text based on input prompts or context. Pre-training through unsupervised learning makes them suitable for tasks such as text generation and dialogue systems; they convert all text tasks into a unified text-to-text problem, providing a more flexible framework that can handle various tasks like translation, summarization, and Q&A.

• In content creation, they can automatically generate practical texts such as articles, news, and reviews, improving content production efficiency, and can assist authors with creative ideation and text polishing.

• In the field of intelligent interaction, they can be applied to intelligent customer service and chatbots, generating natural and fluent responses to enhance user experience.

• In personalized education, they can analyze questions, provide test point explanations, solution ideas, and results, as well as help users with language learning.

• In machine translation, they can achieve automatic translation, and when combined with voice models, can also enable simultaneous interpretation and daily subtitle generation.

  3.1.1 Principle Introduction

  Based on the Transformer architecture (especially the self-attention mechanism), through unsupervised/semi-supervised learning on massive text data, modeling the probability distribution and semantic associations of language to achieve understanding and generation of natural language.

  • Pre-training Logic

    • Autoregressive (AR): Such as the GPT series, predicts the next token through "causal language modeling" (e.g., "Today's weather -> Today's weather is"), learning the sequential dependencies of text.

    • Autoencoding (AE): Such as BERT, predicts masked tokens through "masked language modeling" (e.g., "Today [mask] is sunny" -> "weather"), learning bidirectional semantics from context.

  • Key Technologies

    • Attention Mechanism: Dynamically assigns weights to different words in the text, capturing long-distance dependencies (e.g., "previously mentioned event -> subsequent impact").

    • Prompt Tuning: Activates specific capabilities of the model through templates (e.g., "Please summarize the following: {text}"), adapting to downstream tasks.

  • Emergent Abilities As parameter scale increases (e.g., hundreds of billions), models may exhibit abilities such as logical reasoning, common sense understanding, and few-shot learning that were not explicitly programmed during the pre-training phase.

3.2 Multimodal Large Models

Capable of processing multiple types of input data, such as text, images, audio, video, etc. Through cross-modal learning, they understand the relationships between different modalities of data and integrate multimodal data to fully utilize information from each modality, constructing a unified representation space that enables different modalities to understand and combine with each other, thereby performing more complex and intelligent tasks. They can be used for cross-modal retrieval, retrieving data of one modality based on another; in visual question answering, the model answers text questions based on image content; they can also generate image descriptions, producing natural language text describing image content; and enable multimodal dialogue involving multiple modalities of information, with broad application prospects in complex environments such as healthcare, transportation, and security surveillance.

3.2.1 Principle Introduction

Through cross-modal alignment and joint modeling, learning a unified representation space for different modalities of data, achieving semantic association and collaborative processing between modalities.

• Contrastive Learning: Such as the CLIP model, maps feature vectors of images and text to the same space, trained through "image-text pair matching" (e.g., "picture of a dog -> text 'dog'").

• Encoder-Decoder Architecture: Such as DALL-E, a text encoder extracts semantic features, and an image decoder generates the corresponding image (text -> image).

Fusion Methods

• Early Fusion: Merges multimodal data at the input layer (e.g., concatenating text embeddings with image pixel features).

• Late Fusion: Processes each modality separately and fuses results at the decision layer (e.g., first analyzing text sentiment and image color independently, then making a comprehensive judgment).

3.3 Speech Recognition Models

• Convert input speech signals into text information. Typically based on deep learning algorithms, they first extract features from the speech signal, then input the features into a neural network model for training and recognition. The model learns to recognize different speech patterns and corresponding text content by studying large amounts of speech data. They can assist customer service personnel in quickly recording customer needs and issues, improving service quality and facilitating subsequent queries; can be applied to voice search, freeing up hands, suitable for various search environments such as vehicle navigation and mobile phones; can also convert meeting conversations into text format for easy organization and recording of meeting content; in human-computer interaction, use voice commands to control smart facilities, including robots and other hardware devices and software applications.

  3.3.1 Principle Introduction

  Convert acoustic features of speech signals into text sequences, achieving end-to-end modeling based on deep learning.

  • Feature Extraction: Preprocess speech waveforms (e.g., framing, windowing), extract Mel-frequency cepstral coefficients (MFCC) or acoustic feature vectors (e.g., extracted via CNN).

  • Sequence Modeling: Use Recurrent Neural Networks (RNN/LSTM) or Transformer encoders to capture the temporal dependencies of speech sequences (e.g., "consecutive syllables -> words").

  • Decoding Mapping: Through Connectionist Temporal Classification (CTC) or attention mechanisms, map feature sequences to text sequences (e.g., acoustic features -> "Hello").

3.4 Speech Synthesis Models

• Convert input text into speech signals. Generally, by training a model to learn the mapping relationship from text to speech, the model generates corresponding speech features based on the input text, and then converts the features into audible speech through speech synthesis technology. Widely used in voice assistants, audiobooks, intelligent customer service, and other fields, providing voice interaction services to users, enabling devices to communicate with users in natural and fluent speech.

  3.4.1 Principle Introduction

  Convert text semantics into natural and fluent speech signals, simulating the prosody, intonation, and emotion of human pronunciation.

  Deep Learning Synthesis:

  • Text Analysis: Parse text semantics, part-of-speech, and emotion through NLP models (e.g., "Very happy today" -> cheerful tone).

  • Acoustic Modeling: Use the Tacotron series (encoder-decoder + attention mechanism) to generate speech Mel-spectrograms.

  • Vocoder: Convert Mel-spectrograms into waveform signals, such as WaveNet, HiFi-GAN (improving speech naturalness).

4. AI Model Comparison Summary