How Large Language Models Work: From Text Tokens to Neural Networks

Computers do not read words like humans do. This curriculum explains how large language models turn text into numbers and use a process called attention to understand context. You will see how these models are trained on massive amounts of data to predict the next part of a sentence. By the end, you will understand the mechanics behind modern AI and the current debates about how to make these systems more efficient and reliable.

Breaking text into tokens

The learner will be able to explain how text is split into sub-word units for a computer to read.

Key takeaways

  • Tokenization converts text into numerical IDs that a computer can process.
  • Sub-word tokenization balances vocabulary size and sequence length by breaking words into meaningful pieces.
  • Different algorithms like BPE and WordPiece use frequency or math to decide how to split words.
  • New vision-based methods may eventually replace traditional tokens to better handle typos and different languages.

Sources

Representing words as numbers

The learner will be able to describe how word embeddings map meanings to mathematical vectors.

Key takeaways

  • Word embeddings are lists of numbers that represent the meaning of words in a high-dimensional mathematical space.
  • Static embeddings give each word one fixed vector while contextual embeddings change the vector based on the surrounding words.
  • Cosine similarity is the primary mathematical tool used to measure how closely related two word vectors are.
  • Vector databases and specialized algorithms allow computers to search through millions of these numerical representations almost instantly.

Sources

The attention mechanism

The learner will be able to explain how a model decides which words in a sentence are most important.

Key takeaways

  • The attention mechanism uses Queries, Keys, and Values to compare words and decide which ones provide the most useful context.
  • Importance is calculated by a dot product between vectors, which is then turned into a set of percentages that add up to one.
  • Different layers of a model focus on different things, with lower layers looking at grammar and higher layers looking at meaning.
  • While attention is highly effective, some research suggests that other parts of a model's architecture also play a major role in its performance.

Sources

Building the transformer block

The learner will be able to identify the main parts of a transformer layer including feed-forward networks.

Key takeaways

  • A transformer block consists of a self-attention mechanism for communication and a feed-forward network for computation.
  • Feed-forward networks process each token independently and typically hold the majority of the model's parameters.
  • Residual connections and layer normalization are used to keep the data stable as it moves through many stacked blocks.
  • Decoder blocks include an extra attention layer to connect with the encoder and use masking to prevent the model from seeing future words.

Sources

Training on massive datasets

The learner will be able to describe the self-supervised learning process used to predict the next token.

Key takeaways

  • Self-supervised learning uses the existing order of words in a dataset to create training labels without human help.
  • Causal language modeling requires the model to predict the next token using only the information that came before it.
  • The training process minimizes cross-entropy loss by adjusting model parameters to make the correct next token more probable.
  • Tokenization and positional embeddings are necessary to turn raw text into a format that a transformer can process.

Sources

Teaching models to follow instructions

The learner will be able to distinguish between raw pre-training and supervised fine-tuning.

Key takeaways

  • Raw pre-training teaches a model the patterns of language and general facts but does not teach it how to follow instructions or be a helpful assistant.
  • Supervised Fine-Tuning uses a curated set of high-quality input and output pairs to teach a model how to respond correctly to human prompts.
  • SFT is a stable and efficient way to establish a foundation of competence but often fails to handle complex ethical reasoning or prevent sophisticated safety violations.
  • Most modern AI models use SFT as a foundational step before moving on to more advanced alignment techniques like Reinforcement Learning from Human Feedback.

Sources

The debate over model size

The learner will be able to compare the benefits of giant models against smaller, specialized ones.

Key takeaways

  • Giant models are necessary for complex multi-hop reasoning and emergent behaviors like chain-of-thought logic.
  • Small models offer significant advantages in speed and cost and can often run on local hardware like phones or single GPUs.
  • High-quality training data and architectural innovations allow small models to match the performance of much larger systems in specific tasks.
  • Many organizations use a hybrid approach that routes simple tasks to small models and complex ones to giant models to balance cost and performance.

Sources

New ways to process sequences

The learner will be able to identify alternatives to the standard attention mechanism like state space models.

Key takeaways

  • Standard attention mechanisms grow in cost quadratically, making long sequences very expensive to process.
  • State Space Models like Mamba use a constant-sized hidden state to achieve linear scaling and faster inference.
  • Pure State Space Models often struggle with exact fact recall and complex step-by-step reasoning compared to Transformers.
  • Modern AI design is shifting toward hybrid models that combine the efficiency of State Space Models with the reasoning power of attention.

Sources