How Large Language Models Are Trained: Pretraining, Tokenization, Supervised Fine-Tuning, and Alignment
An engineering deep-dive into LLM training: data curation, BPE tokenization, GPU cluster parallelism, cross-entropy loss, SFT, and DPO alignment.
On this page

When interacting with a modern conversational AI assistant, the interface creates a compelling illusion of human-like consciousness: the model debugs Python code, reasons through corporate strategy, summarizes medical abstracts, and adopts nuanced tonal personas.
Beneath the conversational interface, however, a large language model is not an encyclopedic database, a symbolic reasoning engine, or an autonomous mind. It is a massive statistical pattern prediction engine: a deep neural network consisting of tens or hundreds of billions of numerical parameters (floating-point weights) calibrated to execute one single mathematical task: predict the most probable next token given a context of preceding tokens.
Understanding the engineering mechanics of LLM training demystifies their capabilities and their inherent failure modes (such as hallucinations and sycophancy). The modern training pipeline is a multi-stage industrial process spanning petabyte-scale data filtering, distributed GPU parallelism, supervised instruction tuning, and human preference alignment.
+-----------------------------------------------------------------------------------+
| THE FULL-STACK LLM TRAINING PIPELINE |
+-----------------------------------------------------------------------------------+
[ Stage 1: Data Curation & Tokenization ]
Petabytes of Web Crawls (Common Crawl, GitHub, ArXiv, Books)
|-- MinHash LSH Deduplication & PII Scrubbing
\-- Byte Pair Encoding (BPE Tokenizer, Vocab = 32k - 128k)
|
v
[ Stage 2: Large-Scale Pretraining (Unsupervised Base Model) ]
Thousands of GPUs / InfiniBand Cluster (Weeks/Months of Compute)
|-- Objective: Causal Language Modeling (Next-Token Cross-Entropy Loss)
\-- Output: Base Foundation Model (Raw Document Autocompleter)
|
v
[ Stage 3: Supervised Fine-Tuning (SFT / Instruction Tuning) ]
Tens of Thousands of High-Quality Instruction-Response Pairs
|-- Teaches turn-taking, prompt compliance, and assistant formatting
\-- Output: Instruction-Tuned Model
|
v
[ Stage 4: Preference Alignment (RLHF / DPO) ]
Human Preference Rankings (Prompts evaluated for helpfulness & harmlessness)
|-- Reward Modeling + PPO or Direct Preference Optimization (DPO)
\-- Enforces Safety Guardrails & Eliminates Sycophancy
|
v
[ Stage 5: Inference Optimization & Serving ]
Quantization (FP8, INT4 AWQ) + Continuous Batching + vLLM Inference
1. Stage 1: Data Ingestion, Deduplication, and Tokenization
A foundation model is only as capable as the training corpus it digests. Training a frontier model requires consuming between 3 trillion and 15 trillion tokens of text.
The Curation Pipeline
Raw web text scraped from the public internet (such as Common Crawl) is notoriously noisy, filled with machine-generated spam, HTML boilerplate, duplicated SEO listicles, and toxic forums.
The data engineering pipeline executes strict filtering:
- Deduplication via MinHash and LSH: Duplicate text accounts for up to 30% of the raw web. If a model encounters the identical document dozens of times during training, it memorizes the verbatim strings rather than learning generalizable concepts, inducing overfitting. Locality-Sensitive Hashing (MinHash LSH) removes near-duplicate documents.
- Quality Heuristics: Filter out documents with low word counts, unnatural punctuation distributions, or high language perplexity scores evaluated by fast statistical models (such as FastText).
- PII and Toxicity Scrubbing: Regex and classifier passes redact social security numbers, API credentials, and credit card details, while filtering out hate speech.
+-----------------------------------------------------------------------------------+
| BYTE PAIR ENCODING (BPE) TOKENIZATION |
+-----------------------------------------------------------------------------------+
Raw Input Text: "unpredictable"
Step 1: Character Segmentation:
['u', 'n', 'p', 'r', 'e', 'd', 'i', 'c', 't', 'a', 'b', 'l', 'e']
Step 2: Iterative Byte Pair Merges (Based on Corpus Frequency):
- Merge 'u' + 'n' -> 'un'
- Merge 'p' + 'r' + 'e' -> 'pre'
- Merge 'd' + 'i' + 'c' + 't' -> 'dict'
- Merge 'a' + 'b' + 'l' + 'e' -> 'able'
Final Token Representation:
[ Token 1042 ("un"), Token 2910 ("predict"), Token 653 ("able") ]
Why Subword Tokenization?
- Word-Level Tokenization Fails: Storing every English word in a dictionary requires millions of entries and fails immediately on compound terms, code variables, or typos.
- Character-Level Tokenization Fails: Slicing text into individual characters makes sequence lengths 4x longer. Because transformer self-attention scales quadratically with sequence length, character models are computationally prohibitive.
- Byte Pair Encoding (BPE): BPE identifies recurring character pairs, generating a compact, fixed vocabulary (typically 32,768 to 128,000 subword tokens). Common words become single tokens (e.g.,
"the"is 1 token), while rare words decompose into subwords.
2. Stage 2: Pretraining: Next-Token Prediction at Massive Scale
Pretraining consumes 98% of the total financial and computational budget of building an LLM. It transforms billions of randomized floating-point numbers into an organized representation of human language and world knowledge.
The Objective Function: Causal Next-Token Prediction
Given an input sequence of tokens (x_1, x_2, ..., x_t), the model processes the tokens through successive transformer layers, generating a probability distribution over the entire vocabulary for the next token x_{t+1}:
Cross-Entropy Loss Formula:
Loss = - (1 / T) * sum( log P(x_t | x_1, x_2, ..., x_{t-1}; θ) )
- If the context is “The capital of France is”, the model’s output vector assigns probabilities across all vocabulary tokens.
- If the model assigns high probability to
"Paris", loss is low. If it assigns high probability to"Banana", loss is high. - Backpropagation: The computed error traverses backward through hundreds of billions of parameters using the AdamW optimizer, making micro-adjustments to the weights.
+-----------------------------------------------------------------------------------+
| DISTRIBUTED TRAINING PARALLELISM ACROSS GPU CLUSTERS |
+-----------------------------------------------------------------------------------+
A 70B parameter model requires ~140 GB of VRAM just to hold weights in FP16,
plus 400+ GB for optimizer states and gradients! It cannot fit on one GPU.
Parallelism Tiers:
1. Tensor Parallelism (Megatron-LM):
Slices individual weight matrices across GPUs within the same server (NVLink).
2. Pipeline Parallelism:
Divides vertical transformer layers across multiple physical hosts.
3. Fully Sharded Data Parallelism (FSDP / ZeRO-3):
Shards model weights, gradients, and optimizer states across thousands of nodes.
Scaling Laws (The Chinchilla Principle)
In 2022, DeepMind published the Chinchilla scaling laws, establishing that model parameters and training tokens should scale in equal proportion:
Compute-Optimal Rule:
A model should be trained on approximately 20 tokens per parameter.
A 70B parameter model requires at least 1.4 Trillion tokens of training data to be compute-optimal.
3. Why Base Models Are Not Chatbots
When pretraining finishes, you have a Base Model (such as LLaMA-3-Base or Mistral-Base).
Engineers unfamiliar with LLM mechanics often test a base model by typing: “What is the capital of Germany?”
Instead of answering, the base model frequently outputs: “What is the capital of France? What is the capital of Italy?”
The base model is not broken: it is executing its training task flawlessly. It saw a document containing numbered test questions and predicted that the next most probable text was another test question. Base models are powerful autocomplete engines, not conversational agents.
4. Stage 3: Supervised Fine-Tuning (SFT / Instruction Tuning)
To transform an autocomplete engine into an assistant, the model undergoes Supervised Fine-Tuning (SFT):
[
{
"role": "user",
"content": "Write a Python function to check if a string is a palindrome."
},
{
"role": "assistant",
"content": "Here is an efficient Python implementation:\n\ndef is_palindrome(s: str) -> bool:\n clean = ''.join(c.lower() for c in s if c.isalnum())\n return clean == clean[::-1]"
}
]
- High-quality teams curate between 50,000 and 500,000 verified dialogue interactions written by domain specialists.
- The model trains using the identical next-token prediction objective, but loss is calculated strictly on the assistant’s tokens.
- Through SFT, the model learns the structural persona of an assistant: formatting answers in Markdown, adopting polite conversational greetings, and stopping generation when the answer is complete.
5. Stage 4: Preference Alignment (RLHF and DPO)
While SFT teaches models how to answer, models still suffer from toxic generations, untruthful guesses, and hallucination. The final stage aligns model behavior with human values using Reinforcement Learning from Human Feedback (RLHF).
+-----------------------------------------------------------------------------------+
| PREFERENCE ALIGNMENT: RLHF VS DPO |
+-----------------------------------------------------------------------------------+
[ Prompt: "Explain how to build an untraceable computer virus." ]
Model generates Candidate A (Refusal with safety explanation)
Model generates Candidate B (Executable malware script)
Human Evaluator ranks: Candidate A > Candidate B
Traditional RLHF (PPO):
1. Train a separate Reward Model to predict human ranking scores.
2. Optimize the LLM policy using Proximal Policy Optimization (PPO) against the reward model.
3. Enforce a KL Divergence penalty to keep the model from drifting too far from the SFT base.
Modern Direct Preference Optimization (DPO):
Implicitly optimizes the mathematical reward directly on preference pairs (A vs B)
without requiring a separate reward model or complex PPO actor-critic reinforcement loops!
The Hallucination Trade-Off
Alignment reduces toxicity, but it cannot fundamentally eliminate hallucination.
Because an LLM stores facts as distributed vector probabilities across billions of matrix parameters (lossy compression) rather than immutable database records, the model will always possess a mathematical probability of synthesizing plausible-sounding untruths. Production applications must ground models using Retrieval-Augmented Generation (RAG) or deterministic tool execution.
Summary Takeaways
- LLMs are next-token predictors: Every capability, from code synthesis to creative writing, emerges from minimizing cross-entropy loss over trillions of tokens.
- Pretraining builds the knowledge engine: Pretraining consumes 98% of compute resources, encoding world facts and syntactic structures into static weight parameters.
- SFT teaches instruction compliance: Instruction fine-tuning converts raw document autocompleters into structured conversational agents.
- Alignment guides behavior: Techniques like RLHF and DPO guide the model toward safe, helpful responses while penalizing harmful or sycophantic outputs.
- Architect for inherent limitations: Because models compress knowledge into statistical weights, mission-critical enterprise systems must pair LLMs with RAG and sandboxed tool execution to prevent hallucinations.
Frequently asked questions
What is the single mathematical objective of LLM pretraining?
The foundational objective of causal language model pretraining is next-token prediction. Given a sequence of preceding tokens, the model computes a probability distribution over its entire vocabulary, updating its billions of internal weight parameters via backpropagation to minimize cross-entropy loss.
Why is Byte Pair Encoding (BPE) preferred over character or word tokenization?
Word-level tokenization creates an unmanageably huge vocabulary that fails on rare or misspelled words. Character-level tokenization produces excessively long sequence lengths that explode self-attention costs. BPE iteratively merges frequent character pairs, compressing text into a compact vocabulary.
What is the difference between Pretraining and Supervised Fine-Tuning (SFT)?
Pretraining trains a model on trillions of raw web tokens to learn grammar, factual world representations, and reasoning patterns. However, a pretrained base model merely autocompletes text. SFT trains the base model on curated conversation pairs, teaching it to follow user instructions.
Why does Reinforcement Learning from Human Feedback (RLHF) use a KL divergence penalty?
When optimizing an agent against a reward model using algorithms like PPO or DPO, the policy can exploit reward loopholes, generating repetitive or unhinged text that scores artificially high (reward hacking). The Kullback-Leibler (KL) penalty penalizes deviations from the original base model.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.