AI Training

How AI models are trained — pretraining, fine-tuning, RLHF, and the datasets and techniques behind them.

  1. 1.What Is RLHF?How raw next-token predictors get turned into helpful, well-behaved assistants.
  2. 2.Pretraining: How Models Learn LanguageThe first, most expensive training stage — where a model learns to predict text from a huge unlabeled corpus.
  3. 3.Fine-Tuning vs PromptingTwo very different ways to get a general-purpose model to behave the way you want for your specific use case.
  4. 4.What Is DPO (Direct Preference Optimization)?A simpler alternative to full RLHF for teaching a model human preferences.
  5. 5.AI Hallucinations: Why Models Make Things UpWhy confident-sounding wrong answers are a natural consequence of how these models are trained, not a rare bug.
  6. 6.What Is Model Distillation?How providers train small, fast models that punch above their weight by learning from a larger "teacher" model.
  7. 7.What Is Reinforcement Learning? The Basics Behind RLHFReinforcement learning is how a model learns from feedback on its own actions rather than from labeled examples — the foundation of the training stage that turns a base model into a helpful assistant.
  8. 8.What Is Supervised Fine-Tuning (SFT)?The training stage right after pretraining, where a model learns from curated example conversations what a good response actually looks like.
  9. 9.What Is Instruction Tuning? Teaching a Model to Follow DirectionsInstruction tuning is what teaches a model to actually do what you ask, instead of just continuing your text in the most statistically likely direction.
  10. 10.What Is a Reward Model? The Judge Behind RLHFRLHF needs a way to score responses at scale — that's the reward model's job, trained specifically to predict what a human would prefer.
  11. 11.What Makes a Good AI Training Dataset?The data a model learns from shapes everything about its behavior — here's what actually separates a good training dataset from a bad one.
  12. 12.What Is Gradient Descent? How Models Actually LearnThe algorithm underneath every training run — a simple idea (adjust in the direction that reduces error) repeated billions of times.
  13. 13.What Is a Loss Function? How AI Models Know They're WrongA model needs a precise, numeric way to measure "how wrong was that" before it can improve — that's what a loss function provides.
  14. 14.What Is Overfitting? When an AI Model Memorizes Instead of LearnsA model that performs great on its training data but falls apart on anything new has a specific, well-understood problem: overfitting.
  15. 15.What Is Transfer Learning? Reusing What a Model Already KnowsInstead of training a new model from scratch for every task, transfer learning reuses a model's existing knowledge as a head start.
  16. 16.What Is Quantization? How Models Get Smaller and FasterQuantization shrinks a model's memory footprint by storing its numbers less precisely — trading a little accuracy for a lot of speed and lower cost.
  17. 17.What Is Catastrophic Forgetting in AI Models?Teaching a model something new can sometimes make it suddenly, badly worse at something it already knew — a well-known failure mode in continued training.
  18. 18.What Is Synthetic Data, and Why Do AI Labs Use It to Train Models?Some of a model's training data isn't written by humans at all — it's generated by other AI models. Here's why that's become standard practice.
  19. 19.What Is AI Alignment? Making Models Do What We Actually WantA model can be technically capable and still not do what people actually intend — alignment is the ongoing effort to close that gap.
  20. 20.What Is a Checkpoint in AI Model Training?Training a frontier model can take months — checkpoints are the safety net (and the comparison tool) that makes a run like that manageable.
  21. 21.What Is Masked Language Modeling? An Alternative to Next-Token PredictionBefore decoder-only models dominated, many language models learned by filling in blanks scattered across a sentence — a genuinely different training objective.
  22. 22.What Is Causal Language Modeling? The Training Objective Behind Modern LLMsThe core task nearly every modern LLM is trained on: given everything so far, predict what comes next — nothing more, nothing less.
  23. 23.What Is Next Sentence Prediction? An Older Auxiliary Training TaskA secondary training task from earlier language models, aimed at teaching a model to judge whether two sentences actually belong together.
  24. 24.What Is the Adam Optimizer? The Default Choice for Training Neural NetworksNearly every modern neural network is trained using this specific optimization algorithm — here's what it actually does differently from plain gradient descent.
  25. 25.What Is AdamW? The Refinement Behind Most LLM Training RunsA small but consequential fix to the Adam optimizer, correcting how it interacts with weight decay — now the more common choice for training large models.
  26. 26.What Is Stochastic Gradient Descent (SGD)?The original, simplest version of the algorithm that trains neural networks — updating parameters based on small random batches instead of the entire dataset at once.
  27. 27.What Is Momentum in Optimization? Smoothing Out Noisy Training UpdatesA technique that helps training keep moving consistently in a good direction, rather than reacting jerkily to every noisy, individual update.
  28. 28.What Is a Learning Rate Schedule? Changing How Big Each Training Step Is Over TimeThe learning rate rarely stays fixed through an entire training run — a schedule deliberately changes it over time, and getting that schedule right matters a lot.
  29. 29.What Is a Warmup Period in Training? Starting Slow on PurposeMost training runs deliberately start with a very small learning rate and ramp it up gradually — here's why jumping straight to full speed tends to backfire.
  30. 30.What Is Weight Decay? A Simple Technique to Discourage OverfittingA small, constant nudge applied during every training step, gently pulling parameters toward smaller values — a simple but effective defense against overfitting.
  31. 31.What Is Gradient Clipping? A Safety Cap on Training UpdatesA simple safeguard that caps how large any single training update is allowed to be, preventing one bad batch from derailing an entire training run.
  32. 32.What Is Batch Normalization? Stabilizing Training Using Batch StatisticsAn earlier normalization technique that stabilizes training by rescaling values based on statistics computed across the current batch — with a key limitation for language models.
  33. 33.What Is a Scaling Law in AI? Predicting Performance Before TrainingA set of empirical patterns showing that model performance improves predictably as data, parameters, and compute increase together — letting labs plan huge training runs in advance.
  34. 34.What Is Compute-Optimal Training? Balancing Model Size and DataFor a fixed training budget, there's a specific balance between model size and data volume that gets the best result — and it's not always "bigger model."
  35. 35.What Is Distributed Training? Training a Model Across Many Machines at OnceA frontier model is far too large and slow to train on a single machine — distributed training spreads the work across thousands of machines working together.
  36. 36.What Is Data Parallelism? Splitting the Data Across MachinesOne of the core strategies for distributed training: give every machine a full copy of the model, but split the training data between them.
  37. 37.What Is Model Parallelism? Splitting the Model Itself Across MachinesWhen a model is too large to fit on a single machine, model parallelism splits the model itself across multiple machines instead of just the data.
  38. 38.What Is Pipeline Parallelism? Splitting a Model Into Sequential StagesA specific form of model parallelism that splits a network into sequential stages across machines, keeping every machine busy at once instead of idle.
  39. 39.What Is Mixed Precision Training? Speeding Up Training With Lower Precision NumbersUsing lower-precision numbers for parts of training can dramatically speed things up — mixed precision training does this selectively, without sacrificing overall accuracy.
  40. 40.What Is Gradient Accumulation? Simulating a Larger Batch SizeWhen hardware memory can't fit a large batch all at once, gradient accumulation simulates one anyway by combining several smaller batches' worth of gradients.
  41. 41.What Is LoRA Fine-Tuning? Adapting a Model Without Retraining EverythingA technique that fine-tunes a model by training a small set of new parameters instead of touching the original ones — dramatically cheaper than full fine-tuning.
  42. 42.What Is Parameter-Efficient Fine-Tuning (PEFT)?The broader category of techniques that adapt a large model by training only a small fraction of its parameters — of which LoRA is the best-known example.
  43. 43.What Is an Adapter Layer? Inserting Small Trainable Modules Into a Frozen ModelA parameter-efficient fine-tuning approach that inserts small new trainable layers directly into a frozen model, rather than modifying its existing weights.
  44. 44.What Is Prompt Tuning? Learning a Prompt Instead of Writing OneInstead of hand-crafting a prompt, prompt tuning learns a set of continuous input values automatically, optimized specifically to steer the model toward a target task.
  45. 45.What Is Prefix Tuning? Learning a Steering Signal at Every LayerA close relative of prompt tuning that inserts learned values at every layer of a model, not just the input — giving it more influence over the model's behavior.
  46. 46.What Is Continual Pretraining? Extending a Model's Knowledge After the FactRather than starting a fresh model from scratch, continual pretraining resumes the pretraining process on new data, extending an existing model's base knowledge.
  47. 47.What Is Domain Adaptation? Specializing a General Model for a Specific FieldA general-purpose model often underperforms on specialized fields with their own vocabulary — domain adaptation is the process of closing that gap.
  48. 48.What Is Curriculum Learning? Training on Easy Examples Before Hard OnesBorrowing an idea from how humans learn, curriculum learning deliberately orders training examples from easy to hard, rather than presenting them in random order.
  49. 49.What Is Constitutional AI? Training a Model to Critique ItselfAn alignment technique where a model is trained to evaluate and revise its own responses against a written set of principles, reducing reliance on human-labeled examples.
  50. 50.What Is RLAIF (Reinforcement Learning From AI Feedback)?A variant of RLHF where an AI model — not a human — provides the preference feedback that drives training, dramatically cutting the cost of collecting feedback at scale.
  51. 51.What Is Proximal Policy Optimization (PPO)? The Algorithm Behind RLHFThe specific reinforcement learning algorithm most commonly used to actually implement RLHF, designed to update a model's behavior without changing it too drastically at once.
  52. 52.What Is Reward Hacking? When a Model Games Its Own Training SignalA model optimizing for a reward signal will sometimes find a way to score well without actually doing what that signal was meant to measure — a real risk in RL-based training.
  53. 53.What Is Rejection Sampling Fine-Tuning? Filtering Before TrainingA technique that generates many candidate responses, keeps only the best ones, and fine-tunes on that filtered set — a simpler alternative to full reinforcement learning.
  54. 54.What Is Best-of-N Sampling? Generating Several Answers and Picking the BestA simple technique for improving output quality: generate several candidate responses and select the best one, rather than committing to a single generation.
  55. 55.What Is Data Deduplication in Training? Removing Repeats Before Training BeginsRaw web-scale data is full of near-identical repeated content — deduplication removes it, and skipping this step can meaningfully hurt a model.
  56. 56.What Is Data Filtering in Training? Deciding What Doesn't Make the CutNot all raw data is worth training on — data filtering removes low-quality, harmful, or irrelevant content before it ever reaches the training process.
  57. 57.What Is a Training Data Mixture? Deciding How Much of Each Source to UseA model rarely trains on one single data source — the mixture (how much of each source, at what proportion) is itself a significant, deliberate design decision.
  58. 58.What Is a Pretraining Corpus? The Raw Text a Model Learns Language FromThe massive body of text used for the initial pretraining stage — where it comes from, how large it typically is, and why its composition matters so much.
  59. 59.What Is Web Crawl Training Data? The Internet as a Training SourceA large share of a model's pretraining data comes from automatically crawled web pages — here's what that actually means, and why it needs heavy cleanup.
  60. 60.What Is an Instruction Dataset? The Examples Behind Instruction TuningThe specific kind of paired data used to teach a model to follow directions — an instruction alongside the ideal response to it.
  61. 61.What Is Validation Loss? The Number That Reveals OverfittingTraining loss tells you how well a model fits data it has already seen — validation loss is the number that tells you if it actually generalizes.
  62. 62.What Is a Training Curve? Reading a Model's Progress at a GlanceA plot of loss over time is the most basic diagnostic tool in training — its shape reveals whether a run is progressing well long before training finishes.
  63. 63.What Is Early Stopping? Knowing When to Stop TrainingMore training isn't always better — early stopping halts training once further improvement on real generalization stops, before the model starts overfitting.
  64. 64.What Is Model Selection in Training? Choosing the Best CheckpointA training run produces many saved checkpoints along the way — model selection is the process of choosing which one actually deserves to ship.
  65. 65.What Is FlashAttention? Making Attention Dramatically More EfficientA widely adopted technique that computes the exact same attention calculation using far less memory and time — a genuine efficiency breakthrough, not an approximation.
  66. 66.What Is Gradient Checkpointing? Trading Compute for Memory During TrainingA technique that saves memory during training by deliberately recomputing certain values instead of storing them — trading extra computation for a smaller memory footprint.
  67. 67.What Is Sparse Training? Not Every Parameter Needs to Activate Every TimeA training approach where only a subset of a model's parameters are active for any given input, aiming to cut compute cost without sacrificing overall capacity.
  68. 68.What Is Quantization-Aware Training? Preparing a Model for Lower Precision in AdvanceRather than quantizing a model only after training finishes, quantization-aware training simulates that reduced precision during training itself, to limit the accuracy hit.
  69. 69.What Is Low-Precision Training? Training With Fewer Bits Per NumberBeyond mixed precision, some training runs push numeric precision even lower throughout — a more aggressive version of the same underlying efficiency tradeoff.
  70. 70.What Is Training Run Monitoring? Watching a Training Job While It RunsA frontier training run can take months — monitoring is the real-time practice of watching metrics and hardware health so problems get caught early, not after the fact.
  71. 71.What Is Safety Fine-Tuning? Training a Model to Handle Sensitive Requests WellA dedicated training stage focused specifically on how a model handles harmful, sensitive, or borderline requests — distinct from general helpfulness training.
  72. 72.What Is Refusal Training? Teaching a Model When (and How) to Say NoA specific component of safety training focused on teaching a model to recognize requests it shouldn't fulfill, and decline them appropriately rather than complying or being unhelpfully evasive.
  73. 73.What Is Jailbreak Resistance Training? Hardening a Model Against ManipulationA training process aimed specifically at making a model harder to manipulate into ignoring its own safety rules, using known jailbreak attempts as training examples.
  74. 74.What Is Tokenizer Training? Building the Vocabulary Before the Model Ever TrainsBefore model training begins at all, the tokenizer itself has to be built from data — a distinct, earlier step most people don't think about separately.
  75. 75.What Is Contrastive Learning? Learning by Comparing Similar and Different ExamplesA training approach that teaches a model to represent similar things as close together and different things as far apart, without needing explicit labels for every example.
  76. 76.What Is a Self-Supervised Training Objective? Learning Without Human LabelsThe category of training objective that made large-scale language model pretraining possible — learning from raw data's own structure, with no human-provided labels needed.
  77. 77.What Is Multi-Task Training? Training One Model on Many Tasks at OnceInstead of training a separate model per task, multi-task training trains a single model on many tasks simultaneously — often producing a stronger model on each one.
  78. 78.What Is Model Merging? Combining Multiple Trained Models Into OneA technique that combines the parameters of several separately trained models directly, producing a single model that inherits qualities from each, without further training.
  79. 79.What Is Weight Averaging? A Simple Form of Model MergingThe simplest possible way to combine multiple trained models: literally average their corresponding parameter values together.
  80. 80.What Is Preference Data? The Fuel Behind RLHF and DPOThe specific kind of data that alignment training actually runs on: not a single correct answer, but a judgment about which of two responses is better.
  81. 81.What Is Pairwise Comparison Data? The Specific Format Behind Preference TrainingThe concrete data format most preference training actually uses: exactly two responses shown side by side, with a single judgment about which one is better.
  82. 82.What Is an Annotator Guideline? Standardizing Human Judgment at ScaleWhen many different people are labeling data or judging responses, a shared, detailed guideline is what keeps their judgments consistent with each other.
  83. 83.What Is Elastic Weight Consolidation? Protecting Old Skills While Learning New OnesA technique that specifically protects a model's important existing parameters while it learns a new task, directly targeting the risk of catastrophic forgetting.