LLM Quiz
Questions on how large language models work, with solutions.
Notation: LLM means large language model; is sequence length, the model width, the number of layers, the number of parameters and the number of training tokens.
Architecture
1. Transformer anatomy: What are the components of a decoder-only transformer, in order, from token IDs to next-token probabilities?
Solution
Token embedding → identical blocks → final norm → linear output head → softmax. Each block is norm → self-attention → residual add, then norm → feed-forward network (a multilayer perceptron, MLP) → residual add. Attention mixes information across positions; the MLP transforms each position independently and holds about two-thirds of the parameters. The residual stream carries the hidden state through every block, so each sublayer only adds an update. The output head is often tied to the embedding matrix, and position information enters through rotary position embeddings (RoPE) in the attention or an added position embedding.
2. Encoder vs decoder: What distinguishes a transformer encoder from a decoder, and what is each used for?
Solution
An encoder attends bidirectionally, so every token sees every other; a decoder is causally masked, so each token sees only earlier ones. Encoders (BERT) build representations for classification, retrieval and embeddings. Decoders generate text token by token and dominate LLMs: one causal model can both read and write, and every position provides a training signal. Encoder–decoder models (T5, the original transformer) encode the input bidirectionally and decode with cross-attention to it, which suits tasks with a clear input and output, such as translation.
3. Pretraining objectives: What training loss does each architecture use: encoder-only, decoder-only and encoder–decoder?
Solution
Encoder-only: masked language modelling (MLM). Decoder-only: next-token cross-entropy. Encoder–decoder: cross-entropy on the target sequence, often with span corruption.
- MLM masks about 15% of tokens and predicts them from both sides.
- Next-token prediction scores every position given the tokens before it.
- Span corruption (T5) replaces random spans of the input with sentinel tokens and trains the decoder to generate the missing spans.
MLM learns only from the masked 15% of positions, while next-token prediction learns from all of them; this sample efficiency is one reason decoder-only models won at scale.
4. Cross-entropy vs KL: What is the difference between cross-entropy and Kullback–Leibler (KL) divergence, and why does minimizing either against the data give the same model?
Solution
: cross-entropy is the KL divergence plus the entropy of the data, which does not depend on the model. So minimizing cross-entropy against the data distribution is the same as minimizing , and both are maximum likelihood. They differ when the target is not one-hot: in distillation with a teacher's soft targets, KL is zero at a perfect match while cross-entropy bottoms out at the teacher's entropy. KL is also asymmetric: forward is mass-covering, while reverse , used as the penalty in reinforcement learning from human feedback (RLHF), is mode-seeking.
5. Self-attention step by step: For an input of shape and heads, walk through causal self-attention, giving the tensor shape after each step.
Solution
Project, split into heads, score, mask, softmax, mix, merge, project.
- , , : each .
- Split into heads: , with .
- Scores : .
- Causal mask: set every score where the key comes after the query to .
- Softmax over the key axis: each row becomes a distribution over earlier tokens.
- Multiply by : .
- Merge the heads back to and apply the output projection .
Intuition: each token asks a question (query), compares it with every earlier token's label (key), and takes a weighted average of their contents (values).
6. Scaling the logits: Why are attention logits divided by , where is the key dimension?
Solution
To keep the logit variance near 1 whatever the head size. sums products of roughly unit-variance terms, so its variance grows like . Unscaled, the logits grow like , the softmax saturates to near one-hot, and gradients vanish.
7. Attention cost: For self-attention over a sequence of tokens with head dimension , what are the compute and memory costs, and when does attention dominate the rest of a layer?
Solution
compute and memory for the naive score matrix. is an product and the weighted sum over costs the same again. FlashAttention keeps the floating-point operations (FLOPs) but cuts attention memory to by never storing the matrix. Per layer, attention costs about forward FLOPs against for the projections, so it dominates once exceeds about .
8. Why several heads: Why use several attention heads instead of one wide head, and what do most current models use?
Solution
Each head computes its own attention pattern, so a layer can follow several relationships at once (syntax, coreference, nearby tokens) for the same total cost. A single wide head produces one weighted average per token, which blurs competing signals. Most current LLMs use grouped-query attention (GQA): many query heads share fewer key–value heads, for example 32 query heads and 8 KV heads, which cuts the KV cache by 4× with little loss in quality. Multi-query attention (MQA, a single shared KV head) saves more memory but costs more quality.
9. Telling repeated words apart: In "the cat chased the dog", how does a transformer tell the two "the"s apart? Compare absolute and relative position encodings.
Solution
Only through positional information: attention compares content, so without it the two identical tokens would get the same representation (a causal mask leaks some order implicitly, but explicit encodings work far better). Absolute encodings add a learned or sinusoidal vector for each position to the embedding. Relative encodings depend on the distance between query and key instead: rotary position embeddings (RoPE) rotate each pair of and dimensions by an angle proportional to position, so depends only on . Relative encodings generalize better and can be stretched to longer contexts, which is why most current models use RoPE.
10. Normalization layers: Why do transformers use normalization layers? Which kinds are used and where do they go, and are they strictly necessary?
Solution
They keep activations at a stable scale through many residual blocks, which makes training stable and faster. LayerNorm subtracts the mean and divides by the standard deviation of each token's features, then applies a learned gain and bias. Root-mean-square normalization (RMSNorm) skips the mean and bias and only rescales, which is cheaper and works as well, so most LLMs use it. Pre-LN normalizes each sublayer's input and keeps a clean identity path along the residual stream, so deep models train stably; Post-LN (the original transformer) normalizes after the residual add, needs learning-rate warmup and becomes unstable with depth. Normalization is not strictly necessary: careful initialization and residual scaling, or an elementwise replacement such as a scaled tanh, can train without it, but large models keep it in practice.
11. QK-norm: What is QK-norm, and what training failure does it prevent?
Solution
Normalizing queries and keys before their dot product, which bounds the attention logits. Without it, logits can grow mid-training until attention collapses onto single tokens, causing loss spikes or divergence. It is cheap insurance and is now common in large runs.
12. SwiGLU: Define the SwiGLU feed-forward block. Why is its hidden width usually set to rather than ?
Solution
, a gated unit with three weight matrices instead of two. To match the parameters and FLOPs of a two-matrix MLP, the hidden width becomes , rounded to a hardware-friendly multiple. It is empirically about 0.1–0.2 nats better, with no agreed theoretical explanation.
13. Mixture of experts: How does a mixture-of-experts (MoE) layer work, why does it need load balancing, and how is it parallelized?
Solution
The feed-forward block is replaced by many experts, and a router sends each token to its top- of them (often 2), so the parameter count grows without growing the compute per token. The router is a linear layer followed by a softmax over experts, and a token's output is the gate-weighted sum of its chosen experts' outputs. Left alone, the router collapses onto a few experts, wasting capacity and overloading their GPUs, so an auxiliary load-balancing loss (or a per-expert bias adjusted during training) spreads tokens evenly. Expert parallelism places different experts on different GPUs and exchanges tokens with all-to-all communication before and after the expert layer.
14. Sliding-window attention: What is sliding-window attention, and how can a model with a 4,096-token window use information from further back?
Solution
Each token attends only to the previous tokens, so attention costs instead of and the KV cache is capped at entries, kept in a rolling buffer. Information still travels further through depth: layer can draw on roughly earlier tokens, so 32 layers with reach about 131k tokens. That reach is indirect and lossy, so exact long-range recall degrades; many models therefore interleave some full-attention layers with sliding-window ones.
15. FlashAttention: Why is FlashAttention faster than standard attention when it does the same number of FLOPs?
Solution
Attention is limited by memory traffic, and FlashAttention never writes the score matrix to slow GPU memory (HBM). It tiles , and into on-chip SRAM, fuses the whole multiply–softmax–multiply chain, and uses an online softmax with a running max and running sum. The backward pass recomputes scores from saved softmax statistics. HBM traffic drops from to for SRAM size .
16. Tokenizer algorithms: How do the BPE, WordPiece and Unigram tokenizers differ?
Solution
They build the vocabulary differently: BPE merges the most frequent pair, WordPiece merges the pair that most increases likelihood, and Unigram starts large and prunes. Byte-pair encoding (BPE) starts from characters or bytes and repeatedly merges the most frequent adjacent pair; with the 256 bytes as the base, nothing is ever out of vocabulary. WordPiece (BERT) merges the pair with the highest . Unigram starts from a large candidate vocabulary and repeatedly removes the pieces whose loss hurts the corpus likelihood least, and it can sample alternative segmentations. SentencePiece is a library that runs BPE or Unigram directly on raw text, treating spaces as ordinary symbols.
17. Training FLOPs: Roughly how many floating-point operations (FLOPs) does it take to train a model with parameters on tokens? Derive the rule.
Solution
. Each parameter does a multiply and an add per token in the forward pass, giving . The backward pass computes gradients with respect to both inputs and weights, about twice the forward, giving . Inference costs , and full activation recomputation raises training to about . counts non-embedding parameters.
18. Scaling laws: What are scaling laws and why do they matter? Contrast the Kaplan and Chinchilla answers, and explain why deployed models are trained well past the compute-optimal point.
Solution
Scale parameters and tokens equally, about 20 tokens per parameter (Chinchilla). Loss falls as a smooth power law in parameters, data and compute, so small runs predict large ones and tell you how to split a budget. Both studies fit . The earlier Kaplan fit put most compute into parameters (), largely because of its learning-rate schedule and excluding embeddings. Chinchilla found , shown by a 70B model on 1.4T tokens beating 280B and 530B models. Compute-optimal is not inference-optimal, so deployed models are deliberately trained far past 20:1.
19. Diffusion models: Describe the forward and reverse processes of a diffusion model, and what the network is trained to predict.
Solution
The forward process gradually adds Gaussian noise to data until only noise is left; the reverse process removes it step by step, and the network is trained to predict the noise that was added.
- Forward: with , where the fixed schedule falls from 1 to nearly 0. Any step can be sampled directly from the clean data.
- Training: pick a random , noise a clean sample, and minimize .
- Reverse: start from pure noise and use the predicted noise to step from to , either with many small random steps (DDPM, about 1,000) or fewer deterministic ones (DDIM, tens of steps).
Conditioning such as a text prompt is fed to the network as an extra input, and classifier-free guidance strengthens its effect.
Training at scale
20. Which parallelism when: Describe data, tensor, pipeline and expert parallelism. When would you use each?
Solution
Start with data parallelism; add tensor parallelism when layers are too big, pipeline parallelism when the model is too deep for one node, and expert parallelism for mixture-of-experts models.
- Data (DP): each GPU processes different data with a model copy, and gradients are all-reduced once per step. With ZeRO or FSDP (see below) the weights and optimizer state are sharded as well.
- Tensor (TP): splits each matrix multiply across GPUs and needs communication inside every layer, so it stays within a node's fast links (typically 8 GPUs).
- Pipeline (PP): places consecutive layers on different GPUs and streams microbatches through them; it tolerates slower links but leaves some GPUs idle (the bubble).
- Expert (EP): places the experts of a mixture-of-experts (MoE) layer on different GPUs and routes tokens to them with all-to-all communication.
Large runs combine them, for example TP within a node, PP across nodes and DP across replicas.
21. All-reduce: How does a ring all-reduce decompose, and how much data does each GPU send?
Solution
All-reduce = reduce-scatter + all-gather, about bytes per GPU in total, independent of the number of GPUs. Each half moves . This is why data parallelism scales, and why ZeRO-2 (zero redundancy optimizer, stage 2, which shards gradients and optimizer state) adds no communication: it simply stops after the reduce-scatter.
22. ZeRO and FSDP: What do stages 1, 2 and 3 of the zero redundancy optimizer (ZeRO) shard across data-parallel GPUs, and what does each cost in communication? Where does FSDP fit in?
Solution
Stage 1 shards the optimizer state, stage 2 also the gradients, stage 3 also the weights. Stages 1 and 2 add no communication; stage 3 adds about 50%. Plain data parallelism all-reduces the gradients, which is a reduce-scatter followed by an all-gather. Under stages 1 and 2 each GPU updates only its own shard, so it needs only the reduce-scatter of the gradients, plus an all-gather of the updated weights: the same total traffic. Stage 3 also has to all-gather each layer's weights before using them in the forward and backward passes. With stage 3, per-GPU state drops from about 16 bytes per parameter to about on GPUs. Fully sharded data parallel (FSDP) is PyTorch's implementation of stage 3.
23. Pipeline bubble: With pipeline stages and microbatches, what fraction of time is idle, and how can it be reduced?
Solution
. With : 47% idle at , 10% at . Interleaving model chunks per device gives at the cost of more messages, and zero-bubble schedules split the backward pass into input-gradient and weight-gradient parts to fill the gaps.
24. Keeping GPUs busy: How do you overlap computation and communication in large-scale training, and what else keeps GPUs fully used?
Solution
Start each communication as soon as its data is ready, and run it alongside computation instead of after it. In data parallelism, gradients are all-reduced in buckets during the backward pass, as soon as each group of layers finishes, rather than all at the end. With ZeRO-3 or FSDP, the next layer's weights are gathered while the current layer computes. In pipeline parallelism, sends to the next stage overlap with the next microbatch. Beyond communication: load and prepare batches in background workers, copy them to the GPU asynchronously, pack sequences so no compute is spent on padding, and use per-GPU batches large enough to keep matrix multiplies efficient. Model FLOPs utilization (MFU), useful FLOPs per second divided by the hardware peak, measures how well this is working.
25. What AdamW stores: What does AdamW keep in memory for each parameter in mixed-precision training, and how many bytes does that come to?
Solution
16 bytes: fp32 master weights (4) + fp32 gradients (4) + Adam's first moment (4) + second moment (4). bf16 working copies of the weights and gradients add 2–4 more. A 7B model therefore needs about 112–140 GB for state alone, which is why the state is sharded.
26. Memory budget: Where does the memory go when training a 7B model, and what runs out first on a single 80 GB GPU?
Solution
Optimizer state runs out first: 14 GB bf16 weights + 28 GB fp32 master weights + 56 GB Adam moments = 98 GB, before any activations. Sharded over 8 GPUs with ZeRO-3, that is about 12 GB per GPU, leaving room for activations, which scale with batch × sequence × width × layers. Optimizer state dominates at small scale, activations at long context, and the KV cache at inference.
27. Activation checkpointing: What is the memory–compute trade-off of activation checkpointing?
Solution
Recomputing activations in the backward pass costs about one extra forward, +33% compute (), in exchange for storing only block boundaries. Checkpointing every layers gives memory. Selective recomputation of only the cheap but memory-heavy attention region costs about 2%.
28. Adam vs AdamW: What is the difference between Adam with L2 regularization and AdamW?
Solution
AdamW applies weight decay directly to the weights, separately from the adaptive update; Adam with L2 adds the penalty to the gradient, where the adaptive scaling distorts it. With L2, the decay term is divided by along with the rest of the gradient, so weights with large past gradients are barely regularized and weights with small ones are decayed strongly. AdamW updates , so every weight decays at the same relative rate. The decoupled version generalizes better and lets the learning rate and decay be tuned more independently, which is why LLMs use AdamW.
29. Number formats: Compare bf16, fp16 and fp8 for training.
Solution
bf16 has fp32's range (8 exponent bits) with 7 mantissa bits and needs no loss scaling, so it is the default. fp16 (5 exponent, 10 mantissa bits) is more precise but needs dynamic loss scaling to avoid underflow. fp8 uses e4m3 for weights and activations and e5m2 for gradients, with fine-grained scaling; master weights, optimizer state and sensitive modules (embeddings, output head, norms, router) stay in higher precision.
30. Fused kernels: What is a fused kernel, why does it help, and what are two typical fusions?
Solution
A single GPU kernel that performs several operations without writing the intermediate results to GPU memory (HBM). Elementwise and reduction operations do little arithmetic per byte, so their cost is memory traffic and kernel launches; fusing removes both. Two typical fusions:
- Softmax in one pass: max, subtract, exponentiate, sum and divide on data kept on chip. FlashAttention extends this to the whole attention computation.
- A matrix multiply followed by its bias and activation (bias + GeLU), or a residual add followed by RMSNorm, done in one pass instead of three.
Compilers such as torch.compile and Triton generate many of these fusions automatically.
31. Hyperparameters from scratch: What are the key hyperparameters for pretraining an LLM from scratch, and what values are typical?
Solution
AdamW with , , weight decay 0.1; a peak learning rate around for models of a few billion parameters (lower for larger ones); linear warmup over about 2,000 steps, then cosine decay to about 10% of the peak; gradient clipping at 1.0; a batch of a few million tokens. Initialization uses a standard deviation around 0.02, with residual output projections scaled down by depth. rather than 0.999 lets the second moment react quickly to gradient spikes. Learning rate and batch size interact, and the maximal update parametrization (μP) makes the best learning rate transfer from small proxy models.
32. Loss spikes: What causes loss spikes in large training runs, and which optimizer setting would you change first?
Solution
Lower the learning rate first. Spikes usually come from exploding attention logits, outlier features, bad data batches, fp16 overflow, too high a learning rate, or router collapse in mixture-of-experts models. Next, lower Adam's (to 0.95) or raise , and tighten gradient clipping. Preventive tools are normalizing queries and keys before the dot product (QK-norm), z-losses on router and output logits, bf16 instead of fp16, and careful initialization. Operationally: checkpoint often, then rewind past the spike and skip or reshuffle the offending data.
33. Loss suddenly drops: Training loss suddenly drops much faster than expected. What could have happened?
Solution
Usually the data or the measurement changed, not the model. Check for repeated or leaked data (an epoch boundary, duplicated shards, evaluation data in the training set); a switch in the data mixture or curriculum to easier data; a learning-rate schedule change, such as the end of warmup or an annealing phase; or a bug in the loss (masking, averaging, padding counted as targets). Compare loss on held-out data: if it does not drop too, the training loss is misleading. Genuine sudden improvements do happen, for example when induction heads form early in training, but they also show up on held-out data.
34. Loss plateaus: Training loss stops improving early on. How do you debug it, and what is the first experiment you would run?
Solution
First, overfit a single small batch. If the loss cannot get close to zero, there is a bug, not a capacity or data problem. Then check, roughly in order: the learning rate (too low, too high, or warmup that never ends); gradient norms per layer (vanishing, or clipped on every step); the data and labels (targets not shifted, every token masked, no shuffling); initialization and normalization; and numerical precision. Only then consider model size or data quality.
Inference
35. KV cache: What is the key–value (KV) cache, why does it speed up generation, and how much memory does it need? Work it out per token for a 70B model with 80 layers and 8 KV heads of dimension 128, stored in bf16 (2 bytes).
Solution
It stores every layer's keys and values for all previous tokens, so each new token computes only its own query, key and value instead of reprocessing the whole sequence. Here it takes about 0.31 MiB per token. The size is
where the 2 counts keys and values. Here bytes per token. A single 128k-token sequence then needs about 40 GiB, more than the weights' share on one 80 GB GPU, which is why grouped-query attention, compressed (latent) caches and paged allocation matter.
36. Prefill vs decode: Why are prefill (processing the prompt) and decode (generating tokens) such different workloads?
Solution
Prefill is compute-bound; decode is memory-bandwidth-bound. Prefill processes all prompt tokens in parallel as large matrix products, costing FLOPs. Decode produces one token at a time, reading every weight from memory for a matrix–vector product, so its arithmetic intensity is about the batch size. Time to first token (TTFT) therefore depends on prefill FLOPs, and per-token latency on bytes read divided by memory bandwidth.
37. Speculative decoding: How does speculative decoding speed up generation while producing exactly the same output distribution as the target model?
Solution
A cheap draft model proposes tokens, the target scores them all in one pass, each is accepted with probability , and on the first rejection the token is resampled from renormalized. This rejection-sampling rule makes the output provably identical to sampling from the target. With acceptance rate , each step yields tokens on average. It reduces latency at small batch but can lower throughput at large batch, where the extra FLOPs are no longer free.
38. PagedAttention: What problem does PagedAttention solve, and how?
Solution
Memory fragmentation in the KV cache: it stores the cache in fixed-size blocks with a per-sequence block table, like virtual-memory paging. Contiguous pre-allocation wastes 60–80% of KV memory because sequence lengths are unknown in advance; paging cuts the waste to under 4%. Blocks can also be shared across sequences, copy-on-write, for parallel samples and common prompt prefixes. Serving throughput rises 2–4×.
39. Weight quantization: How do the two main 4-bit weight-only quantization methods, GPTQ and AWQ, work?
Solution
GPTQ quantizes a layer column by column and uses second-order (inverse Hessian) information to adjust the remaining weights to compensate; AWQ protects the ~1% of salient weight channels, found by activation magnitude, by scaling them up before quantizing. Both use a small calibration set and no retraining. AWQ needs no backpropagation and generalizes better across calibration sets.
40. Fast inference on a CPU: How would you design or adapt a language model so that it generates text quickly on a CPU?
Solution
Generating each token reads every weight once, so speed is limited by memory bandwidth, not arithmetic: reduce the bytes read per token.
- Use a smaller model, distilled from a larger one and trained on far more tokens than is compute-optimal.
- Quantize the weights to 8 or 4 bits.
- Shrink the KV cache with grouped-query attention and a sliding window.
- Use fused, vectorized kernels written for the CPU's instruction set.
- Add speculative decoding with a tiny draft model, so each pass over the weights yields several tokens.
Batch requests together where latency allows, since a batch reuses each weight read.
Evaluation
41. Tracking pretraining: Beyond training loss and perplexity, how do you track a model's progress during pretraining?
Solution
Run cheap few-shot benchmarks on regular checkpoints and watch their trends rather than single values. Typical choices are MMLU (knowledge), HellaSwag and ARC (commonsense and reasoning), GSM8K (grade-school math) and HumanEval (code), plus loss on held-out sets for each data source (code, math, other languages) to catch mixture problems. Early in training, accuracy sits at chance and is noisy, so score multiple-choice tasks by the log-likelihood the model assigns to the correct answer, which gives a smoother signal.
42. Perplexity: Define perplexity and state its main pitfall as a benchmark.
Solution
, the exponentiated mean negative log-likelihood per token. It depends on the tokenizer and corpus, so it is not comparable across models with different vocabularies; use bits per byte (total negative log-likelihood in bits divided by UTF-8 bytes) instead. It also tracks downstream ability only loosely after post-training.
43. Benchmarks: What do MMLU, GPQA, MATH, AIME, HumanEval, SWE-bench and LiveCodeBench each measure?
Solution
Knowledge, expert science, math, contest math, simple code, real software fixes and uncontaminated competitive programming, respectively.
- MMLU: 4-way multiple choice in 57 subjects (about 14k questions); saturated and partly mislabeled.
- GPQA: graduate-level science questions designed to resist web search; the Diamond subset has 198.
- MATH: 12.5k competition problems graded on the final answer.
- AIME: 30 problems per year, so each is worth 3.3 points and gaps under about 7 points are noise.
- HumanEval: 164 Python functions with unit tests; saturated above 95%.
- SWE-bench: 2,294 real GitHub issues graded by each repository's tests; agentic results usually use a 500-task human-validated subset.
- LiveCodeBench: date-stamped contest problems, so models can be tested only on problems released after their training cutoff.
44. Contamination: How do you detect and avoid benchmark contamination?
Solution
Detect it with n-gram overlap against the training corpus, canary strings, likelihood tests and verbatim-completion probes; avoid it with time-split, refreshed or private evaluations. The standard sweep checks 8–13-gram overlaps. Likelihood tests compare the canonical item with shuffled or perturbed versions. A strong tell is a model doing much better on items from before its cutoff than after.
45. Evaluating RAG: How do you evaluate a retrieval-augmented generation (RAG) system?
Solution
Evaluate retrieval and generation separately, then end to end.
- Retrieval: on queries with labelled relevant documents, measure recall@ and ranking metrics such as mean reciprocal rank or normalized discounted cumulative gain (nDCG). Did the right passages come back, and how high?
- Generation from the retrieved context: faithfulness (every claim is supported by the context), answer relevance, and correctness against reference answers, scored by exact match or F1 where possible and otherwise by a model judge or human raters.
- End to end: task accuracy and citation accuracy.
Also test questions whose answer is not in the retrieved context: a good system says so instead of guessing.
Post-training
46. SFT vs RLHF vs DPO: Describe the training flow of supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). How do they differ?
Solution
SFT imitates good responses; RLHF learns a reward model from preferences and optimizes the model against it with reinforcement learning; DPO learns from the same preferences directly, with no reward model and no sampling.
- SFT: next-token cross-entropy on prompt–response demonstrations.
- RLHF: people rank pairs of responses; a reward model is trained on those rankings; the model is then optimized with an RL algorithm such as proximal policy optimization (PPO) or group relative policy optimization (GRPO) to maximize reward minus a penalty for drifting from the SFT model. It is online: it samples from the current model.
- DPO: solves the same penalized objective in closed form and turns it into a classification-style loss on log-probability ratios of the preferred and rejected responses. It is offline, simpler and cheaper, but cannot explore beyond its data.
A typical pipeline runs SFT first, then DPO and/or RL.
47. SFT vs pretraining: How does supervised fine-tuning differ from pretraining in its data and its loss?
Solution
Same next-token cross-entropy, but on formatted conversations, and only the assistant's tokens count. Each example is a prompt and response (or a multi-turn conversation) rendered with a chat template: special tokens mark the system, user and assistant turns, and the model must learn to emit the end-of-turn token. Prompt tokens are masked out of the loss. Packed sequences need document masking so examples don't attend to each other, and averaging the loss per sequence over-weights short responses. Training uses far less data than pretraining, typically a learning rate around – for 1–3 epochs.
48. Reward-model loss: Write the reward-model loss. What about the learned reward is not identifiable?
Solution
Only differences within a prompt are identified, so the reward is defined up to a per-prompt constant and absolute values mean nothing without centering. The reward model is usually the SFT model with a scalar head, trained for about one epoch because it overfits quickly.
49. PPO objective: Write the proximal policy optimization (PPO) objective used in reinforcement learning from human feedback (RLHF).
Solution
Maximize with and . The advantage comes from generalized advantage estimation on the reward . The KL term keeps the policy where the reward model is accurate. It needs four models in memory: policy, reference, reward and value.
50. DPO: Derive the key idea and loss of direct preference optimization (DPO).
Solution
The Kullback–Leibler (KL)-regularized optimum is , so the reward can be written through the policy; substituting into the Bradley–Terry likelihood cancels the normalizer and gives
It needs no reward model, sampling or critic, but still holds a frozen reference model, and being off-policy is its main limitation compared with online reinforcement learning (RL).
51. GRPO: What is group relative policy optimization (GRPO), and why can it drop the value model?
Solution
For each prompt it samples a group of responses and uses the group-normalized reward as the advantage. The group mean is the baseline, so no learned critic is needed, saving a model's worth of memory. It then uses the clipped probability-ratio loss of proximal policy optimization (PPO) with these advantages plus a KL penalty to the reference.
52. RLVR: What is reinforcement learning with verifiable rewards (RLVR), and how does it differ from RLHF?
Solution
It replaces the learned reward model with a deterministic checker: answer matching for math, unit tests for code, rule checks for format. The policy-gradient machinery is unchanged. There is no reward-model drift and it scales to large compute, though it can still be gamed, for example by special-casing tests. It drives the recent long-reasoning models.
53. Reward hacking: What is reward hacking, and how do you detect and mitigate it?
Solution
The policy maximizes the measured reward instead of the intended behavior: longer answers, flattery, confident formatting, special-casing tests, or exploiting the judge. Detect it when reward rises while held-out human or independent-judge scores stay flat, by tracking length and KL, and by reading the top-reward samples. Mitigate with the KL penalty, reward-model ensembles, length-controlled rewards, retraining the reward model on fresh on-policy data, and verifiable rewards where possible.
54. LoRA: How does low-rank adaptation (LoRA) work, and what does it cost and save?
Solution
Freeze and learn a low-rank update, , with initialized to zero so training starts from the base model. Trainable parameters and optimizer state drop 100-fold or more, and can be merged into for no inference cost, or kept separate to swap adapters. Typical settings are –, , on all linear layers. It falls short of full fine-tuning when the task needs new knowledge or a new language.