LLM Concepts

How large language models are built, trained, served and tuned.

LLM Concepts

How large language models are built, trained at scale, served, evaluated and aligned, with the key formulas and numbers for each stage.

Architecture

Transformer layer – A decoder-only language model stacks LL identical layers over token vectors of width dd, the model dimension. Each layer has an attention block, which mixes information across positions, and a feed-forward block (MLP, multi-layer perceptron), which transforms each position independently. Each block is wrapped in a residual connection, x←x+block(x)x \leftarrow x + \text{block}(x).

Attention – Each token produces a query qq, key kk and value vv by linear projection. With Q,K,V∈Rn×dkQ, K, V \in \mathbb{R}^{n \times d_k} for a sequence of nn tokens,

Attn⁡(Q,K,V)=softmax⁡(QK⊤dk+M)V\operatorname{Attn}(Q,K,V) = \operatorname{softmax}\Big(\frac{QK^\top}{\sqrt{d_k}} + M\Big)V

where the causal mask MM is −∞-\infty above the diagonal, so a token sees only earlier tokens.

q⋅kq \cdot k is a sum of dkd_k roughly unit-variance terms, so its variance grows like dkd_k. Dividing by dk\sqrt{d_k} keeps the logits near unit variance and stops the softmax from saturating, where gradients vanish.

Attention cost – Computing QK⊤QK^\top and then multiplying by VV takes O(n2d)O(n^2d) time per layer, and storing the n×nn \times n score matrix takes O(n2)O(n^2) memory per head. Per layer the forward pass spends about 4n2d4n^2d FLOPs (floating-point operations) on attention scores against 8nd28nd^2 on the projections, so attention dominates only once nn is comparable to or above 2d2d.

Multi-head attention variants – Attention is split into hh heads of width dhead=d/hd_{\text{head}} = d/h, each with its own projections.

VariantQuery headsKey/value headsEffect
MHA (multi-head attention)hhhhthe original
MQA (multi-query attention)hh11KV cache hh times smaller, some quality loss
GQA (grouped-query attention)hhgg, each shared by h/gh/g query headse.g. 64 query and 8 KV heads: 8× smaller cache at near-MHA quality
MLA (multi-head latent attention)hhrebuilt per head from one cached latent vectorcache as small as GQA, quality at or above MHA

In MLA (from DeepSeek-V2), each token is compressed to a latent vector of about 512 dimensions plus a small 64-dimensional positional key, and only these are cached. The up-projections that rebuild per-head keys and values are folded into the query and output matrices, so decoding pays no decompression cost.

KV cache – During generation, the keys and values of past tokens are stored so they are not recomputed. Its size is

bytes=2×L×nkv heads×dhead×sequence length×batch×bytes per number\text{bytes} = 2 \times L \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{sequence length} \times \text{batch} \times \text{bytes per number}

Example: a 70B model with 80 layers, 8 KV heads of width 128, in bf16 (2 bytes) stores 2×80×8×128×2=327,6802 \times 80 \times 8 \times 128 \times 2 = 327{,}680 bytes, about 0.31 MiB, per token. A single 128k-token sequence needs about 40 GiB.

Rotary position embedding (RoPE) – Each consecutive pair of query and key coordinates is rotated by angle mθim\theta_i, where mm is the token position and θi=10000−2i/d\theta_i = 10000^{-2i/d}. Since rotations satisfy R(m)⊤R(n)=R(n−m)R(m)^\top R(n) = R(n - m), the attention score depends only on the relative offset n−mn - m. It adds no parameters and is applied in every layer.

Extending the context window – Ways to run a RoPE model beyond its trained length:

MethodIdea
Position interpolationdivide all positions by the scale factor ss; simple, but blurs fine local detail
NTK-aware scalingenlarge the RoPE base instead, so high-frequency dimensions are barely changed
YaRNinterpolate only the low-frequency dimensions, keep the high-frequency ones, and add an attention temperature of about 0.1ln⁡s+10.1\ln s + 1; reaches 128k with roughly 10× fewer training tokens than interpolation

Learned absolute position embeddings cannot extrapolate at all. ALiBi (attention with linear biases) adds a per-head penalty proportional to distance and extrapolates smoothly, but behaves like a recency prior and is weaker at long-range retrieval.

Normalization placement – Post-LN applies layer normalization after the residual addition, which rescales the residual stream every layer and becomes unstable in deep models without careful warmup. Pre-LN normalizes the input of each block instead, leaving a clean identity path for gradients. Pre-LN is the default.

RMSNorm – Layer normalization without the mean subtraction or bias: x/mean⁡(x2)+ϵ⋅gx / \sqrt{\operatorname{mean}(x^2) + \epsilon} \cdot g, with a learned gain gg. It is cheaper and works as well, so most current models use it.

QK-norm – Normalizing queries and keys before their dot product bounds the attention logits. This prevents logits from growing during training until attention collapses onto single tokens, a common cause of loss spikes.

SwiGLU – A gated feed-forward block, FFN(x)=(SiLU⁡(xW1)⊙xW3)W2\text{FFN}(x) = (\operatorname{SiLU}(xW_1) \odot xW_3)W_2, where SiLU⁡(z)=z σ(z)\operatorname{SiLU}(z) = z\,\sigma(z) and ⊙\odot is elementwise multiplication. It has three weight matrices instead of two, so the hidden width is set to dff=83dd_{\text{ff}} = \tfrac83 d to match the parameter count of a standard MLP with dff=4dd_{\text{ff}} = 4d.

Mixture of experts (MoE) – The feed-forward block is replaced by EE smaller "expert" networks. A linear router scores the experts for each token, the top kk are run, and their outputs are combined with softmax-normalized weights. Total parameters grow with EE but compute per token grows only with kk.

  • Load balancing – Without a penalty the router collapses onto a few experts. The usual auxiliary loss is αE∑ifiPi\alpha E\sum_i f_iP_i, where fif_i is the fraction of tokens sent to expert ii, PiP_i its mean router probability, and α≈10−2\alpha \approx 10^{-2}. A router z-loss penalizes large router logits for stability.
  • Capacity factor – Each expert has a fixed buffer of C⋅Tk/EC \cdot Tk/E slots for TT tokens, with CC around 1.0–1.25. Tokens that arrive at a full expert are dropped and skip the block through the residual connection.
  • Shared and fine-grained experts – Many small experts plus one or more shared experts that every token uses, so routed experts can specialize. A later refinement replaces the auxiliary loss with per-expert biases on the router logits, adjusted online toward balance.

Parameter count – Per layer, attention has about 4d24d^2 parameters (query, key, value and output projections) and the MLP about 8d28d^2, so

N≈12Ld2N \approx 12Ld^2

plus VdVd for an embedding table with vocabulary size VV.

Training compute – Each parameter costs about 2 FLOPs (a multiply and an add) per token in the forward pass and about 4 in the backward pass, so training on DD tokens costs

C≈6NDC \approx 6ND

Inference costs about 2N2N FLOPs per token. Recomputing activations during the backward pass raises training to about 8ND8ND.

Scaling laws – Loss is fitted as L(N,D)=E+A/Nα+B/DβL(N, D) = E + A/N^\alpha + B/D^\beta. The Chinchilla result is that for a fixed compute budget, NN and DD should grow together, each as C0.5C^{0.5}, which works out to about 20 training tokens per parameter. Models meant for heavy use are deliberately trained far beyond this ratio, since a smaller model is cheaper to serve.

μP (maximal update parametrization) – Initialization scales, output multipliers and per-layer learning rates are set as functions of width so that activations and updates stay the same size as the model widens. Optimal hyperparameters then stay fixed across widths, so they can be tuned on a small model and transferred to a large one.

Tokenization – Byte-pair encoding (BPE) starts from a base vocabulary and repeatedly merges the most frequent adjacent pair. Byte-level BPE uses the 256 byte values as the base, so every input can be encoded and no "unknown" token is needed, at the cost of longer encodings for non-Latin scripts. Tokenization causes artifacts such as inconsistent digit splitting, and rare under-trained "glitch" tokens.

FlashAttention – Attention is limited by memory bandwidth, not arithmetic. FlashAttention splits QQ, KK and VV into tiles that fit in fast on-chip memory (SRAM), fuses the score, softmax and value steps, and uses an online softmax (a running maximum and running sum, rescaling earlier partial results), so the n×nn \times n matrix is never written to slower main GPU memory (HBM, high-bandwidth memory). The FLOPs are unchanged, but memory drops to O(n)O(n) and the kernel runs much faster.

Sliding-window and linear attention – Sliding-window attention restricts each token to the last WW tokens, costing O(nW)O(nW) and capping the KV cache at WW. Stacking LL layers still gives a receptive field of about WLWL, e.g. about 131k for W=4096W = 4096 and 32 layers. Linear attention and state-space models (such as Mamba) replace softmax attention with a fixed-size recurrent state: linear-time training and constant memory per generated token, but limited exact recall. Production designs mix them with a few full-attention layers.

Training at scale

Memory per parameter – Training with Adam keeps fp32 (32-bit) master weights, gradients, and two moment estimates mm and vv:

4+4+4+4=16 bytes per parameter4 + 4 + 4 + 4 = 16 \text{ bytes per parameter}

Mixed precision adds 16-bit copies of the weights and gradients, about 18–20 bytes per parameter. A 7B model therefore needs over 100 GB before any activations, more than one 80 GB GPU.

At small scale optimizer state dominates memory; at long context, activations; at inference, the KV cache.

Collective operations – An all-reduce sums a tensor across NN devices and gives everyone the result. A ring all-reduce sends about 2Ψ2\Psi bytes per device for a tensor of Ψ\Psi bytes, regardless of NN. It splits into a reduce-scatter (each device ends with one summed shard) followed by an all-gather (each device collects all shards).

Data parallelism (DP) – Every device holds the full model, processes different data, and all-reduces gradients each step.

ZeRO and FSDP – ZeRO (zero redundancy optimizer) shards training state across the NN data-parallel devices instead of replicating it. Stage 1 shards optimizer state (about 4× saving), stage 2 also gradients (about 8×), and stage 3 also parameters (memory falls as 1/N1/N). Stages 1–2 cost no more communication than plain data parallelism; stage 3 gathers parameters in both passes, about 1.5× the volume. PyTorch FSDP (fully sharded data parallel) is ZeRO stage 3.

Tensor parallelism (TP) – Each weight matrix is split across devices. In the MLP the first matrix is split by columns and the second by rows, so one all-reduce completes the block; attention is split by heads, with one all-reduce. That is two all-reduces of activations per layer in each direction, so tensor parallelism stays within one node on fast links, at most 8 ways. Sequence parallelism additionally splits the normalization and residual regions along the sequence dimension so their activations also shrink.

Pipeline parallelism (PP) – Layers are split into PP consecutive stages on different devices, and each batch is cut into MM microbatches that flow through. The idle fraction, the "bubble", is

P−1M+P−1\frac{P - 1}{M + P - 1}

so P=8P = 8 wastes 47% at M=8M = 8 and 10% at M=64M = 64. The 1F1B schedule alternates one forward and one backward pass, keeping only about PP microbatches of activations in memory instead of MM.

Context parallelism – The sequence is split across devices. In ring attention, each device keeps its query block and passes key/value blocks around a ring, combining partial results with online softmax, so context length scales with the number of devices.

Expert parallelism (EP) – Experts sit on different devices, and tokens travel to them by all-to-all communication. Uneven routing makes the busiest expert the bottleneck.

Choosing a layout – Put the most communication-heavy parallelism on the fastest links: tensor parallelism inside a node, pipeline parallelism across nodes, and sharded data parallelism outermost. Use enough microbatches, M≥4PM \ge 4P, to keep the bubble small, and gradient accumulation to reach the target batch of millions of tokens.

Number formats – bf16 has 8 exponent bits (the same range as fp32) and 7 mantissa bits, so it needs no loss scaling and is the default. fp16 has 5 exponent and 10 mantissa bits and needs dynamic loss scaling to stop small gradients underflowing to zero. fp8 comes as e4m3 (more precision, for weights and activations) and e5m2 (more range, for gradients) and needs fine-grained scaling factors.

Activation checkpointing – Store only the inputs to each block and recompute the rest during the backward pass. It costs one extra forward pass (about +33% compute) and greatly reduces activation memory. Selective recomputation of just the attention softmax costs only a few percent.

MFU (model FLOPs utilization) – The fraction of the hardware's peak arithmetic spent on useful model computation:

MFU=6N×tokens per secondpeak FLOP/s per device×devices\text{MFU} = \frac{6N \times \text{tokens per second}}{\text{peak FLOP/s per device} \times \text{devices}}

An H100 GPU peaks at about 989 TFLOP/s in dense bf16. Well-tuned large runs reach roughly 35–50%.

Loss spikes – Sudden jumps in training loss. Common causes are growing attention logits, outlier activations, bad data batches, fp16 overflow, too high a learning rate, or router collapse. Fixes include gradient clipping at 1.0, QK-norm, z-losses, bf16, lower Adam β2\beta_2, and rewinding to a checkpoint before the spike while skipping the offending data.

Inference

Prefill and decode – Prefill processes the whole prompt in parallel with large matrix multiplications and is compute-bound. Decode produces one token at a time: every step reads all weights from memory to multiply them by a single vector, so it is memory-bandwidth-bound. Time to first token (TTFT) is set by prefill; time per output token is set by bytes read divided by memory bandwidth.

Arithmetic intensity – The FLOPs a GPU can perform per byte it reads from memory, peak FLOP/s divided by bandwidth. For an H100 it is 989×1012/3.35×1012≈295989 \times 10^{12} / 3.35 \times 10^{12} \approx 295 FLOPs per byte. A decode step does about 2 FLOPs per weight byte per sequence, so it only becomes compute-bound at a batch size of about 150–300. Below that, extra requests in a batch are almost free.

Example: a 70B model in bf16 has 140 GB of weights, so it needs two 80 GB GPUs. Each reads about 70 GB per step at 3.35 TB/s, about 21 ms, giving roughly 35–45 tokens per second at batch size 1. Storing weights in 4 bits roughly quadruples that.

Speculative decoding – A small draft model proposes γ\gamma tokens; the large target model scores all of them in one forward pass. Each proposed token xx is accepted with probability min⁡(1,p(x)/q(x))\min(1, p(x)/q(x)), where pp and qq are the target and draft probabilities. At the first rejection, a replacement is sampled from the normalized residual max⁡(0,p−q)\max(0, p - q). This makes the output distribution exactly the target model's. With acceptance rate α\alpha, the expected tokens per target pass is

1−αγ+11−α\frac{1 - \alpha^{\gamma+1}}{1 - \alpha}

It reduces latency at small batch sizes but can lower throughput at large ones, where the extra FLOPs are no longer free. Variants replace the draft model with extra prediction heads (Medusa, EAGLE) or with n-gram lookup in the prompt.

PagedAttention – The KV cache is stored in fixed-size blocks (e.g. 16 tokens) addressed through a per-sequence block table, like virtual memory pages. This cuts wasted memory from 60–80% to under 4%, and lets sequences share blocks, for example a common system prompt (prefix caching).

Continuous batching – The scheduler adds and removes requests after every decode step rather than waiting for a whole batch to finish. Chunked prefill splits long prompts into pieces that interleave with decode steps, which keeps per-token latency stable.

Weight quantization – Storing weights in 3–4 bits after training. GPTQ quantizes one column at a time and uses second-order information from a small calibration set to adjust the remaining weights to compensate. AWQ (activation-aware weight quantization) finds the roughly 1% of channels with large activations and scales them up before quantizing so they keep their precision.

Activation quantization – Using 8-bit weights and activations (W8A8) speeds up compute-bound work such as prefill. The difficulty is outliers: a few hidden dimensions carry values 20–100× larger than the rest. LLM.int8() keeps those dimensions in 16 bits; SmoothQuant moves the scale from activations into the weights offline.

KV cache quantization – Storing keys and values in 8 bits halves the dominant memory term at long context. Keys are more sensitive than values, so they usually get per-token scales.

Throughput versus latency – Throughput (tokens per second per GPU) improves with large batches, paged caches, prefix caching and quantization. Latency improves with small batches, tensor parallelism (each GPU reads only a slice of the weights), speculative decoding and chunked prefill. Disaggregated serving runs prefill and decode on separate GPU pools sized for compute and for memory respectively.

Evaluation

Perplexity – The exponentiated average negative log-likelihood per token:

PPL=exp⁡(−1T∑t=1Tlog⁡p(xt∣x<t))\text{PPL} = \exp\Big(-\frac1T\sum_{t=1}^T\log p(x_t \mid x_{\lt t})\Big)

It depends on the tokenizer, so models with different vocabularies are compared in bits per byte: total negative log-likelihood in bits divided by the number of bytes of text.

pass@k – The probability that at least one of kk samples is correct. With n≥kn \ge k samples of which cc are correct, the unbiased estimate is

pass@k=1−(n−ck)/(nk)\text{pass@}k = 1 - \binom{n - c}{k} \Big/ \binom{n}{k}

Pass@1 with greedy decoding measures single-attempt accuracy; maj@k (majority vote over kk samples) measures accuracy at a fixed extra inference budget; pass@k at large kk measures coverage and overstates usable capability.

Common benchmarks –

BenchmarkWhat it measures
MMLU4-choice questions across 57 subjects; saturated and partly mislabeled
GPQAgraduate-level science questions designed to resist web search; its Diamond subset has 198 items
MATH, AIMEcompetition math with answer checking; AIME has only 30 problems per year, so average many samples
HumanEval164 Python functions with unit tests; saturated
SWE-benchreal repository issues fixed with patches, graded by the repository's own tests
LiveCodeBenchdate-stamped new problems, so evaluation can use only items after the training cutoff

Statistical uncertainty – Treat accuracy as binomial, with standard error p(1−p)/n\sqrt{p(1-p)/n}. At n=200n = 200 and p=0.5p = 0.5 that is 3.5 points, so small gaps are noise. Compare models on the same items (paired), and average over samples for generated answers.

Contamination – Benchmark items leaking into training data. Detect it by n-gram overlap between training data and test sets, hidden canary strings, and testing whether the model prefers the original ordering of an item over shuffled variants. Avoid it with time-split or private test sets. A model much better on older items than on newer items of the same benchmark is a warning sign.

Model-as-judge biases – Language models used to grade outputs favor the first or second position, longer answers, and outputs from their own model family, and they check math poorly. Mitigations: swap positions and average, control for length, use a judge from a different family, and give reference answers. Strong judges agree with humans about as often as humans agree with each other, roughly 80%.

Pairwise leaderboards – Human pairwise votes are fitted with a Bradley–Terry model, P(i beats j)=σ(ri−rj)P(i \text{ beats } j) = \sigma(r_i - r_j) with σ\sigma the logistic function, and reported on an Elo scale where a 400-point gap means 10:1 odds. They measure preference rather than correctness and reward style.

Long-context evaluation – Needle-in-a-haystack tests retrieval of one hidden fact from a long filler text, which models now ace. Harder suites add multiple needles, multi-hop tracing and aggregation, and show that the effective context length is usually much shorter than the advertised one.

Trusting an improvement – Before believing a benchmark gain, check the new data for contamination, confirm the evaluation setup is identical (prompt format, few-shot examples, answer extraction and scoring can each move results by several points), check the gain is larger than noise, and confirm it shows up on held-out benchmarks without regressions elsewhere. Repeatedly picking checkpoints on one small public suite overfits to it.

Post-training

Pipeline – A pretrained model is turned into an assistant in stages: supervised fine-tuning (SFT) on demonstrations, preference tuning with either a reward model and reinforcement learning or a direct method such as DPO, and reinforcement learning with verifiable rewards on math and code. Rejection sampling or distillation often generates SFT data, and a final pass tunes safety behavior.

SFT – Ordinary next-token cross-entropy, but computed only on the response tokens; the prompt gets no loss. Packed sequences need masks so examples don't attend to each other. Typical learning rates are 10−510^{-5} to 2×10−52 \times 10^{-5} for 1–3 epochs.

Reward model – The SFT model with a scalar output head, trained on pairs where a response ywy_w was preferred to yly_l:

L=−log⁡σ(r(x,yw)−r(x,yl))\mathcal{L} = -\log\sigma\big(r(x, y_w) - r(x, y_l)\big)

This is the Bradley–Terry likelihood, so only differences within a prompt are meaningful; absolute reward values are not.

RLHF with PPO – Reinforcement learning from human feedback (RLHF) optimizes the policy (the model) πθ\pi_\theta with proximal policy optimization (PPO). With the probability ratio ρt=πθ(at∣st)/πold(at∣st)\rho_t = \pi_\theta(a_t \mid s_t)/\pi_{\text{old}}(a_t \mid s_t) and an advantage estimate A^t\hat A_t (how much better an action was than expected), PPO maximizes

E[min⁡(ρtA^t, clip⁡(ρt,1−ϵ,1+ϵ)A^t)],ϵ≈0.2\mathbb{E}\big[\min\big(\rho_t\hat A_t,\ \operatorname{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\hat A_t\big)\big], \qquad \epsilon \approx 0.2

The reward is rRM−β KL(πθ∥πref)r_{\text{RM}} - \beta\,\mathrm{KL}(\pi_\theta \Vert \pi_{\text{ref}}), where the Kullback–Leibler penalty keeps the policy close to a frozen reference model, inside the region where the reward model is reliable. It needs four models in memory: policy, reference, reward and value.

DPO – Direct preference optimization skips the reward model. The optimal policy under the KL-regularized objective satisfies π∗(y∣x)∝πref(y∣x) er(x,y)/β\pi^*(y \mid x) \propto \pi_{\text{ref}}(y \mid x)\,e^{r(x,y)/\beta}. Solving for rr and substituting into the Bradley–Terry loss gives

LDPO=−log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))\mathcal{L}_{\text{DPO}} = -\log\sigma\Big(\beta\log\frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta\log\frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\Big)

It trains directly on fixed preference pairs (off-policy), which is simpler but usually weaker than online reinforcement learning.

DPO variants –

MethodChange
IPOsquared loss toward a target margin, so near-certain preferences can't push the policy to extremes
KTOlearns from individually labeled good or bad responses instead of pairs
ORPOadds an odds-ratio penalty to the SFT loss; one stage, no reference model
SimPOuses the length-normalized average log-probability as the reward, plus a margin; no reference model

GRPO – Group relative policy optimization samples a group of GG answers per prompt and uses each answer's normalized reward, A^i=(ri−mean⁡(r))/std⁡(r)\hat A_i = (r_i - \operatorname{mean}(r))/\operatorname{std}(r), as its advantage. The group mean replaces a learned value model, saving memory. Dividing by response length and by the reward standard deviation introduces biases (favoring long wrong answers and very easy or very hard prompts), which later variants remove.

DAPO refinements – Four fixes for reinforcement learning on reasoning: an asymmetric clip range (upper bound about 0.28) so unlikely tokens can still grow; resampling groups where every answer is right or every answer is wrong, since they carry no gradient; averaging the loss over tokens rather than responses; and a soft penalty for overly long answers instead of scoring truncated ones as wrong.

RLVR – Reinforcement learning with verifiable rewards replaces the reward model with a checker: exact answer match for math, unit tests for code. The reward can't drift and scales to large compute, though it can still be gamed, for example by special-casing tests. Applied directly to a base model, it can produce long chains of reasoning, self-checking and backtracking from the reward alone. Small models distilled from such a reasoning model's outputs can beat the same small models trained with reinforcement learning directly.

Reward hacking – The policy maximizes the measured reward instead of the intended behavior: longer answers, flattery, confident formatting, special-casing tests. Warning signs are reward rising while independent evaluation stays flat, and growing length or KL divergence. Mitigations are the KL penalty, reward-model ensembles, length control, retraining the reward model on fresh samples, and verifiable rewards.

Rejection sampling (best-of-n) – Sample nn answers, keep the best by reward or checker, and fine-tune on them. It is simple and stable but only reinforces whole answers the model already produces, and its gains level off as nn grows.

Distillation – Training a smaller student from a teacher: by fine-tuning on the teacher's sampled outputs, by matching the teacher's next-token distributions (more sample-efficient, but needs teacher probabilities and a shared tokenizer), or on-policy, by sampling from the student and scoring with the teacher, which avoids train–test mismatch.

Constitutional AI and RLAIF – Reinforcement learning from AI feedback (RLAIF): the model critiques and revises its own answers against written principles, and the revisions become SFT data. An AI judge prompted with the same principles then labels preference pairs for reward modeling. This scales safety supervision with few human labels; over-refusal is controlled by mixing in helpfulness data.

LoRA – Low-rank adaptation freezes a weight matrix WW and learns a low-rank update:

h=Wx+αrBAx,B∈Rd×r, A∈Rr×kh = Wx + \frac{\alpha}{r}BAx, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k}

BB starts at zero, so training starts from the original model. Trainable parameters and optimizer state shrink by over 100×, and BABA can be merged into WW for no inference cost. Typical settings are rank rr from 8 to 64 with α=2r\alpha = 2r, applied to all linear layers. It matches full fine-tuning for style and format changes but falls short for large shifts such as a new language. QLoRA keeps the frozen base in a 4-bit format suited to normally distributed weights (NF4), quantizes the scaling constants too, and pages optimizer state to CPU memory, which allows fine-tuning a 65B model on one 48 GB GPU.

Chat templates – The exact token format a model was trained on for system, user and assistant messages, including special tokens and the trailing marker that signals the assistant's turn. A mismatch between training and serving silently degrades quality. Common bugs: omitting the generation prompt, adding the beginning-of-sequence token twice, or a wrong end-of-sequence token so generation never stops. Use the tokenizer's own template on both sides and inspect the tokenized output once.