Fine-tuning,
step by step.

1. What is fine-tuning?

Starting from pre-trained weights, doing more training on smaller task-specific data.

Three ways to teach a model

Reading this: a side-by-side of the three options, sorted by how invasive they are.

Fine-tuning is the middle path: cheap enough for individuals, durable enough to ship.

2. The post-training pipeline

How a base model becomes a chatbot, in four stages.

The pipeline

Stages 1 and 4 are the bookends. Most "fine-tuning" work happens in stages 2 and 3. That is the focus of this essay.

SFT teaches what to say. Preference learning teaches taste. Together, they turn a base model into an assistant.

3. SFT: Supervised Fine-Tuning

Train on (instruction, response) pairs. Same loss as pre-training, smaller curated dataset.

The math

The optimizer is almost always AdamW (Adam with decoupled weight decay). One step:

Plain English first. AdamW keeps two running averages per weight: m_t (average gradient direction, the first moment) and v_t (average squared gradient, the second moment). The first smooths noisy gradients; the second shrinks the step for weights with large or erratic gradients. Both averages start at zero, so m̂ and v̂ divide out that initial bias. wd (weight decay) gently pulls weights toward zero each step.

The code

HuggingFace TRL's SFTTrainer. Real, runnable, current API.

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model_id  = "meta-llama/Llama-3.2-3B"
model     = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(model_id)
dataset   = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")

cfg = SFTConfig(
    output_dir              = "./sft-out",
    num_train_epochs        = 3,
    per_device_train_batch_size = 4,
    gradient_accumulation_steps = 4,    # effective batch = 16
    learning_rate           = 2e-5,
    lr_scheduler_type       = "cosine",
    warmup_ratio            = 0.03,
    max_length              = 2048,     # TRL >=0.12: use max_length (older versions: max_seq_length)
    packing                 = False,    # incompatible with completion_only_loss
    bf16                    = True,
    logging_steps           = 10,
    save_strategy           = "epoch",
    completion_only_loss    = True,     # mask prompt tokens (prompt/completion datasets only)
)

trainer = SFTTrainer(
    model           = model,
    args            = cfg,
    train_dataset   = dataset,
    processing_class = tokenizer,
)
trainer.train()
trainer.save_model("./sft-final")

The parameters, what each one changes

In order of "tweak this first if results are bad". Production-cited ranges.

Live: SFT in your browser

A tiny 1→16→1 MLP regresses y = sin(2π x) on 32 noisy points, the simplest non-linear regression that needs a hidden layer. Click "Train" to run 80 deterministic SGD steps. Move the sliders to see how each parameter shapes the loss curve.

SFT is just supervised next-token learning, masked to the response. Every other method below is a variation on this theme.

4. LoRA: Low-Rank Adaptation

Don't train the whole weight matrix. Train two skinny matrices that, multiplied, equal the change you want.

The math

Parameter count math

Concrete: a single 4096×4096 projection layer.

The code

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")

lora = LoraConfig(
    task_type     = TaskType.CAUSAL_LM,
    r             = 16,                       # rank
    lora_alpha    = 32,                       # often 2 * r
    lora_dropout  = 0.05,
    bias          = "none",
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"],
    # alternative: target_modules = "all-linear"  for max coverage
)

model = get_peft_model(model, lora)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 3,219,562,240 || trainable%: 0.21
# now plug `model` into SFTTrainer exactly as before

Parameters, what each does

Live: watch a LoRA adapter form

A 16×16 frozen weight W0 is shown as a heatmap. Move the rank slider to see B and A grow / shrink. The product BA is the proposed delta. W0 + (α/r) · BA is what the model uses.

LoRA assumes the change you want is low-rank. For most fine-tuning tasks, that assumption holds, and you save 100× on memory.

5. QLoRA: LoRA on a quantized base

4-bit the frozen base model, train LoRA adapters in 16-bit. Fit a 70B model on one GPU.

The math

The 16 NF4 bucket values (verbatim from the QLoRA paper / bitsandbytes)

Notice the spacing: tighter near 0 (where most weight values live), looser at the tails.

Memory: how much you save

Sizes for loading a Llama-7B base model. Quantization shrinks the resident weight footprint; LoRA shrinks the trainable param + optimizer state.

The code

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit              = True,
    bnb_4bit_quant_type       = "nf4",        # vs "fp4"; nf4 is better for normal weights
    bnb_4bit_use_double_quant = True,         # extra ~0.37 bit/param savings
    bnb_4bit_compute_dtype    = torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B",
    quantization_config = bnb,
    device_map          = "auto",
)
model = prepare_model_for_kbit_training(model)   # freezes base, casts norms to fp32

lora = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear",
                  lora_dropout=0.05, task_type="CAUSAL_LM")
model = get_peft_model(model, lora)

# now use SFTTrainer / DPOTrainer as before, only the LoRA adapters train

Parameters

QLoRA is just LoRA with a quantized base. The quality loss is small; the memory savings are huge. For most users with one GPU, this is the default.

6. DPO: Direct Preference Optimization

Given pairs of (chosen, rejected), nudge the model toward chosen and away from rejected. No reward model needed.

The math

Notation primer. log π(y|x) is the model's log-probability of producing response y given prompt x: at each token the model predicts a probability; we take logs (multiplications become additions) and sum across the response. Log-probs are always ≤ 0; closer to 0 = more likely. σ is the sigmoid 1/(1+e^-z), squashing any real number into (0,1). The Bradley-Terry model derives a preference probability from a score difference (think Elo). KL (Kullback-Leibler divergence) measures distance between two probability distributions; β controls how much the trained model can drift from the reference.

Intuitively: Δ is "how much more does the current model prefer chosen-over-rejected, compared to the reference". If Δ > 0, we're already there; if Δ < 0, we're worse than the reference. The loss penalizes Δ << 0 sharply (sigmoid).

The code

from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

# both: the SFT checkpoint
model     = AutoModelForCausalLM.from_pretrained("./sft-final", torch_dtype="bfloat16")
ref_model = AutoModelForCausalLM.from_pretrained("./sft-final", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("./sft-final")

# preference dataset format: {prompt, chosen, rejected}
dataset = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")

cfg = DPOConfig(
    output_dir          = "./dpo-out",
    num_train_epochs    = 1,
    per_device_train_batch_size = 2,
    gradient_accumulation_steps = 8,
    learning_rate       = 5e-7,           # MUCH smaller than SFT
    lr_scheduler_type   = "cosine",
    beta                = 0.1,            # KL strength; the key DPO hyperparameter
    loss_type           = "sigmoid",      # standard DPO; "ipo", "hinge", "kto" are variants
    max_length          = 1024,
    max_prompt_length   = 512,
    bf16                = True,
)

trainer = DPOTrainer(
    model           = model,
    ref_model       = ref_model,
    args            = cfg,
    train_dataset   = dataset,
    processing_class = tokenizer,
)
trainer.train()

Parameters

Live: a single DPO step on a tiny preference pair

Watch the loss change as you move the β slider. Toy log-probs are deterministic so you can compute the result yourself.

DPO converts pairwise preferences directly into weight updates, the simpler, almost-always-better successor to RLHF.

7. PPO & RLHF, briefly

The classic alignment recipe. Largely replaced by DPO; included for context.

The PPO math (clipped surrogate objective)

RL primer first. The policy π is the model, at each step it samples an action a_t (the next token) from a state s_t (prompt + tokens so far). After the response, a reward R is scored by the reward model. The value V(s_t) is a separate small network estimating expected future reward; subtracting it gives the advantage, how much better the action taken was than the policy's average. The importance ratio r_t = (new prob)/(old prob) measures how far the policy moved; clipping prevents drift in any one step. γ (discount) and λ (GAE smoothing) propagate later rewards back to earlier tokens.

Where the KL penalty actually goes. In RLHF the per-token reward is r_t = R_RM · 1[t = last] - β_KL · log( π_θ(a_t|s_t) / π_ref(a_t|s_t) ): the reward-model score is paid at the final token, and a per-token KL penalty pulls every step back toward the reference. The KL enters via the reward, not the advantage.

The code (sketch. PPO is significantly more complex to set up)

from trl import PPOTrainer, PPOConfig
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification, AutoTokenizer

policy        = AutoModelForCausalLM.from_pretrained("./sft-final")
ref_policy    = AutoModelForCausalLM.from_pretrained("./sft-final")        # frozen
# reward_model and value_model both have a scalar regression head (num_labels=1),
# not a causal LM head. TRL's PPOTrainer expects sequence-classification models here.
reward_model  = AutoModelForSequenceClassification.from_pretrained("./reward-model", num_labels=1)
value_model   = AutoModelForSequenceClassification.from_pretrained("./sft-final",    num_labels=1)

cfg = PPOConfig(
    learning_rate     = 1e-5,
    cliprange         = 0.2,
    cliprange_value   = 0.2,
    kl_coef           = 0.05,
    gamma             = 1.0,
    lam               = 0.95,
    num_ppo_epochs    = 4,
    per_device_train_batch_size = 8,
    mini_batch_size   = 2,
    num_mini_batches  = 4,
)
# Note: PPOTrainer's API has changed across TRL versions; the snippet above is
# illustrative, consult the TRL docs for the exact arg names in your version.

trainer = PPOTrainer(cfg, processing_class=tokenizer,
                     model=policy, ref_model=ref_policy,
                     reward_model=reward_model, value_model=value_model,
                     train_dataset=dataset)
trainer.train()

PPO/RLHF works and is real, but the operational overhead is high. Reach for it only if DPO underperforms on your task.

8. Catastrophic forgetting

The model gets better at your task, worse at everything else.

Mitigations, in increasing order of effectiveness

Live: forgetting curve, full FT vs LoRA

Synthetic illustration: as training proceeds, both approaches improve task accuracy. Full FT loses general accuracy; LoRA preserves it.

If you don't have a reason to do full fine-tuning, do LoRA. The base model stays intact.

9. Which method should I use?

An interactive decision tree. Answer 3-5 questions, get a concrete recipe.

The combinations matter. SFT alone gives you instruction-following. SFT then DPO gives you alignment. SFT then DPO with QLoRA fits on consumer hardware.

10. Method comparison matrix

All seven methods, side by side. Concrete numbers, not vibes.

Most production teams in 2026 ship with: QLoRA + SFT + DPO. That's the canonical recipe.

11. Common mistakes

Things that look fine until they don't.

Half of fine-tuning bugs are formatting bugs. The rest are LR bugs. Eval often.

12. End-to-end recipe: QLoRA SFT, then DPO

A real, copy-pasteable script for the canonical 2026 recipe.

# requirements: transformers, trl, peft, bitsandbytes, datasets, accelerate
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
from trl import SFTTrainer, SFTConfig, DPOTrainer, DPOConfig
from datasets import load_dataset

BASE = "meta-llama/Llama-3.1-8B"

# === Step 1: shared 4-bit base loader ===
def load_4bit_base():
    bnb = BitsAndBytesConfig(
        load_in_4bit              = True,
        bnb_4bit_quant_type       = "nf4",
        bnb_4bit_use_double_quant = True,
        bnb_4bit_compute_dtype    = torch.bfloat16,
    )
    model = AutoModelForCausalLM.from_pretrained(BASE,
        quantization_config = bnb, device_map = "auto")
    model = prepare_model_for_kbit_training(model)
    return model

tokenizer = AutoTokenizer.from_pretrained(BASE)
tokenizer.pad_token = tokenizer.eos_token

# === Step 2: SFT on instructions ===
sft_model = load_4bit_base()
sft_model = get_peft_model(sft_model, LoraConfig(
    r              = 16,
    lora_alpha     = 32,
    lora_dropout   = 0.05,
    target_modules = "all-linear",
    task_type      = "CAUSAL_LM",
))

sft_data = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft")

sft_trainer = SFTTrainer(
    model = sft_model,
    args  = SFTConfig(
        output_dir                  = "./sft",
        num_train_epochs            = 1,
        per_device_train_batch_size = 4,
        gradient_accumulation_steps = 4,
        learning_rate               = 2e-4,
        lr_scheduler_type           = "cosine",
        warmup_ratio                = 0.03,
        max_length                  = 2048,    # TRL >=0.12: max_length (older: max_seq_length)
        packing                     = False,   # incompatible with completion_only_loss
        bf16                        = True,
        completion_only_loss        = True,
        optim                       = "paged_adamw_8bit",
        logging_steps               = 10,
        save_strategy               = "epoch",
    ),
    train_dataset    = sft_data,
    processing_class = tokenizer,
)
sft_trainer.train()
sft_trainer.save_model("./sft-adapter")     # adapter only; ~50 MB

# === Step 3: DPO on preferences ===
# Reload base + SFT adapter as the trainable model. DPO continues updating the
# same LoRA matrices SFT trained, not a fresh adapter.
# Alt for clean separation:
#   dpo_model = dpo_model.merge_and_unload()
#   dpo_model = get_peft_model(dpo_model, fresh_lora_cfg)
dpo_model = load_4bit_base()
dpo_model = PeftModel.from_pretrained(dpo_model, "./sft-adapter", is_trainable=True)
# ref_model=None: with PEFT, TRL disables the adapter to compute reference
# log-probs, saves the entire base model in memory.

dpo_data = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")

dpo_trainer = DPOTrainer(
    model     = dpo_model,
    ref_model = None,
    args      = DPOConfig(
        output_dir                  = "./dpo",
        num_train_epochs            = 1,
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 8,
        learning_rate               = 5e-7,
        lr_scheduler_type           = "cosine",
        beta                        = 0.1,
        loss_type                   = "sigmoid",
        max_length                  = 1024,
        max_prompt_length           = 512,
        bf16                        = True,
        optim                       = "paged_adamw_8bit",
    ),
    train_dataset    = dpo_data,
    processing_class = tokenizer,
)
dpo_trainer.train()
dpo_trainer.save_model("./dpo-adapter")

~80 lines. Two adapters saved. Total cost: a few hours on a single GPU. Output: a model that follows instructions and matches preferences.

13. What we left out

Honest list of techniques worth knowing about.

Five canonical methods cover 95% of real fine-tuning. The list above is what the other 5% looks like.