Fine-Tuning an AI Model

Why Fine-Tune Instead of Prompting

Prompt engineering and retrieval-augmented generation can go a long way toward customizing an LLM's behavior without touching its weights, and are usually worth trying first since they're cheaper and faster to iterate on. Fine-tuning becomes worthwhile when a task requires a consistent output format or style across thousands of requests (making a long instruction prompt impractical), when the required behavior is hard to describe in words but easy to demonstrate with examples, or when a smaller fine-tuned model needs to match the quality of a larger general-purpose model on one specific task at lower cost.

Preparing a Training Dataset

A fine-tuning dataset typically consists of many examples of the desired input and output, formatted consistently.

{
  "messages": [
    { "role": "user", "content": "Summarize: [long article text]" },
    { "role": "assistant", "content": "[concise summary in the house style]" }
  ]
}

Dataset quality matters far more than dataset size for most fine-tuning jobs - a few hundred carefully curated, consistent examples often outperform tens of thousands of noisy or contradictory ones, since the model will learn whatever patterns are actually present in the data, including unintended ones.

Full Fine-Tuning vs Parameter-Efficient Methods

Full fine-tuning updates every parameter in the model, which can produce the most thorough adaptation but requires storing and updating a full copy of the model's weights, along with the significant GPU memory needed to hold optimizer state for every parameter. For large models this is often impractical outside of well-resourced teams. Parameter-efficient fine-tuning (PEFT) methods instead freeze most of the original model and train only a small number of additional parameters, dramatically reducing memory and storage requirements while still achieving strong task-specific performance.

LoRA

Low-Rank Adaptation (LoRA) is one of the most widely used parameter-efficient techniques. Instead of updating a model's existing weight matrices directly, LoRA freezes them and injects small, trainable low-rank matrices alongside them; only these new, much smaller matrices are trained. Because the added matrices are tiny compared to the original model, a LoRA fine-tune can often be trained on a single consumer GPU and stored as a small file (megabytes rather than gigabytes) that gets combined with the original frozen model at inference time.

Overfitting and Catastrophic Forgetting

Two failure modes are common in fine-tuning. Overfitting happens when a model trained too long or on too little data starts memorizing training examples rather than learning generalizable patterns, hurting its performance on new inputs. Catastrophic forgetting happens when aggressive fine-tuning on a narrow task causes the model to lose general capabilities it had before - for example, a model fine-tuned heavily on customer support transcripts might become noticeably worse at unrelated tasks like writing code. Techniques like using a lower learning rate, limiting the number of training epochs, and mixing in some general-purpose data help mitigate both problems.

Evaluating a Fine-Tuned Model

After fine-tuning, the resulting model should be evaluated on a held-out test set - examples not used during training - to check that improvements on the target task are real and not just memorization. It's also good practice to test the model on a broader set of general prompts to check for unintended regressions in capabilities the fine-tune wasn't meant to affect.