O&A
O&A Consulting · Answer

When is fine-tuning actually the answer?

Rarely. Think of the options as a ladder ordered by what each one commits you to: prompting commits nothing, retrieval commits an index, a written skill commits documentation, supervised fine-tuning commits labeled data and a training run, and reinforcement fine-tuning commits a grader plus a base model that already succeeds sometimes. Most problems resolve on the first two rungs, because what looks like a fine-tuning problem is usually a context problem. And the base model keeps improving underneath whatever you trained — your fine-tune doesn't.

Fine-TuningRAGDomain AdaptationLoRAModel Strategy
01 Answer

The ladder, by what it commits you to

Ordering these by cost is less useful than ordering them by commitment — what you are still carrying six months later.

  • Prompt engineering. No data. Changed in minutes. Reversible.
  • Retrieval (RAG). An index to build and keep fresh. No training run, and new knowledge lands the moment a document does.
  • Skills or written procedure. Documentation the model reads. Versioned like code, still no training run.
  • Supervised fine-tuning. Labeled examples, a training run, and upkeep — the first rung that creates an artifact needing maintenance.
  • Reinforcement fine-tuning. A grader, and a base model that already succeeds sometimes. If every attempt fails, every reward is zero and there is nothing to reinforce.
02 Answer

Why most problems stop at the second rung

The usual symptom is a model that answers confidently and wrongly about your domain. That reads like ignorance and is almost always absence: nothing put the right material in front of it at the right moment. Retrieval fixes that directly, and it has a property fine-tuning does not — when the underlying fact changes, you update a document rather than retrain.

Fine-tuning teaches behaviour well and facts poorly. If your problem is "it doesn't know our products," that is retrieval. If it is "it won't produce output in our house format no matter how I prompt it," that is a genuine candidate for fine-tuning.

03 Answer

The four reasons that do justify climbing

  • Voice and format a prompt can't hold. Consistent structure or register across thousands of generations, where prompt-level instruction keeps drifting.
  • A genuinely specialized domain. Vocabulary and reasoning patterns the generalist lacks — not facts it could retrieve.
  • Latency and cost. Distilling a large model's behaviour into a smaller one for a bounded task. Often the strongest business case, and the one least about quality.
  • A simple, bounded task. High-volume classification or extraction that never needed frontier reasoning in the first place.
04 Answer

The argument against, which keeps getting stronger

BloombergGPT is the cautionary case: 50B parameters trained from scratch on financial data, and GPT-4 subsequently beat it on almost every public financial benchmark with no finance-specific training at all. The base model improved underneath it.

That dynamic has not slowed. Context windows keep growing, inference costs keep falling, and each new frontier release erodes the case for a fine-tune built against the last one. A fine-tuned model is an asset that depreciates on someone else's release schedule — and needs maintaining the whole time.

05 Answer

If you do climb: three independent choices

"Fine-tuning" bundles three decisions people often treat as one:

  • Signal. Supervised learning from demonstrations, or reinforcement from a grader that scores attempts.
  • Weights. Full fine-tuning rewrites every parameter; PEFT methods (LoRA, QLoRA, adapters) train a small add-on for a fraction of the cost.
  • Data. Hand-written, synthetic from a task spec, or distilled from a stronger model.

You pick one from each, so LoRA + supervised and LoRA + reinforcement are both ordinary combinations. And whatever the source, the dataset needs a gate: generate several candidates, keep only those passing an objective check — matched against the source, executed as code, a citation that resolves, a known label. The gate after the generator is what does the work.

06 Further

Where this is argued properly

The short answer above is ours, and we've written at length about how we got to it:

How do you evaluate a RAG system? →A better model won't fix a vague question ↗Goldens aren't ground truth ↗