Prompt, RAG or Fine-Tune? How to Choose

You have an AI that is not doing what you want. What do you change? Teams often jump to the most costly answer, such as training a model. There is a better order. This lesson gives you a simple ladder. You start at the cheapest step and climb only when you must.

The ladder

• Step 1: better prompt. Clearer instructions, examples and format.
• Step 2: better context. Give the model the facts it needs, with search over your documents. This is RAG.
• Step 3: a stronger or more thoughtful model. Use a bigger model, or let it think longer.
• Step 4: fine-tuning. Train on your examples for steady style and behaviour.
• Step 5: building a model from scratch. Almost nobody needs this.

Each step costs more time and money than the one before. Most problems are fixed by steps one to three.

Match the problem to the fix

• Missing or out-of-date facts: use RAG or paste the facts in. Fine-tuning is a poor way to add facts.
• Wrong tone or format every time: improve the prompt, then consider fine-tuning.
• Fails on hard multi-step problems: try a stronger model or more thinking.
• Too slow or too expensive: try a smaller model, a tuned small model or caching.
• Needs live data or actions: add tools.

Why facts and behaviour need different fixes

Facts change. Prices, policies and news are different next month. A search over your documents stays current, and it can point to its source. Behaviour, such as always answering in three bullets, is steady. Training can bake that in. Mixing these up wastes months.

Always measure

Before you change anything, write a small test sheet of real questions with good answers. Run it before and after each step. If the score does not rise, the step did not help. This is called an eval. Later lessons show how.

Problem: answers quote last year's prices.
Layer: 2 (facts).      Try first: RAG over the current price list.
Not first: fine-tuning (prices change, so training goes stale).
Measure: 30 real price questions, before and after.

A common mistake

A team trains a model on its handbook. The next month the handbook changes. The model still gives the old answers. Retraining is slow. Had they used search over the live handbook, the fix would have taken minutes.

Try it on a real case

A shop wants an assistant for returns. First, write the policy in the prompt. Test twenty real questions. If the answers are right, stop. If it misses the newest rules, add search over the live policy page. If it is right but too wordy, tighten the prompt. Only if the style is still off after all that, think about fine-tuning. Most teams never reach that step.

Write down what you tried and what happened. A short log stops the team from repeating the same test next month.