Lesson 1: Training Data Versus Live Retrieval

Two Ways an Assistant Answers About You

When someone asks an AI assistant "Who makes the best invoicing software for freelancers?" or "Is Acme Plumbing reliable?", the answer comes from one of two places, or from both:

1. Training data. This is what the model learned before it was released. The model has no live link to those pages. It writes from patterns it absorbed, so it shows no sources, gives no links and cannot know anything after its knowledge cutoff. Vendors publish cutoff dates in their model documentation. The gap between the cutoff and today is often many months.
2. Live retrieval. At answer time the assistant runs a web search, reads the pages it finds, writes the answer from them and cites them. Perplexity runs a live search on nearly every query and shows numbered citations. ChatGPT search shows inline citations and a Sources panel. Claude adds numbered inline links when web search is on.

Think of a consultant answering from memory compared with one who looks it up and hands you the references. Only the second gives you somewhere to click.

Why Only Retrieval Sends Clicks

A training-based answer has no link, so nobody can click through to your site. If the assistant mentions your brand from memory, the reader may remember the name, but you get no visit and no referral to measure.

A retrieval-based answer links to its sources. If your page is one of them, readers can click through to you. If a competitor's page or a third-party review is cited instead, that site gets the click.

The practical rule is this: you can only win clicks from AI answers by becoming a source that gets retrieved and cited.

Task 1: Classify an Answer

Ask any assistant about your brand. Then check the answer against this list:

• Citations: Are there numbered markers, inline links or a Sources panel? These point to retrieval.
• Dates: Does it mention recent events, current prices or this year's products? Recent facts point to retrieval. Old facts or "as of my last update" point to training.
• Links: Can you click through to a real page that supports the claim? A working link that backs up the claim points to retrieval.

Then label the answer:

• Training-based: no citations, no links, and facts that may be out of date.
• Retrieval-based: each main claim is tied to a cited source.
• Mixed: some claims are cited, while others, often general background, come from the model's memory without a source. Mixed answers are common. Check which claims have citations and which do not.

Task 2: Run a Baseline With Search On and Off

This baseline is the starting point for every later lesson.

1. Write 10 prompts. Make 5 of them brand prompts ("What is [Brand]?", "Is [Brand] good for [use]?", "[Brand] pricing", "[Brand] vs [Competitor]", "Who founded [Brand]?"). Make the other 5 category prompts with no brand name ("Best [product type] for [audience]", "How do I choose a [service] in [city]?").
2. Pick one or two assistants you can switch between modes. In ChatGPT, you can turn off web search in its settings and run a search by hand with the web button in the chat box. In Claude, you can turn web search on or off before you send a prompt. Perplexity searches on nearly every query, so use it only for the "search on" runs.
3. Run all 10 prompts with search ON. Start a new chat for each prompt so earlier answers do not shape later ones.
4. Run all 10 prompts again with search OFF.
5. Record the results in a spreadsheet with these columns: prompt, assistant, mode, date run, brand mentioned (Y/N), facts correct (Y/N/partly), classification (training/retrieval/mixed), URLs cited, and whether your own site was cited.
6. Compare the two runs. Look for out-of-date facts in search-off answers, competitors that appear only when search is on, and which websites are cited instead of yours.

AI answers vary from one run to the next, so treat one run as a snapshot, not a final verdict. Keep the sheet. You will re-run the same prompts later.

Why Fixes Move at Different Speeds

Suppose an assistant says your company still offers a product you dropped last year.

• If the error comes from training data, you cannot edit the model. The fix only arrives when the vendor trains and releases a newer model with a later cutoff, and that new model has to have picked up corrected sources. That takes months, and you do not control when it happens.
• If the error comes from retrieval, the assistant is repeating a page it found today. If you update that page, or ask the owner of an outdated third-party page to update theirs, the next retrieval can pick up the corrected version. Results can show within weeks, depending on how quickly the page is crawled again.

This is why classifying an answer matters. It tells you whether a fix is something you can act on now or something you have to wait for.

How to Measure It

From your baseline sheet, work out three numbers:

• Accuracy rate: the share of answers with correct brand facts, for search on and for search off.
• Citation share: the share of search-on answers that cite your own domain.
• Retrieval gap: the number of prompts where search-on and search-off answers disagree about you.

Re-run the same prompts on the same assistants every month. Fixes you make to retrieved sources should show up in the search-on numbers first.

Key Takeaways

• Assistants answer from training data (memory, no links), live retrieval (fetched pages with citations) or a mix of both.
• Citations, recent dates and working links show that an answer used retrieval.
• Only answers that cite your pages can send you clicks.
• A 10-prompt baseline with search on and off shows where you are now.
• Errors from retrieval can be fixed within weeks by correcting the source pages. Errors from training data last until a newer model is released.