You ask a chatbot about your own refund policy. It answers in three tidy sentences, in a confident tone, citing a fourteen-day window. Your policy says seven. Nobody typed fourteen anywhere. Where did it come from?

This is the question we get asked most often by non-technical teams, and it deserves a straight answer — because the answer determines what you can safely build. The short version: the model wasn't looking anything up. It was doing something else entirely, and doing it exactly as designed.

What the model is actually doing

A large language model is, mechanically, a very sophisticated next-word predictor. It has read an enormous amount of text and learned the statistical shape of language — which words, phrases, and ideas tend to follow which others. When you ask it something, it produces an answer one piece at a time, each piece chosen because it plausibly continues what came before.

Think of an extraordinarily well-read colleague answering entirely from memory, at speed, without checking anything, and without the social instinct to say 'hold on, let me look'. They'd be right a lot of the time. They'd also fill small gaps smoothly, because that's what fluent recall does.

Fourteen days is a very common refund window. Statistically, it's an excellent guess. That's precisely the problem: the model has no mechanism for distinguishing 'this is the usual answer' from 'this is your answer'.

Why it sounds so sure

Here's the part that trips people up. The confidence in the writing has nothing to do with whether the content is right. The model learned tone from human text, and human text about policies is written confidently. So a wrong answer arrives in the same crisp, authoritative register as a correct one.

Every other information tool we've used sets a different expectation. A search engine that has nothing returns nothing. A database query with no match returns no rows. A form field validates. Thirty years of software have trained us to read fluent output as evidence of a lookup. With language models, that instinct is simply wrong, and it's the single most important thing to unlearn.

Where invented answers come from

• It doesn't know your specifics. Your prices, policies, staff, and product catalogue were never in its training data, so it produces the industry-average version.
• Its knowledge has an end date. Models are trained up to a cutoff. Ask about something more recent and it answers from the world as it was, with no sense that time has passed.
• The question presumes something false. Ask 'what does section 4 of the agreement say' when there is no section 4, and the model will often write you a section 4, because the question implies one exists.
• The detail is precise and unmemorable. Numbers, dates, citations, names, ID codes. These are exactly the things that can't be inferred from context — and exactly the things that get invented most.

Notice the pattern: risk goes up as answers get more specific. 'Explain what a warranty is' is safe. 'What is our warranty period on the model X-200' is not — unless you've built the part that looks it up.

The fixes that actually work

Give it the documents

The standard approach — you'll see it called RAG, retrieval-augmented generation — is simpler than the acronym suggests. Before the model answers, your system searches your own content for the relevant passages and pastes them into the question: 'Here are three sections from our policy. Using only these, answer the customer.'

Now the model isn't recalling, it's reading. It's doing the thing it's genuinely excellent at — understanding a question and finding the answer in text you provided. Most of the difference between an AI feature that works and one that embarrasses you is whether this step exists.

Let it say 'I don't know'

Sounds obvious; it's routinely skipped. If your search finds nothing relevant, the system should say so and offer a human, not pass the question through to a model with no material. Whatever comes back in that situation is a guess by construction.

Teams resist this because 'I can't find that — want me to connect you to someone?' feels like a failure. It isn't. Customers forgive not knowing. They don't forgive being told the wrong return window and finding out at the counter.

Show your sources

When an answer links to the page it came from, three things happen at once: the customer can verify it, your team can spot a bad answer in seconds instead of arguing about it, and the whole feature stops being a black box. Sources are the cheapest quality control in the entire field.

Keep it away from irreversible actions

Drafting a reply, summarising a thread, suggesting a category, ranking search results — good uses. Issuing the refund, sending the email, updating the record, changing the price — put a human in the loop, or restrict the model to proposing an action that something deterministic then validates and executes.

How to test it before customers do

Write down thirty real questions from your actual support inbox, along with the correct answers, before you build anything. That's your exam paper. Run it after every change to the system and count how many the feature gets right. It's unglamorous and it's the only way to know whether last week's 'small tweak to the prompt' quietly made things worse.

Include a few questions the system should refuse — things outside its scope, or where the answer genuinely isn't in your documents. A system that never says 'I don't know' hasn't been tuned. It's just been given permission to guess.

The reasonable middle

None of this is an argument against using AI. The technology is genuinely good at reading, summarising, drafting, classifying, and finding meaning in messy text. Those capabilities are worth real money in most businesses, and it's why we build with them.

It's an argument against one specific mistake: treating a language model as a source of truth about your business. It isn't one, was never designed to be one, and no amount of prompt-tweaking will make it one. Build the looking-up part, and the same model becomes reliable enough to put in front of customers.

The teams that get burned aren't the ones who moved too fast. They're the ones who skipped that single architectural step, because in a demo — where every question is one you already know the answer to — a model with no documents looks exactly as good as one with them.

If you're weighing an AI feature and want a frank read on whether it'll hold up in front of real customers, the discovery call is the right place. Sometimes the recommendation is a better search index and no model at all.