Ask an LLM to write a paragraph. Great. Ask it to summarise a document. Fine. Ask it to calculate your client's CTR based on raw campaign data. That's where things go wrong.
And the frustrating part: it won't tell you it's guessing. It will hand you a confident, cleanly formatted number with zero disclaimer. No uncertainty. Just a wrong answer that looks exactly like a right one.
Here's why.
LLMs don't calculate. They predict what a calculation looks like.
When you ask an LLM "what is 1,847 divided by 23", it doesn't run that calculation. It predicts what the answer to that kind of question typically looks like, based on patterns from training.
Most of the time it gets simple math right. Because it has seen enough examples to predict correctly. But the moment numbers get slightly more complex, multi-step, large, or context-dependent, the prediction starts to drift.
This is what people mean when they talk about hallucinations. The model doesn't know it's wrong. It's not lying. It's confidently predicting, and sometimes that prediction is just off. With text, a hallucination is annoying. With numbers, it's a problem.
Why this matters in practice
Think about what people do with AI every day. They feed it data and ask for insights. Revenue trends, week-over-week comparisons, conversion rates.
If the LLM gets a number wrong by 3%, nobody notices in a draft. But that number ends up in a report. Someone reads it. Trust is gone.
I ran into this problem while building an AI tool for marketing agencies. The solution wasn't to prompt the model better or ask it to double-check its work. The solution was to remove numbers from its responsibility entirely.
Every metric gets calculated in code, deterministically, before the LLM ever writes a single word. The AI writes the analysis. It never touches the numbers.
That's not a workaround. That's the correct architecture for any AI system where accuracy actually matters.
The rule
Never let an LLM calculate anything that will be presented as fact. Use code for numbers, use the LLM for language.
They are genuinely good at different things. The mistake is asking one to do the other's job.
Next week: what is a prompt, really? And why "just ask it differently" is both correct and completely misunderstood.