AI Capabilities & Limitations

The fake court case that cost real money

A customer used your contract review tool to check a data-privacy regulation. The AI returned a confident two-paragraph analysis citing "Chen v. DataCorp (2024), §14(b)." The customer's legal team built a contract clause around it. Three days later, someone checked — the case doesn't exist. The AI invented it.

This isn't a bug. It's not a glitch. It's how LLMs work. The model predicted that the next most likely tokens after "as established in" would be a case citation — so it generated one that looks real but isn't. And it did it with zero hesitation and zero uncertainty.

Hallucination isn't going away. It's a structural property of text prediction. You can reduce it dramatically, but you can't eliminate it. As the PM, your job is to design the defences before launch — not after the first incident report.

Why LLMs hallucinate: three root causes

Before reading this section: You know hallucination is a problem. But why does it happen? Take 30 seconds and write down your best guess for one root cause — then see if it matches what follows.

Every hallucination traces back to one of three causes. Each cause has a specific product-level defence:

Cause 1 → Defence 1: RAG (Retrieval-Augmented Generation)

The problem: The model's training data has a cutoff date. Anything after that date — new laws, updated pricing, recent events — it doesn't know. But it won't say "I don't know." It'll generate something plausible.

The defence: Give the model real, verified documents to reference. Before it generates an answer, retrieve the relevant pages from your own database and inject them into the prompt.

Think of it like: A closed-book exam (without RAG) vs. an open-book exam (with RAG). Same student, dramatically different accuracy on factual questions.

Cause 2 → Defence 2: Require citations

The problem: The model can write confident, authoritative text that sounds like it's backed by sources — but isn't.

The defence: Require the model to cite its sources for every factual claim. "If you can't point to a specific document in the context, you can't make the claim." This turns invisible hallucinations into visible ones — a missing citation is a red flag you can catch.

Cause 3 → Defence 3: Human review gate

The problem: When the model processes very long contexts, it's worst at using information stuck in the middle (the "lost in the middle" problem — where LLMs struggle to recall information in the middle of very long contexts (Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," 2023) — though the severity varies by model and task). Critical facts can be right there in the prompt — and the model misses them.

The defence: For high-stakes outputs (legal, financial, medical), require a human expert to review before the output reaches the customer. This catches errors that no automated system can — nuance, context, and "does this actually make sense?"

Real example: Priya's contract review tool

Priya, a PM at a legal-tech startup, shipped a contract review tool in Q1 — without RAG. Here's what happened:

Weeks 1-6: Everything looked great

94% user satisfaction. The model handled established case law fluently. The team celebrated.

Week 8: The incident

A customer asked about a data-privacy regulation passed after the model's training cutoff. The model returned a confident analysis citing a case that doesn't exist. The customer's legal team missed it, drafted a clause around the fake ruling, and the error was caught three days later — after the contract reached the counterparty.

The customer's general counsel said: "I trusted it because it cited a case number. That's worse than if it had said 'I don't know.'"

The damage

Satisfaction score dropped from 94% to 71% in two weeks. One incident.

The fix

RAG backed by a verified legal database, updated weekly. The model now must cite a real document for every factual claim — if it can't cite, it can't state. Satisfaction recovered to 91% within six weeks.

The eval that would have caught it

Citation accuracy on post-cutoff laws. That eval didn't exist at launch. It does now.

The reliability spectrum: not all tasks are created equal

Here's a mental model for quickly assessing whether an LLM will be reliable for a given task:

The PM rule: Start by shipping transformation tasks (low risk). Use those wins to build trust. Then move to pattern-matching. Save knowledge recall for when you have RAG and evals in place.