Canadian Legal AI Benchmark · Assessment A

Can a free AI chatbot tell a real Canadian case from a fake one?

We tested six of them with 100 case citations. 65 of them are fake. See below for the results.

Research by Tom Macintosh Zheng at Courtready.ca · August 26, 2026

ChatGPTClaudeCopilotDeepSeekGeminiKimi

↓

You are probably here because you have seen our research tracking AI-hallucinated case law in Canadian courts, where judges and adjudicators have caught more than 200 instances of someone relying on a case that does not exist. We decided to test the tools themselves.

What We Tested, and Why These Six

We tested the free tiers of ChatGPT, Claude, Copilot, DeepSeek, Gemini and Kimi. Specifically, we picked the default “engine” offered to the user once they create an account. We assume that most Canadians, especially self-represented litigants, do not pay for an upgraded tier.

We gave every tool the same 100 Canadian case citations in four equal batches. 35 refer to real decisions. 65 are fake. The fake citations fall into two groups. We drew 32 of them from our public database of citations that Canadian courts and tribunals have already identified as AI hallucinations in published decisions. We invented the other 33 ourselves.

Here were the testing conditions: we created a fresh account for each tool, turned memory off, left web search on, and did not use any reasoning or research modes. We ran each round once per tool in a new chat, sending the citations in four blocks of 25. Every conversation is linked at the bottom of this page.

We Asked for a Summary

What It Looks Like

Here are four more examples. Every word inside the quotation marks comes from the tool, and none of these cases exists.

We Asked Directly

We gave every tool the same 100 citations in a fresh chat and challenged them directly. Every tool made fewer errors than it did in Round 1, and the errors that remained ran in two directions.

We Asked, “What Happened?”

We opened another fresh chat and asked, “what happened?”, in those 100 case citations.

What It Looks Like

Here are four more examples from this round. None of these cases exists.

We Also Measured the Opposite Error: Doubting Real Law

Each tool saw the 35 real cases three times, once per round, which gave every tool 105 encounters with genuine decisions. We counted every encounter where the tool declined to treat a genuine case as real, whether it said the case was fake, called it likely fabricated, or answered that it could not verify it.

What It Looks Like

Here are four examples of this error. Every one of these cases is real, and each citation links to the decision on CanLII.

Use the Tools. Then Check.

We are not telling Canadians to avoid AI. We believe generative AI can genuinely improve access to justice. For people navigating courts and tribunals without a lawyer, these tools can explain procedure, organize facts, and make the law less intimidating.

On the specific task of telling real Canadian cases from fake ones, however, all six tools had their own flaws. The cautious tools doubted real law almost as readily as fake law. The confident tools narrated convictions, disciplinary hearings, and refugee decisions that never happened. Remember this, each failure we recorded happened with web search turned on, so the tools had access, or at least the ability, to check external resources before returning with an answer.

We Published Our Conversations with these Tools

The links below open the actual chats, exactly as we ran them, so you can verify what we entered and what each tool wrote. You can also download the full dataset for this study, with every response from every tool on every citation, as a CSV file.