AI Hallucinations What Accountants Need to Know

 In June 2023, a courtroom in the Southern District of New York was packed beyond capacity. An overflow room had been opened. Judge P. Kevin Castel was about to deliver his opinion on what seemed like an ordinary personal injury case.

It became something else. A lawyer had filed a brief citing multiple cases. The names sounded authoritative. The citations were formatted correctly. The reasoning read like real judicial language.

There was only one difficulty. None of the cases existed.

The lawyer had used AI for research. When the opposing counsel could not locate the cases, he went back to the tool and asked if they were real. It assured him they were.

He believed it. Judge Castel fined the lawyers $5,000 and wrote something every professional should read. He said they had abandoned their responsibilities.

I keep returning to that moment — a courtroom listening to judgments that were never written, attributed to judges who never wrote them. Because what happened there is not a legal problem. It is a problem of confidence without truth.

And our profession, built on the idea of a true and fair view, has now entered the same territory. A more recent example made that risk harder to ignore.

A government-commissioned report by a global consulting firm had to be revised after reviewers identified references that could not be verified, including academic citations and incorrect attributions.

The document was well-written. Structured. Persuasive. And yet, parts of it were not real. This was not a failure of technology.

It was a failure of professional judgment. We call this phenomenon “hallucination.” The term is misleading. These systems are not designed to tell the truth.

They are designed to produce the most plausible next sentence. When they do not know, they do not stop.

They continue. Fluently. Persuasively.

For casual use, this is harmless. For professional work, it is not.

When an AI tool drafts a memo on Ind AS, summarises audit evidence, or interprets a regulatory notification, the cost of plausibility without truth is borne by the person who trusted it.

What makes this particularly dangerous for our profession is that we are trained to trust well-written work.

A clean memo. A structured argument. A paragraph reference.These are signals of competence. AI produces those signals effortlessly. And when it fails, it does not announce the failure. It looks like everything else.

AI is not lying. It simply has no concept of truth. That indifference is what we now have to manage. Let me speak to two groups.

If you are a partner setting policy:

Your challenge is not whether people are using AI. They are. Your challenge is whether your processes assume they might be.

One firm I spoke with has a simple rule: Every citation, every number, every reference in AI-assisted work must be independently traced to source before it leaves the firm.

That rule costs time. It costs far less than one fabricated reference reaching a client.

The second issue is disclosure. When AI use is not transparently acknowledged, small errors become reputational events.

The third is incentive design. If AI-driven productivity is captured without reinvesting time in verification, you create the conditions for failure.

This is not a skill problem. It is a system design problem.

If you are a senior manager:

You already use these tools. You are not going to stop. What you can change is this:

Do not review AI outputs. Trace them.

If it cites a standard, verify it. If it presents a number, rebuild it. If it makes a claim, locate its source.

The second habit is better prompting.

“Explain AS 10” invites hallucination.

A better prompt defines context, scope, and constraints—and explicitly asks the model not to invent references. The output will not be perfect. But it will be more honest. The third habit is more subtle.

When AI makes your work easier, pause. Relief is often the moment professional scepticism disappears.

For those entering the profession:

The verification burden sits with you.

For now, your reviewers will assume the work is yours. That assumption protects you if you are right—and exposes you if you are not.

Build the habit of tracing every claim early. We are living through a quiet inversion.Earlier, the problem was finding answers.

Today, answers are abundant.

What is scarce is the discipline to verify them. Before you trust any AI output, ask:

1. Can I trace it to a source?

2. Does the logic hold without the AI?

3. Would I sign my name to it?

AI does not reduce responsibility. It sharpens it.

And the value of our profession will depend less on producing answers, and more on defending them.

We cannot afford unintentional trust. We have to earn, each day, the confidence our signature carries. It Is What It Is