Real case · AI agents · Reliability

My AI assistant was making up emails and I didn't notice Why AI agents make up data, and how to catch it in time

One morning in July, the daily briefing my AI assistant sends me listed two emails that never existed. Looking for the reason, I found out it had been handing me false data for days, without a single error and looking perfectly healthy. Here is what broke, why I didn't notice, and the six rules I have used since to catch it in time.

Holti Bitri Holti Bitri • • 9 min read
3
stacked failures, none visible from outside
16/28
tasks reported by the wrong model
$4.50
a month: the saving that multiplied the damage
6
rules for checks that came out of it
Illustration: a morning briefing showing status OK and five emails on the list, two of them drawn with dashed outlines and marked as not found
The status said ok. Two of the emails on the list had never existed.

An email that wasn't in my inbox

I have a personal AI assistant. Every morning, shortly after six, it sends me a briefing: unread mail, the day's appointments, what is still open, the state of the systems I keep running. I read it before coffee.

On the morning of July 27, 2026, the unread mail included two messages from senders I had never seen. One was about a service I actually use, with a perfectly plausible subject line. I looked for it in my inbox and it wasn't there. I looked everywhere, spam and trash included. Nothing.

That email had never existed. The real unread messages were three, weeks old, all from the same automated sender. The model had picked the name of that service out of its own instructions, which contained the rules for handling its notifications, and had stitched a believable sender and subject around it.

So far it sounds like the usual story about AI making things up. The interesting part came later, when I looked for the reason: working backwards, the false data had been going on for days.

Three failures, stacked

Underneath there were three independent failures. None of them was visible from the outside. Together they produced a briefing that looked normal.

status: ok The briefing arrives every morning, neatly formatted

  1. 01Mail access has expiredMail and calendar can no longer be read
  2. 02The script that reads them was half rewrittenEven with valid access, it reads nothing
  3. 03The instructions ask for two incompatible thingsWith no data, the model fills the gaps
Three independent failures, stacked. From the outside you see none.

1. Mail access had expired, and it was meant to

The assistant reads mail and calendar through a Google authorization. The app I had registered was still in "Testing" status in the console, and in that status Google expires the authorization after seven days. It wasn't a fault: it was the expected behavior.

My notes said "the token renews itself, no action needed". True for the token that lasts an hour. False for the one used to renew it, which was the one dying. It had already happened the week before: I had signed in again and everything had started working, for exactly seven more days.

2. The script that talks to Google had been half rewritten

The file the assistant uses to read mail and calendar was 4 KB instead of 43. At the bottom there was a comment: the rest of the logic is omitted for brevity, the full version is in the repository.

It wasn't a truncated file. It was a file rewritten by a model during a working session the evening before: it had carefully copied the header and replaced everything else with a pointer. A summary where the program should be.

That morning's automatic update didn't repair it, and for a sensible reason: Hermes Agent, the tool the assistant runs on, does not overwrite files the user has modified. But from the outside, a corruption looks exactly like a customization.

3. The instructions asked for two incompatible things

At the top of the briefing's instructions there was a rule, in capitals: never return an empty answer. At the bottom there was another: if a piece of data is unavailable say so, never make anything up.

As long as everything works, the two rules live together without trouble. When the sources stop answering they contradict each other, and the model picked the one that came first and was written louder.

Conflicting instructions get resolved in the worst possible way.

The moment they matter is exactly the moment they contradict each other.

Why I didn't notice

This is the part that matters more than the failures.

A crash gets noticed. A list of made-up emails doesn't.

Every check I had verified that something was present: the file is there, the job started, the assistant answered. None of them looked at the content. A service that answers is not a service that works.

The multiplier: the wrong model for the job

To save money I had set cohere/north-mini-code:free as the main model: a free model built for writing code, doing a job that has nothing to do with code, which is reading mail, calendar and tasks and summarizing them. The estimated saving was about $4.50 a month.

That same morning, after repairing the sources, I ran the exact same briefing on deepseek/deepseek-v4-flash.

north-mini-code, 06:07deepseek-v4-flash, 12:19
Unread mail2 made-up senders1 email, real
Open tasks16 of 28, the rest cut without saying so28 of 28
Anomalies from another system2 identical lines, duplicated5 distinct anomalies, with date and cause

On mail the comparison is not like for like: at 06:07 access was broken, and the first model had nothing to read. On the other two rows it is: those sources worked in both runs, and the difference is all down to the model.

The same model, when I asked it in chat to look for that email, answered four times without executing anything. At one point it asked me which mail provider I used. It had it in its own configuration.

An unsuitable model doesn't produce errors, it produces plausible results.

It turned silent failures into false data, and that makes the saving a false saving. This is no criticism of that model, which is built for a different job. The mistake was mine: putting it on this one.

What I changed

The last one is the piece that matters. It is deliberately a script and not an agent: it doesn't interpret and it doesn't summarize, so it cannot report an outcome different from the real one. If everything is fine it stays silent. It checks three things, one for each failure of that day, and it checks the first by making a real call to the service, not by looking at whether the credentials exist.

The six rules I have used since

In the months that followed I wrote about ten checks like that one. Going back over the cases where they had failed, I kept finding the same four shapes: they verified presence instead of state, they detected without repairing, they went quiet when they were the ones breaking, and what was running no longer matched what I had written down. Six rules came out of that.

1. Check the state, not the presence

The question is "can it do its job?", not "does it answer?". A check that can pass while the function is broken is not a check of that function. Mine, on the credentials, passed even with revoked credentials: it looked at whether the file was there.

2. Repair on your own, if the repair is safe and can be undone

For weeks an alert told me "retry manually". If the only action an alert suggests is one the script could have done itself, that is the repair. On two conditions: keep a copy of what was there before, and never delete.

3. Speak in two cases only

It repaired: one line, with the clue about what had broken it. It cannot repair: one line with the exact action to take, not generic advice. Silence means one thing only: I checked and everything was fine.

4. "I couldn't check" must not look like "ok"

Retry, and if it still fails, say so. And never take a destructive action on a passing error: one of my checks deleted the credentials on every failed verification, including the ones that failed because the network was down at that moment.

5. It must be testable without touching the real thing

A check whose failure mode is silence has to be put in front of a fake breakage, in every branch, before you trust its silence. Otherwise you are just hoping.

6. What runs must be what is written

One copy of the check, under version control, and from there it reaches the place where it runs. And a check that has something to say is not a failed check: if the system treats every message it sends as an error, within a week you stop reading them.

If you have an AI agent working for you, or for your team

Five questions. If you can't answer one of them, that is where I would look.

  1. Do your checks look at the content, or only at whether something arrived?
  2. When a source stops answering, does the agent say so or fill the gap? Try it: unplug a source and look at what it writes
  3. Do the instructions hold when something breaks? Look for pairs that contradict each other under failure, like "always answer" and "never make things up"
  4. Was the model chosen for the job, or for the price?
  5. Is the thing checking the agent another model? Then it can fail in the same way. A real check is a script with nothing to interpret

You cannot fully stop a model from making things up. What you can do is build things so that, when it happens, you are the one who notices. And before whoever receives the result.

If you are setting up something similar and want to compare notes, reach out.

Frequently asked questions

Why does AI make up answers?

A language model produces the most plausible text, not the true text. As long as it has the data in front of it, the two almost always coincide. When the data is missing and the instructions push it to answer anyway, it fills the gap with something believable. In my case it took the name of a service from its own instructions and built an email around it.

Can AI hallucinations be avoided?

Not completely. They can be made much rarer: by giving the model the data instead of letting it imagine it, by removing instructions that force it to always answer, by choosing a model suited to the job. What you can really do is build your checks so that when it happens, you notice.

How do you tell that an AI agent is making up data?

Not by reading the output, which is plausible by construction. You tell by checking the sources: was the agent actually able to read what it says it read? You need a separate check, one that is not itself a model, that verifies the state of the sources with a real call before the agent does its work.

Does a more expensive model make things up less?

What matters more is that it fits the job. I had put a free model built for writing code on summarizing mail and calendar, to save about $4.50 a month. On the exact same task another model reported 28 tasks out of 28 instead of 16, without making anything up.

What does human in the loop mean?

It means a person approves before an AI action takes effect. It works for actions: publishing, sending, paying. It is not enough for information: a false but plausible briefing gets past anyone's approval, because the reader has no way of knowing it is false. That is where you need checks on the sources.

What are guardrails for an AI agent?

They are the limits and checks placed around an agent: what it can do, what it has to ask, what gets verified before and after. The most useful ones do not live in the instructions given to the model, which the model can ignore, but outside it: scripts and rules that run anyway, whatever the model decides.

Holti Bitri

Holti Bitri

CTO & AI Innovator, TIG Factory. I design and lead workflows where AI agents and people build software together: written rules, a check at every step, human decisions. LinkedIn · holtibitri.com