What The Summary Left Out

Every summary you've ever read was wrong in one specific way. It left something out.

That's the job of a summary, obviously. You can't compress a forty minute call into six bullets without dropping things. The question was always whether it dropped the right things.

For most of history, a person made that call. Your assistant, your project manager, you after the meeting. They weren't perfect, but they knew what mattered to you and they knew when something small was actually big.

Now a model makes that call. Every call recap, every email thread digest, every "here's what happened in Slack while you were out" summary in your business is a model deciding what you don't need to know. And a piece of research that landed last week should make you look a lot harder at those decisions.

Researchers at Georgetown and the University of Washington looked at AI summaries of real accidents from ChatGPT and Gemini. The dominant failure wasn't making things up. It was omission. The summaries left out things that mattered. Then they ran an experiment with 328 people. Those who read a misleading recap remembered a key detail correctly 44.8 percent of the time. Those who didn't, 83.6 percent.

Read that again. The bad recap knocked people's memory backwards. That's worse than no help at all.

A separate study out of Carnegie Mellon, using synthetic hospital records, found something that points the same direction. The models were better at retrieving specific evidence than at summarizing a whole patient.

Put those together and you've got the thing worth auditing this week. Your summaries are probably accurate. Fine. The worry is what they quietly drop. Once you've read one, you trust it enough to stop digging.

Hallucination gets all the attention because it's dramatic. A model invents a quote, a number, a court case. Somebody catches it, it goes viral, everyone learns to check.

Omission doesn't go viral. Nothing in the summary is false. Every bullet checks out against the transcript. You'd pass it in any spot check. It's just that the client mentioned, around minute 31, that their budget gets approved by a new CFO now, and that sentence didn't make the cut. Three weeks later the deal stalls and nobody knows why.

Here's the shape of it in a small business:

Call recaps that capture the agreed next steps and miss the objection the prospect raised and then dropped.

Email thread summaries that tell you where the conversation landed and skip the one line where the client changed the deadline.

Support ticket digests that group twelve tickets as "login issues" and bury the one that mentions a possible data exposure.

Contract summaries that list the payment terms and the term length and leave out the auto renewal clause.

Team update rollups where "on track" is technically true for nine of ten projects and the tenth is the one your biggest client cares about.

Not a lie in the bunch. Any one of them can still cost you real money.

This takes about ninety minutes the first time. Here's the process.

Step one: list every summary you rely on. Not every summary that exists. Every one you actually make decisions from. Call recaps from your notetaker. Inbox digests. Slack catch ups. Weekly reports your team generates with AI. CRM activity summaries. Document summaries you ask for before signing something.

For each one, write down what decision it feeds. "I use the call recap to decide on the follow up." "I use the inbox digest to decide what to answer today." If you can't name a decision, that summary isn't load bearing and you can skip it.

Step two: pick the three highest stakes ones. Usually that's sales call recaps, anything touching contracts or money, and client communication digests. Start there.

Step three: pull five real examples of each. You need the summary and the source it came from. The recap and the full transcript. The digest and the actual thread. If you use Fathom for calls, both are sitting right there in your account.

Step four: read the source first, cold. This is the important part. Read the full transcript or thread before you look at the summary. Write down, on your own, the three to five things that actually mattered. Not what was discussed. What mattered. The commitment, the risk, the change, the thing you'd be annoyed to find out about later.

Step five: now read the summary and score it. For each thing on your list, mark it as captured, softened, or missing. Softened means it's technically in there but stripped of the part that made it important. "Discussed budget" is softened. "Budget now needs approval from a new CFO who wasn't on the call" is captured.

Do that for fifteen examples and you'll have your answer. In my experience most people find that two or three of their five things are captured, one is softened, and one is missing, consistently, and it's almost always the same kind of thing that goes missing.

That pattern is what you're really looking for.

When I run this audit with people, the same categories of information keep falling out of summaries. Watch for these.

Changes. A deadline that moved, a price that changed, a person who's no longer the decision maker. Summaries tend to report the current state and lose the fact that it changed, which is often the most important part.

Hedges. "We're probably going to go ahead" becomes "client is moving forward." The probably was the whole story.

Asides. Things said in passing, at the end, after "oh, one more thing." Models weight what got the most airtime. People often say the most important thing in the least airtime.

Negatives. What someone said they won't do, can't do, or don't want. "We don't want another agency that just sends reports" is gold, and it's exactly the kind of sentence that gets dropped because it isn't an action item.

Numbers mentioned once. A figure that came up one time, mid sentence, without emphasis. Headcount, a competitor's price, a target date.

Emotion. Frustration, hesitation, enthusiasm. Summaries flatten tone, and tone is often the best predictor of what happens next.

FROM THE AI NEWSROOM

The extraction prompt, the gap checker, the omission audit scorecard with categories already set up, the house rules for your team and the ten minute monthly spot check. Everything in this issue, ready to copy into your own process. Free for readers.

Reply RECAP and I'll send it over

.....

The fastest fix costs nothing. Change the instruction.

Most summary prompts, including the defaults inside most tools, ask for some version of "summarize the key points and action items." That's asking for the main thread. It's the opposite of what protects you.

Here's the prompt I use instead for anything high stakes. Paste the transcript or thread underneath.

Two things make this work. The quotes force the model to point at the source instead of paraphrasing, which kills a lot of softening. And "none found" forces it to look in every category instead of only reporting what's obvious. It also lines up with the Carnegie Mellon finding. You're asking the model to retrieve specific evidence, which it's good at, instead of summarizing the whole thing, which it's less good at.

It's a longer read than a five bullet summary. For a sales call that might decide a five figure deal, that's a trade you should take every time.

For anything where a miss would really hurt, run a second model as a checker. Different model, ideally from a different company, so it doesn't share the first one's blind spots.

That's it. You're not asking it to rewrite the summary. You're asking it to find the gap. Having two or three models available without separate accounts makes this easy. I run the checker through Galaxy.ai so the first draft and the second opinion come from different labs.

You can automate this in Make.com: transcript lands, model one extracts, model two checks for gaps, and anything the checker flags gets appended to the recap under a heading like "Also worth knowing." Most days that section will be empty. The days it isn't are the days it pays for itself.

This one's procedural, and it's the cheapest insurance you'll ever buy.

Every summary that feeds a decision should link to its source. Every recap links to the full transcript. Every digest links to the thread. Every contract summary links to the contract.

It's about how people actually behave. If the source is one click away, you'll check it when something feels off. If it's buried in a folder somewhere, you won't, and you'll make the decision from the summary every single time.

While you're at it, write one rule for your team: summaries are for remembering, sources are for deciding. Anything involving money, a commitment, or a client's feelings, somebody reads the source.

The audit isn't a one time thing. The models change, the tools change their defaults, and your business changes what matters.

Once a month, pick one summary at random from your highest stakes category. Read the source cold. Write your list. Score the summary. Ten minutes.

You're building a habit of noticing what got left out. Honestly, that's a skill worth having for summaries written by people, too. Some of the worst omissions I've ever seen came from humans who were trying to make a bad week sound fine.

One more thing about that study, because it's the part that bugs me most.

A bad summary is worse than no summary. People who read the misleading recap did worse than people who read nothing. The summary replaced their memory of the real event with a cleaner, wrong version.

That's what happens in your business, too. Once you've read the recap of the client call, the recap becomes the call. You won't remember the hesitation in their voice. You'll remember the bullet that said "client is interested in phase two."

Summaries are one of the best things AI does for a busy operator. I use them every day and I'm not giving them up. But anything that saves you that much time is also picking what you never get to see. Keep using it. Just tell it what to keep instead of letting it guess.

FROM THE AI NEWSROOM

For operators who would rather build it with someone than build it alone. The full stack, installed alongside you, with the decisions made in the room instead of guessed at. Ninety seven dollars.

Join the Accelerator for $97

.....

Pick your most important recurring summary. For most people reading this, that's sales call recaps.

Pull three recent ones along with their full transcripts. Read each transcript cold and write down the three to five things that actually mattered. Then score each recap: captured, softened, missing.

If anything important went missing, swap your summary prompt for the extraction prompt above and run it on the same three transcripts. Compare.

Thirty minutes, and you'll know exactly how much your recaps have been leaving on the table.

Jordan

The AI Newsroom | Practical AI for people with a business to run.