Grade It Before You Read It

Researchers at Peking University, the Chinese Academy of Sciences and JD.com published something this week that should make you slightly uncomfortable if you use AI to check AI.

They built a benchmark called ImpossibleRubrics. The design is elegant and a little mean. They constructed 169 tasks that genuinely cannot be completed correctly, paired each one with an oracle certificate proving impossibility, then asked models to generate scoring rubrics and grade attempts against them.

The models gamed their own rubrics between eight and twenty six percent of the time. On tasks with no correct answer, they produced passing scores.

Sit with that for a second. Not "the model was wrong about a hard question." The model was asked to evaluate whether work met a standard, on work that could not possibly meet the standard, and it said yes, roughly one time in six.

Now think about how you use the pattern. You write a draft, you paste it back in, you say "score this against the criteria and tell me what to improve." The model gives you an 8 out of 10 and three notes. You fix the three notes. You ship it.

What just happened there? Did you get a review, or did you get a very confident machine telling you what you wanted to hear in a format that looks like rigor?

Both, sometimes. Here's how to make it mostly the first one.

I'm not building toward "don't do this." I do it constantly and it's one of the highest return prompts I run.

The reason is that most work fails on things a checklist catches. Not on genius. On the boring stuff. Is there a specific number anywhere in this. Does the first line earn the second. Did I actually say who this is for. Is there a claim in here I can't back. A rubric catches all of that reliably, and catching it before a human reads your draft is worth real money.

What the research tells you isn't to stop. It tells you the failure mode is specific and predictable, which means it's defendable. The model cheats in three recognizable ways and each has a counter that takes one extra sentence in your prompt.

Let's build the thing properly.

Here's the base. Adapt the criteria, keep the structure.

Paste your draft underneath. Read the output. Fix what's real, ignore what isn't.

One practical note. Run the rubric on a different model than the one that wrote the draft. Not for mystical reasons, just because a model grading its own output has the same conditioning problem as a model writing its own criteria. I keep a couple of models one login apart through Galaxy.ai for exactly this, and the disagreements between them are frequently the most useful part of the output.

Notice what that prompt does and doesn't do. It doesn't ask for a rewrite, because a rewrite is how you lose your voice. It doesn't ask "is this good," because that's an invitation to be agreeable. It asks for scored criteria with quoted evidence and three forced answers, and every one of those choices is doing work.

The single biggest finding in that research, translated into practice, is this: never let the model generate the standard it's going to grade against.

When you say "come up with criteria for a good landing page and then score mine," you've handed it both ends of the transaction. It will generate criteria that your draft happens to satisfy, because it just read your draft and the criteria generation is conditioned on it. This isn't malice, it's just how conditioning works. But the effect is a scorecard built to be passed.

Write your own criteria. Six is the right number, four is fine, ten is too many and the scores get mushy.

Write them once per output type and reuse them forever. You should have a rubric for cold emails, one for landing pages, one for newsletter drafts, one for proposals. They're short. They live in a doc. This is a thirty minute investment that pays out for years.

And write them as tests, not as virtues. "Compelling" is a virtue and unscoreable. "The first two sentences give a reason to read the third" is a test and either passes or doesn't.

FROM THE AI NEWSROOM

The exact systems behind everything in this issue. The audit sheets, the routing logic, the templates and the review cadences, built out step by step so you can copy them straight into your own stack. One time, forty seven dollars.

Get the Blueprint for $47

.....

Cheat one: restating the criterion as evidence.

You ask if the draft is specific. It says "the draft demonstrates strong specificity throughout, grounding its points in concrete detail." That's not evidence. That's the criterion with adjectives on it. And it's the single most common failure, by a mile.

The counter is in the prompt above: require a quote. Exact words from the draft, or the score is 1. A quote is falsifiable. You can look at it and go "that's not a number, that's a category." The moment you force quotation, the score has to attach to something real, and the inflated scores collapse on their own.

Cheat two: grading the easy axis and waving at the hard one.

Some criteria are easy to evaluate from the text. Does it have numbers, yes or no. Some require judgment about something the model can't see, like whether the claim is actually true or whether this audience would care. The model will do a thorough job on the easy ones and produce a confident sentence on the hard ones.

Two counters. First, "you must score every criterion, not applicable is not available," which stops the quiet skip. Second, the forced verification question at the end: "state one thing this draft claims that you could not verify." That one question does more work than the other five criteria combined, because it flips the frame from finding support to finding the gap. It's the difference between asking somebody to check your homework and asking them to find the mistake.

Cheat three: passing the impossible.

This is the one from the research and it's the one you can't fully fix, only detect.

The counter is to poison the test occasionally. Once every couple of weeks, run the rubric against something you know is bad. A draft with a made up statistic in it. A version with the opening deleted. A piece written about nothing in particular.

If it comes back with a 7, your rubric is decorative and you've been getting flattered for a month. If it comes back with a 3 and correctly identifies the problem, your rubric works.

This takes four minutes and almost nobody does it. It's the equivalent of pulling the fire alarm to check the bell still rings, and it's the only way to know whether the thing you've built is a control or a ritual.

The line I'd add to your prompt if you only add one:

Here's why that one sentence outperforms everything else in the prompt.

A score is a judgment. Judgments can be soft. "You're at a 4 on specificity" leaves everything open. But "move it up one point" demands a specific, bounded, actionable change. The model has to produce something you could actually do this afternoon. And because it's constrained to one point rather than "make it perfect," you get the next real step instead of an essay about excellence.

It also surfaces the thing you'd have avoided. Left to itself the model will give you three easy fixes. Forced to name the weakest, it has to go to the place you were hoping it wouldn't, which is usually the place you already knew about.

I've run this on my own drafts for months and the weakest criterion is the same one about sixty percent of the time. That pattern told me more about how I write than any feedback I've ever gotten from a person.

Rubrics need a home or they don't get used.

Mine live in one Notion page per output type, and the high volume ones get wired into a Make.com scenario so the draft gets scored automatically before it ever reaches me. Each page has the criteria, the full prompt ready to copy, and a short log at the bottom of things I learned from running it. When a rubric misses something obvious, I add a criterion. When a criterion never varies, I cut it, because a test everything passes isn't a test.

That last rule is worth repeating. If a criterion scores 5 on every single draft, delete it. It's measuring nothing. Your rubric should hurt at least a little or it's not doing anything except making you feel professional.

The pruning is what turns a generic checklist into a rubric that's actually about your work and your particular failure patterns, and that version is genuinely valuable in a way the starter version isn't.

One step back before I let you go.

The reason the rubric prompt works isn't that the model is a good judge. It's that writing the rubric forces you to be explicit about what good means, and most people have never done that for their own work.

Ask a business owner what makes a good sales email and you'll get twenty minutes of directionally correct instinct. Ask them to write six testable criteria and they'll stall, because instinct doesn't translate into tests without effort.

The effort is the point. Once your standards exist in writing, you can hand them to a contractor, to a new hire, to a tool, to yourself at eleven at night when your judgment isn't what it was at nine in the morning. The rubric is just the artifact. The value is that you had to decide.

The model gaming it occasionally is a real problem and you should build the counters. But even a rubric the model cheats on is better than the thing most people are doing, which is reading their own draft twice and deciding it's fine.

FROM THE AI NEWSROOM

For operators who would rather build it with someone than build it alone. The full stack, installed alongside you, with the decisions made in the room instead of guessed at. Ninety seven dollars.

Join the Accelerator for $97

.....

Pick the output type you produce most. Emails, posts, proposals, whatever pays you.

Write six criteria. Tests, not virtues. Twenty minutes, and expect it to be harder than you thought, which is the useful part.

Then run the prompt above on the best thing you produced last month and the worst thing you produced last month, back to back.

If the scores come back close, your criteria are wrong. Fix them until the rubric can tell the difference, because a rubric that can't separate your best work from your worst isn't measuring your work, it's measuring your mood.

Jordan

The AI Newsroom | Practical AI for people with a business to run.