Pin The Version

Twelve frontier models shipped in the first eighteen days of this month.

Anthropic put out Fable 5.1 and Mythos 5.1 on the first. Google followed with Gemini 3.8 Flash on the second. OpenAI's GPT-6 Astra landed with a million token context window and a price tag that makes you do arithmetic. DeepSeek shipped V4.1 Flash on the tenth. Sakana pushed Fugu Max and Fugu Ultra v2.0 on the eleventh. Z.ai's GLM 5.3 Flash showed up and immediately took roughly half the traffic when Bolt opened up open model access to its eleven million builders.

You didn't choose any of them. At least one of them is already running inside your business.

That's not a complaint about the pace. The pace is good, the models are better, prices keep falling, everybody wins. It's a complaint about the fact that you're running a production system on components that get swapped out underneath you and you have no test that would tell you when it happened.

Let's fix that. It's a two hour job and then it's a twenty minute monthly habit.

Most people think they know which model they're using. Most people are wrong, and there are three separate mechanisms at fault.

The alias. You wired up an integration six months ago and picked something like "latest" or "default" or just accepted whatever was preselected. That pointer moved. Maybe twice. Your prompt is identical, your output is not, and there's no event anywhere in your logs marking the day it changed.

The router. This one is newer and it's spreading fast. A lot of platforms now sit a router in front of several models and pick per request based on cost, speed, or a guess about difficulty. That's genuinely useful and it also means two identical calls can hit two different models, which means your "it works fine" test passed on a model your customer's request will never touch.

Your own team. Somebody upgraded the model in one node of a six node workflow because they were testing something, it looked better, and they left it. Now step three is running on a different brain than steps one, two, four, five and six. Nobody wrote it down because nobody writes anything down.

Each one is individually forgivable. Stacked, they mean you genuinely do not know what's producing your output.

Spreadsheet. Seven columns. One row per place a model runs in your business.

Where it runs. The specific node, not the tool. "Make.com scenario 14, step 3, summarizer" is a row. "Make.com" is not a row.

What it does. One clause. "Classifies inbound leads into four buckets."

Which model string. The literal string, copied out of the config. Not "Claude," not "the fast one." The actual identifier including version. If your models sit behind an aggregator like Galaxy.ai, the string is in there too and it is worth digging out, because an aggregator is exactly the kind of place a default quietly moves.

Pinned or floating. Is that string a specific version or an alias that can move? If you can't tell, mark it unknown and come back.

Who owns it. Name.

What breaks downstream if the output shape changes. This is the column that tells you where to spend your attention. A summarizer feeding a human reader can drift a lot before anybody cares. A classifier whose output routes a scenario can drift a little and break everything silently.

Last verified. The date somebody actually ran a test.

Fill this in and two things happen. You find at least one node running on something you thought you'd retired. And you find at least one high consequence node with an alias instead of a version, which is the row you fix today.

For most small operations this table has between eight and thirty rows. Mine had forty one the first time I built it, and eleven of them were things I'd forgotten existed.

Here's the part that actually protects you, and it's the part people skip because it feels like homework rather than work.

You need twelve prompts. Not a hundred. Twelve. And you need them stored somewhere you can rerun them in under ten minutes.

Build them like this.

Eight normal cases. Real inputs from your actual business, pulled from the last month. Not made up examples. Real ones, with the real mess in them. If your classifier reads inbound emails, grab eight real inbound emails, including the one that was badly formatted and the one where somebody replied above the thread.

Two edge cases. The inputs that have broken things before. Everybody has these. The customer with the extremely long name. The form submission with an emoji in the phone field. The invoice in a currency you don't normally handle. If you don't have any, ask whoever handles exceptions and they'll name four in thirty seconds.

Two adversarial cases. Inputs designed to make the model do the wrong thing. An email with instructions buried in it. A request for something outside scope. A prompt that invites the model to be more confident than the data supports.

For each of the twelve, store three things: the input, the output you got when it was working, and a short description of what "still correct" means. That third item matters more than the second. "Returns one of the four category values" is a definition of correct. "Returns exactly this paragraph" is not, because good models produce different words and still be right.

That's the whole set. It takes about ninety minutes to build and it's the difference between knowing your stack works and assuming it.

FROM THE AI NEWSROOM

The inventory sheet, the twelve prompt regression set with worked examples, the drift log, and the five failure signatures with what each one looks like in real output. Everything in this issue, built out and ready to copy. Free for readers.

Reply DRIFT and I will send it over

.....

When the model under you changes, it rarely announces itself with an error. It shows up in five specific ways and once you know them you'll spot them in a week instead of a quarter.

Format drift. Yesterday it returned clean JSON. Today it returns clean JSON wrapped in a code fence with a friendly sentence in front. Your parser dies or, worse, silently gets a null.

Length drift. Newer models are often more verbose by default, especially the reasoning oriented ones. Your summary that fit in a Slack message now runs four paragraphs. Nothing errored. Your team just stopped reading them.

Refusal drift. A prompt that worked for months starts getting declined, or hedged into uselessness, because the safety calibration moved. This one is brutal in regulated verticals and in anything touching health, finance, or legal.

Confidence drift. The model stops saying "I'm not certain" in places where it used to. Everything reads authoritative. Your team, reasonably, stops double checking. Then something wrong goes out the door with total conviction attached to it.

Tool call drift. If your workflow uses function calling, a model change can shift how often it decides to call the tool versus answer from its own head. Your search step quietly stops firing. Your outputs are still fluent. They're just now based on nothing.

That last one is the nastiest because everything looks healthy. The scenario runs. No errors. The answer reads great. It's just made up.

Two failure modes here and both are common.

The first is what we've been talking about: floating versions, no tests, surprise drift.

The second is the overcorrection. Somebody gets burned, pins everything to a specific version, and then never touches it again. Eighteen months later they're paying four times the going rate for a model three generations behind while a competitor runs the current cheap one at a tenth the cost and better quality.

The answer is to pin, and then schedule the unpin.

Pin every high consequence node to a specific version today. Then put a recurring block on the calendar, quarterly, ninety minutes, called something honest like "model upgrade window." In that block you do exactly three things: run the twelve prompt set against the current pinned model to establish the baseline, run it against the two or three candidates you're considering, and compare.

Not vibes. The twelve prompts, side by side, same day. This is the one job a multi model subscription genuinely earns its keep on, because running the same set across four models without four separate accounts turns a half day into forty minutes. You'll make the decision in twenty minutes because the answer will be obvious, and you'll have a record of why.

Between those windows, you don't chase releases. You read about them, you note the interesting ones, and you wait for your window. That's how you get the benefit of a fast moving market without living inside it.

While you're in there, look at what you're actually paying per unit of work rather than per token.

The list prices got a lot of attention this month and mostly didn't move. What moved was everything around them. Cache read pricing dropped sharply on some models, which changes the math completely for any workflow that sends the same long context repeatedly. Introductory pricing on a couple of the flash tier models has a printed end date now, which means a budget that works today doesn't work in the spring.

More importantly, reasoning models burn a different number of tokens for the same task than the previous generation did. A model with a lower per token price can easily cost more per completed job. If you're tracking spend by token, you're tracking the wrong number.

Track cost per completed unit of work. Cost per lead classified. Cost per draft produced. Cost per transcript summarized. Put that number in your inventory sheet next to the model string, and update it during the quarterly window. When it moves, you'll know whether it moved because of price or because of behavior, which are two very different conversations.

If all of that sounds like more process than your operation can carry, here's the ninety percent version in three steps.

One: find every place in your business where a model's output gets consumed by software rather than read by a human. Those are your only high consequence nodes. Everything else, a person is looking at it and will notice if it goes strange.

Two: pin those. Specific versions, written down somewhere findable.

Three: build the twelve prompt set for those nodes only, and run it on the first Monday of every month. Set a recurring reminder. Ten minutes.

That's it. You've now got better model governance than the overwhelming majority of companies ten times your size, and it cost you a morning.

I run the whole thing through a Notion database with the inventory as one table and the regression results as a linked table underneath. One row per run, one column per prompt, pass or fail. Twelve months in, the pattern of what breaks when tells you more about your stack than any dashboard.

FROM THE AI NEWSROOM

For operators who would rather build it with someone than build it alone. The full stack, installed alongside you, with the decisions made in the room instead of guessed at. Ninety seven dollars.

Join the Accelerator for $97

.....

Open whatever tool does the most important AI work in your business. Find the model setting. Look at the actual string.

If it's an alias, change it to a specific version right now. That's a thirty second fix for the single most likely cause of a weird week you can't explain.

Then grab three real inputs from last month, run them, and save the outputs in a doc with today's date on it. You just started your regression set, and future you is going to be glad the first entry exists.

Jordan

The AI Newsroom | Practical AI for people with a business to run.