How I'm thinking about the size of a first test, when building is cheap but launching isn't.
In my last post, I wrote that a launch doesn't have to involve the whole company, "but only if someone makes the first version small enough on purpose."
Then I left it there. Which is a bit unfair, because "make it small" is the kind of advice everyone agrees with and nobody knows how to apply. Small how? Fewer features? Fewer customers? Less code?
I've been thinking about it since, and here's where I've landed so far.
Small isn't about the code
With AI, almost everything is small in code terms now. A first version takes days. So if "small" means "not much code," then everything qualifies, and the word stops being useful.
The measure I find more useful is this: how many people does this pull in, and what does it promise?
A test that only two internal teams see is small, even if the idea is big. A tiny feature released to every customer is big, even if it's ten lines of code, because now support has to answer questions about it, someone has to document it, and customers start relying on it.
Thinking about it that way, I see a few rungs, each one pulling in more people and making more promises than the last.
The rungs
1. Look at the data. No product, no code that customers touch. Just a sample of data in front of the people who can judge it.
2. Do it by hand. The thing exists, but it's manual, rough and sent to a handful of customers directly. No interface, no announcement.
3. Release it quietly. Real feature, behind a flag, turned on for a small group. No marketing, minimal docs, but these customers will start to depend on it.
4. Launch it. Everyone gets it, and everyone in the company gets involved.
The idea is simple: start on the lowest rung that can actually answer your question, and only climb when the answer tells you to.
Rung 1: the fraud data
Here's a real one. We want to pull more fraud-related signals from an ecommerce platform's API. It sounds promising. But nobody can say for sure that the data will actually help.
The "just build it" version is tempting: add the data retrieval, update our APIs to expose the new signals, and see what happens. With AI, the code isn't even the scary part.
The scary part is the API. Once a new field is in a public API, clients build on it. It's a promise. Changing or removing it later means versioning, migration guides, and conversations with every client who started using it. That's rung 4 territory, with consequences that last for years.
So we decided to start on rung 1. When we pick this project up, the first step won't be building anything. It will be getting an offline sample of the data and handing it to customer success and data science. One question: is there anything useful in here?
If the answer is yes, we climb: build the retrieval, and change our APIs if we need to. If the answer is no, we stop, and finding out cost a data pull and some of two teams' time. No promises made to anyone outside the company.
What I like about this one is that the question ("is the data useful?") doesn't need a product to answer it. It just needs the right people looking at the right data.
Rung 2: the checklist
The second example is a feature I wrote about last time: an integration checklist. Our integration connects to a large ecommerce platform that merchants can customize in almost any way they want, and those customizations can quietly break parts of our integration. The checklist would be a page inside the merchant's admin that shows what's working and what's missing, so they stop wondering what's wrong. The engineering estimate came back at 84 hours.
This is a different kind of question. It's not "is the data useful?" It's "will merchants trust this and act on it?" That one needs real merchants, so rung 1 isn't enough.
But it doesn't need rung 3 either. One idea I keep coming back to is a script that runs the checks for 20 merchants, and an email to each one saying "here's what's working and what's missing in your setup." It's still only an idea, but it would show us whether the checks are accurate and whether merchants actually use the information, before anyone builds a dashboard, writes help articles or trains support.
The mistake I want to avoid
The easy way to choose a rung is by what's easy to build. And with AI, everything is easy to build, so teams jump straight to rung 3 or 4. "It's just a flag, let's turn it on for some customers and see."
But "easy to build" was never the right question. The right question is: what am I trying to find out, and who can tell me?
- If the question is about the data, the people who can answer are inside the company. Rung 1.
- If the question is whether customers will understand and act on something, you need a few real customers. Rung 2.
- If the question is whether it works in real day-to-day use, at some scale, you need rung 3.
- Rung 4 isn't a test. It's what you do once you already know.
What small tests can't tell you
I don't want to oversell this. Small tests have real limits.
Twenty merchants aren't all merchants. A manual email isn't the same as a polished dashboard. And a data sample that looks promising in a spreadsheet can still disappoint once it's running live.
So a good result on a low rung doesn't mean "launch it." It means "this is worth climbing one more rung." And a bad result is often the most valuable outcome, because you found out while very few people were involved.
Where I've landed
When building was expensive, "keep it small" meant writing less code.
Now that building is cheap, I think it means something else: involve as few people as you can, and make as few promises as you can, until you know it's worth it.
I'm curious where other teams start. What's the smallest test you've run that saved you from a big launch?