Why your marketing tests don't teach you anything
Most companies try new things all the time. Few can say what any of it proved. The problem is rarely the ideas. It is how the tests are set up, judged and remembered.
Marketing tests stop teaching when they have no written hypothesis, change several things at once, run too briefly, measure clicks instead of customers, or end without a recorded decision. Run fewer and larger tests, decide in advance what result would count as success, and keep a single log of what was tried and what was decided.
Ask a leadership team what it has tried in marketing over the last three years and the list is usually long: a new channel, a redesigned landing page, a price promotion, a webinar series, a change of agency, a new offer for a particular segment. Ask what each of those tests proved, and the room tends to go quiet.
This is not a failure of effort or ideas. It is a failure of design and memory. Most tests in mid-sized companies are set up in a way that makes their results impossible to interpret, and even when a result is clear, it is rarely written down. Two years later someone proposes the same idea, and nobody can say how it went the first time.
Six reasons tests teach nothing
1. There was no hypothesis
"Let's try LinkedIn ads" is an activity, not a test. A test states what you expect to happen and why: "Ads aimed at operations managers in companies of 50 to 200 people will produce at least five qualified inquiries a month at under a set cost, because that is the role that initiates most of our projects." Without that statement, any result can be read as success or failure depending on who is reading it.
2. Several things changed at once
A new website launched the same month as a new campaign, a price change and a new sales hire. Inquiries rose. Which of the four changes caused it? Nobody can say, so nobody learns anything they can repeat.
3. The test was too short or too small
Many mid-sized companies receive tens of inquiries a month, not thousands. At those volumes, a two-week test of a small change produces numbers that move around by chance. Tests are often stopped when the early numbers look good or bad, which is exactly when they are least reliable.
4. The wrong thing was measured
Clicks, open rates and form fills are easy to measure quickly. Customers and revenue take longer. A test that improves click-through rate while attracting people who never buy has made things worse, but it will look like a success if clicks were the measure.
5. Nobody made a decision
The test ended. The numbers were shared in a meeting. Everyone nodded. Nothing changed, or the test simply kept running because stopping it required a conversation. A test without a recorded decision is an expense, not an experiment.
6. The result was never written down
This is the most common and the most expensive reason. People leave, agencies change, and memories blur. Without a record, the company pays for the same lesson more than once.
A test that ends without a written decision is an expense, not an experiment.
What a useful test log looks like
The fix does not require a testing platform. It requires one shared document with one line per test, filled in before the test starts and completed when it ends.
| Test | Hypothesis | Success means | Result | Decision |
|---|---|---|---|---|
| Reply within 1 hour | Faster first replies will raise inquiries that reach a real conversation | Conversation rate up from 45% to 55% over 90 days | 58% | Adopt as standard |
| Industry trade show | Meeting buyers in person will produce larger first orders | Average first order 30% above web inquiries | Not measured: leads not tagged | Rerun with tagging |
| Shorter quote form | Fewer fields will raise form completions without lowering fit | Completions up 20%, qualified share unchanged | Completions up 34%, qualified share down sharply | Revert |
| Follow-up call on quotes | A call two days after each quote will raise the win rate | Win rate up 5 points over two quarters | In progress | Review in Q2 |
How mid-sized companies should test
The testing methods used by large consumer brands, with thousands of daily visitors and dedicated analysts, do not transfer well to a company with a few hundred inquiries a year. A more useful approach for a company of that size looks like this.
- Test fewer, larger changes. A change big enough that its effect would be obvious without statistics is more useful than a small change you cannot measure.
- Prefer process tests to creative tests. Changing how fast inquiries are answered, or whether quotes are followed up, often has a far larger effect than changing an ad headline, and is easier to measure.
- Measure as close to revenue as you can. Qualified conversations, proposals and customers, not clicks.
- Change one thing at a time in each part of the system, and note anything else that changed during the period.
- Set the duration and the success measure in advance, and do not stop early because the numbers look exciting.
- Accept inconclusive results. "We could not tell" is a legitimate outcome, and much better than a confident conclusion drawn from noise.
Why this belongs in the growth system
Testing and learning is the fifth area of a growth system. It does not produce customers directly. What it produces is the company's ability to improve the other five areas over time. When it is weak, every year starts from roughly the same knowledge as the last, and decisions about channels and spending are made on opinion rather than evidence.
It also depends on the area before it. A company that cannot measure what a customer costs cannot run a test that is judged on customers. That is why, in our assessments, we look at what has been tried and what was recorded alongside how results are measured. The two usually explain each other.
Tried a lot, kept little of what you learned?
A short call will tell you whether your records can show what has worked, and what it would take to learn faster from here.
Request an introductory callCommon questions
How long should a marketing test run?
Long enough for the measure you chose to accumulate enough results to be meaningful, and at least as long as one typical sales cycle if the measure is customers. For many service businesses that means weeks or months, not days. Decide the length before the test starts, not when the numbers look good.
Do we need statistical significance for every test?
For small changes, yes, and at the volumes most mid-sized companies have, small changes rarely reach it. That is a reason to test fewer, larger changes whose effect would be visible without advanced statistics, and to be honest when a result is inconclusive.
What should we record about a test?
At minimum: what was changed, why, what result would count as success, the dates, what actually happened, and what was decided as a result. One line per test in a shared document is enough.
Who should own testing in a company without a growth team?
One named person, often the marketing lead, should keep the test log and make sure each test ends with a recorded decision. Leadership should review the log quarterly. Ownership matters more than tooling.