"Data-driven" is the new religion. But has anyone checked if the god is actually listening?

I was listening to the REWORK podcast by 37signals — the episode “Built on Intuition.” And that same day, someone asked me to review an A/B test. Perfect timing.

I open it up: 4 landing page variants. Traffic — a couple thousand users per week. I felt uneasy.

You’d think — I’m an analyst, I should be all for this kind of thing. But looking at those 4 variants, my first thought was: guys, you don’t need more tests. You need to talk to your customers.

Here’s the math. 2,000 users, 4 variants — that’s 500 per variant. At a 1% conversion rate, you’re looking at 5 conversions per variant. Five. What statistical significance are we even talking about? You’re waiting three weeks for numbers that might — maybe — show a microscopic lift. Meanwhile, twenty customer conversations could tell you exactly why people aren’t buying.

I’ll admit it: data-driven has become a kind of religion. Everyone worships at its altar — from a three-customer startup to a corporation with millions of visitors. As if any decision not backed by a dashboard is somehow illegitimate. Or not real.

And AI is only making this worse. It feels like you can throw a few prompts at it, get instant insights, and save a ton of time. But it’s a shortcut illusion — you’re just arriving at the same meaningless conclusions faster, on the same thin data.

Take 37signals. They launched HEY — their email service — with zero market research. Simply because they hated email themselves.

Or Basecamp’s flat pricing. Any SaaS consultant would call it insane. None of these decisions were backed by data. And every single one worked.

Will every gut-driven decision work? Of course not.

But here’s the thing: data is a tool for optimization, not creation. It’ll tell you which button converts better. It won’t tell you whether that button should exist at all.

At scale, metrics are indispensable — I see that every day. But in the early stages, 5 in-depth customer interviews will give you more than a month of A/B testing.

So maybe the god is listening after all — just not through a dashboard.

Want all my best GA4-BQ queries in one place? I turned them into a Chrome extension — top SQL queries you can search and copy in seconds.

Go here to install it for FREE.

Prefer the web version? It’s here.