A friend sent me an article about how Anthropic automated their analytics with Claude. Link in the comments.

The headline number is striking: 95% of data queries are now handled by AI.

First reaction reading it: well, that's it then.

Every manager will just query the database themselves, get charts, draw conclusions – and analysts are obsolete.

But then it gets interesting.

Claude didn't get accurate because of some SQL magic.

SQL isn't even the hard part.

The hard part is knowing what to measure.

Say someone asks: "How many active users do we have?"

Sounds simple enough.

Then the fun begins.

What counts as active? Someone who logged in? Clicked a button? Made a purchase? Came back within 7 days? Do we filter out fraud? Test accounts? Employees?

If none of that is documented anywhere, what does AI do?

Exactly. It confidently picks an interpretation.

And the answer might be completely wrong.

That's, in my opinion, the most dangerous type of error in analytics – when everything looks smart but is actually just noise dressed up nicely.

What did Anthropic actually do?

They defined the rules of the game first:

  • which tables are authoritative
  • how metrics are calculated
  • where fresh data lives
  • who owns each metric
  • what errors come up frequently
  • how to verify an answer

There are fancy terms involved – semantic layer, evals, provenance.

But in plain language:

Semantic layer – "here's how we officially calculate our metrics." Evals – "test questions where we already know the right answer." Provenance – "where did this number come from, and can we trust it."

After that, Claude started giving accurate answers.

Without that foundation, accuracy was around 21%. With it – above 95%.

Now that's the real story.

It turns out an AI analyst is essentially a diagnostic tool for how well your analytics is actually set up.

If your metrics are documented – AI speeds up the work. If your metrics are a mess – AI just produces that mess faster and with more confidence.

Want all my posts in one place? I put 350+ articles on GA4, BigQuery, attribution, and metrics into one searchable library.

Go here to explore it for FREE.