Text to SQL feels like magic until the number it returns is quietly wrong. The query runs, hands back a clean figure, and nobody notices the join was off.
Why the query runs but the answer lies
Valid SQL is not correct SQL. The model can produce something that compiles perfectly and still answers the wrong question.
It guesses join keys, picks the wrong grain, ignores a soft-delete flag, or double counts across a one-to-many join. Because it compiles and returns a number, it looks finished.
Where text to SQL fails most on analytics data
Your warehouse is exactly where it slips. The GA4 export alone has nested and repeated fields, event scope versus session scope, session_start caveats, and time zones that don't match your reporting.
Dedup is another classic. Ask for revenue and a naive query sums a line that's duplicated by a join to shipping rows, so the total looks great and is wrong by a third.
Aggregations hide it best, because a SUM over duplicated rows still returns one tidy number with nothing visibly off.
A model that doesn't know these quirks writes confident nonsense over them, and the figure still lands in a slide.
How to actually trust it
Treat AI SQL like a junior's pull request, not an oracle. Start with the grain: is this one row per order, or per order line? Half of bad numbers die right there.
Check the row count against something you expect. Compare the result to one figure you already know by heart, like last month's revenue.
Run it on a small partition first, and ask the model to explain its joins in plain English. If the explanation is vague, so is the query.
Then rebuild the query once from that explanation. If you can't reconstruct the same SQL from the model's own words, the words are hiding the bug.
Never ship a figure you haven't reconciled against something real.
Text to SQL is a genuine accelerator, but it writes SQL that runs, not SQL you can trust on sight. The moment it hands you a clean number is exactly when you should check it hardest.
Want a stronger data analyst role or a raise? Grab the FREE Product Analyst Playbook and get the exact roadmap to your next offer.
