The "vibe check" plateau
Here's how most AI products evolve: you write a prompt, it works surprisingly well, you ship it. Users love the demo. Then edge cases pile up. You patch the prompt. It fixes one thing, breaks another. The prompt balloons to 3,000 tokens of special-case instructions. Progress stalls.
This happens because teams jump straight to changing behavior (prompt engineering, fine-tuning) without ever building a way to measurebehavior. You can't improve what you can't measure. And "I tried a few queries and it seemed fine" is not measurement — it's a vibe check.
Three things, in this order
Building a great AI product requires three activities. Most teams only do the third one.
Look at your data
Read traces. Read logs. Read what users actually send and what the model actually returns. If you're not doing this regularly, you're flying blind.
Evaluate quality
Build tests and checks that tell you — automatically and objectively — whether the AI is doing a good job. This is evals.
Change behavior
Prompt engineering, fine-tuning, better RAG, new tools. This is the fun part — but it only works if you've done #1 and #2 first.
Do all three and you get a virtuous cycle: data informs evals, evals reveal what to fix, fixes show up in data. Skip #1 or #2 and you're stuck guessing.
Three levels of evals
Not all evals are equal. Think of them as a pyramid — each level catches different things at different costs.
Unit tests — fast, cheap, run on every change
Assertions you'd write in pytest. The backbone of your eval system.
Break your AI's surface area into features and scenarios. For each scenario, write assertions that verify the output. These should run fast enough to execute on every code change.
Feature-scoped tests
Test each capability independently. "When a user asks for X, does the model return Y?" Simple input → output assertions.
Generic guardrail tests
Things that should never happen: leaking UUIDs, exposing system prompts, returning PII, hallucinating URLs.
Synthetic test cases
Use an LLM to generate diverse inputs for each scenario. 50 variations of the same request, all hitting different edge cases.
Track pass rates over time
Run in CI. Track results in a dashboard. You don't need 100% — your pass rate is a product decision.
Human + model eval — the quality gut-check
Some things can't be tested with assertions. You need human eyes and LLM judges.
Every prompt, every response, every tool call. You can't evaluate what you can't see. Make traces searchable, filterable, and easy to read.
Seriously. Read them. Read all the test case traces, read a sample of production traces. Keep reading until you stop learning new things. You will never fully stop.
Use a powerful model to critique your production model's outputs. But validate it — track agreement between your LLM judge and human reviewers. If they disagree >20% of the time, your judge needs work.
Start with good/bad. Granular scoring systems (1-5 stars) are harder to calibrate and harder to act on. You can always add nuance later.
A/B testing — the real-world verdict
Does the new version actually make users more successful?
Once your AI is stable and well-evaluated, A/B test significant changes against real users. This is the most expensive level — save it for meaningful product changes, not prompt tweaks. Measure what matters: task completion, user satisfaction, not just engagement.
The payoff: evals unlock everything else
Here's the magic nobody talks about: once you have a solid eval system, other hard problems become easy.
Debugging
A user reports a problem? Search your traces, reproduce it, write a test for it, fix it, confirm the test passes. Done.
Fine-tuning
Your eval system is already generating labeled data. Good outputs → training examples. Bad outputs → the gaps you need to fill.
Prompt iteration
Change a prompt, run your L1 suite, see exactly what improved and what regressed. No more guessing.
Mistakes we see teams make
- ⚠Skipping L1 entirely. "We'll just use GPT-4 to judge everything." No. Start with assertions. They're fast, deterministic, and catch 80% of obvious failures.
- ⚠Never looking at the data. You're reading dashboards, not traces. Dashboards tell you that something is wrong. Traces tell you what and why.
- ⚠Using generic eval frameworks. Off-the-shelf "LLM quality scores" don't know your domain. Build evals specific to your product, your users, your failure modes.
- ⚠Trusting the LLM judge blindly. An LLM judge is only as good as its alignment with human reviewers. Measure agreement. Iterate on the judge prompt. Treat it as a mini eval problem inside your eval system.
- ⚠Demanding A/B tests too early. A/B testing a broken product tells you nothing. Get L1 and L2 right first. Then A/B test the things that actually matter.
How to start (this week)
Instrument your AI to log every prompt, response, and tool call. If you can't see what it's doing, nothing else matters.
Read 50 production traces. Write down every failure mode you spot. These become your first test cases.
Write L1 assertions for your top 5 failure modes. Use an LLM to generate 20+ synthetic inputs for each. Run them in CI.
Set up a simple trace review workflow. Even a shared spreadsheet works — label traces as good/bad, note why.
Every bug report becomes a test case. Every prompt change runs through the suite. The system compounds over time.
Stop vibing, start measuring
The teams that break through the demo plateau are the ones that invest in evals early. It feels slow at first — writing tests instead of shipping features. But the compound effect is massive: every improvement is measurable, every regression is caught, and every prompt change is grounded in data instead of hope.
Your AI product deserves better than vibe checks. Build the eval system. The rest follows.