You built a switchboard, not an analyst
When someone asks your AI for last quarter's sales, the LLM pattern-matches the intent, calls GET /api/sales/kpi?quarter=Q1, formats the JSON into a sentence, and sends it back. No reasoning. No analysis. Just a very expensive cable plugged into the right slot.
The Switchboard Operator — English-to-API translation
User says a phrase. AI maps it to an endpoint. That's the whole trick.
The uncomfortable question
If user says X and system does Y every time — why are we chatting with it? A “Download Sales Report” button is faster, cheaper, and doesn't charge by the token. You haven't built an intelligent assistant. You've built a CLI for people who are afraid of spreadsheets.
The Scripted Intern — multi-step playbooks
Now it follows a recipe. Still can't cook.
Instead of one API call, the assistant chains three or four together from a Markdown “playbook.” Fetch regional sales, then headcount, then calculate revenue-per-rep, then summarize.If the logic is fixed and the steps are known in advance, this should be a scheduled report. You're burning tokens to follow a rigid script that a cronjob could run for free. It's like hiring a Ph.D. to read a recipe card out loud.
- ⚠Fragile. Change one API contract and the playbook breaks silently. The LLM just summarizes garbage confidently.
- ⚠Slow. Sequential LLM hops add latency. The user waits 8 seconds for something a dashboard renders in 200ms.
- ⚠Can't improvise. Ask “why is revenue-per-rep down in APAC?” and it stalls. The playbook doesn't have a step for why.
The Senior Analyst — reasoning over data
This is where the LLM actually earns its keep.
Stage three is where the AI stops being a middleman and starts being a thinker. It reasons about data, connects dots across systems, and notices things nobody asked about. This is the only stage that justifies paying for an LLM.
Same question, three different answers
A VP of Sales asks: “How are we doing on churn?”
“Your churn rate is 5.2% this quarter.”
Called GET /api/metrics/churn. Reformatted JSON. Done.
“Churn is 5.2%, up from 4.1% last quarter. Enterprise is stable at 2.3%. SMB is at 8.7%, driving the increase.”
Followed a 3-step playbook: total churn, segment breakdown, comparison.
“Churn is 5.2%, up from 4.1%. SMB is the problem at 8.7%. But here's what caught my eye — I cross-referenced support tickets and found a 3x spike in APAC SMB complaints about delivery times since we switched logistics partners in January. The churn isn't a pricing problem. It's a fulfillment problem. Want me to pull the warehouse SLA data and draft a summary for the ops team?”
Noticed the anomaly, investigated the cause across three systems unprompted.
Stage 1 gave you a number. Stage 2 gave you a breakdown. Stage 3 told you why your ship is sinking and what to do about it. That's the difference between a middleman and a partner.
The infinite long tail
You can build a dashboard for the 10 questions people ask every day. You can write playbooks for the 50 workflows that are well-understood. You cannot pre-build for the 10,000 weird, cross-functional questions that keep leadership up at night:
"Which product line has the highest support cost per dollar of revenue, and is it getting worse?"
"Our APAC renewal rate dropped — is it correlated with the rep turnover we had in Q4?"
"If we cut the bottom 20% of SKUs by margin, what happens to warehouse utilization?"
No one built a button for these. No one wrote a playbook. These require an expert to pull data from six systems, reason about causation, and form a hypothesis. That's where an LLM earns its salary.
The bottom line
If your AI strategy is just hiding fixed business logic behind a chat prompt, you're making your users' lives harder. You're paying for a middleman to stand between your employees and the data they already know how to access.
Don't spend six figures on a telephone operator who just repeats what you said. Build a partner who knows who you should have been calling in the first place.