The demo took three weeks. The product took eleven months. That ratio — not model choice, not prompt cleverness — is the number most teams get wrong when they green-light an AI feature. An impressive demo is roughly 20% of the work. The remaining 80% is not magic and not more prompting; it is measurable engineering, and it has names: evaluation harnesses, observability, guardrails, fallbacks, and budgets.
The failure statistics are blunt. MIT's Project NANDA studied 300 public enterprise deployments and concluded that about 95% of GenAI pilots deliver little to no measurable P&L impact — the researchers call it the "GenAI Divide," and they locate the cause not in model quality but in a learning gap: tools that don't adapt to real workflows and organizations that can't tell whether the tool is working. RAND, after structured interviews with 65 experienced AI engineers and data scientists, estimates that more than 80% of AI projects fail — twice the failure rate of IT projects that don't involve AI, and its top root causes are misunderstood problems, missing data, and underinvested infrastructure, not weak models.
Both reports describe the same gap from different altitudes. Here is what crossing it looks like from inside the codebase.
Demos optimize for the happy path; production punishes it
A demo is a curated conversation. Production is your angriest customer pasting a 40-page PDF at 2 a.m. in Turkish. The distribution of real inputs is wider, weirder, and more adversarial than anything the team tried in the sprint review, and LLMs fail non-deterministically: the same feature that answered correctly a hundred times will answer differently on the hundred-and-first.
Classic software engineering has an answer to non-determinism — you don't trust behavior you haven't measured. Yet most AI features ship with less testing than a date-picker. Every prompt tweak, model version bump, or retrieval change is a deploy-and-pray. One pattern we've seen repeatedly: a support-drafting assistant works well for months, the provider upgrades the underlying model, and reply quality quietly shifts — shorter answers, new hedging, a different tone. Nobody notices for weeks because nothing errored. There was no harness that would have turned "the vibes changed" into a red number on a dashboard.
Evals are the test suite your AI feature never had
The single strongest predictor we see for whether an AI feature survives its first year is whether the team practices eval-driven development: define the expected behavior as an evaluation first, then iterate against it — the same loop OpenAI's engineering guide describes as analogous to behavior-driven development, with graders, versioned test datasets, and regression detection as you iterate on prompts.
A minimal eval harness is unglamorous and buildable in days, not months. It contains:
- A golden set: 100–500 real inputs sampled from actual traffic (or honest synthetics before launch), each with an expected output or a grading rubric. Not marketing examples — the ugly, ambiguous, malformed cases included.
- Graders: exact-match or string checks where answers are categorical, code-graded checks where structure matters (valid JSON, correct citation format), and LLM-graded rubrics for subjective qualities like tone.
- A baseline score and a diff view: every change to prompt, model, retrieval, or tools reruns the set and shows movement per case, not just an aggregate.
- A CI gate: regressions block the merge, exactly like failing unit tests.
Anthropic's evaluation guide makes a counterintuitive point that matches our experience: prioritize volume over perfection — "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." Hundreds of automatically graded cases catch regressions that twenty hand-curated ones miss.
One number is never enough, though. Stanford's HELM benchmark exists because single accuracy scores hide trade-offs: it evaluates models across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency precisely because a model can improve on one axis while degrading on another. Your product evals need the same multi-metric honesty — a summarizer that got 4% more accurate and 60% more expensive did not simply "improve."
LLM-as-judge works — if you audit the judge
For subjective qualities, the pragmatic grader is another LLM. The foundational study here, Zheng et al.'s "Judging LLM-as-a-Judge", found that strong LLM judges agree with human preferences over 80% of the time — the same level of agreement humans reach with each other. That makes LLM grading a legitimate, scalable proxy for human review.
The same paper documents why you can't deploy a judge blindly. LLM judges show position bias (favoring whichever answer appears first in a pairwise comparison), verbosity bias (preferring longer answers regardless of quality), and self-enhancement bias (favoring outputs in their own style — acute when the judge model and the product model are related). The mitigations are mechanical: swap answer positions and average, grade against explicit rubrics instead of open preference, use a judge from a different model family than the one being judged, and — non-negotiable — calibrate the judge: hand-label 50–100 cases and measure judge–human agreement before you trust the judge's numbers. An uncalibrated judge is a random-number generator with good grammar.
Put humans where the model is weakest, not everywhere
The MIT report's successful 5% share a design pattern: AI embedded in a workflow with feedback loops, not a chatbot bolted on the side. In practice the highest-leverage decision is where to place the human checkpoint. Blanket review of every output burns the efficiency you were buying; no review at all is how AI features get executive-banned after one bad incident.
The placement rule we use: humans review where errors are expensive and irreversible, models act alone where errors are cheap and detectable. Concretely, that means confidence-based routing — the system drafts everything, auto-sends the high-confidence 70%, and queues the rest for review. Every human correction flows back into the golden set, which is how the eval harness and the human loop compound: reviewers stop being a cost center and become the generator of next quarter's training and test data.
Trace everything, because the failures are silent
Traditional monitoring watches for errors and latency. LLM systems fail while returning HTTP 200. The canonical story — we've now seen versions of it at three different clients — is retrieval rot: a RAG assistant's answers slowly degrade because the company wiki moved, half the ingested URLs now 301 to a login page, and the pipeline dutifully embeds redirect stubs. No exception, no alert, just answers that get vaguer by the week until a customer complains.
The fix is tracing at the step level, not the request level: log the query, the retrieved chunks and their scores, the assembled prompt, the model version, the raw output, token counts, and cost — per request. This is standardizing fast: the OpenTelemetry GenAI semantic conventions define vendor-neutral spans for agent invocations, model calls, and tool executions, with token usage and cost as first-class attributes, so LLM traces land in the same observability stack as the rest of your system instead of a proprietary silo. Production traces also feed the loop from the previous sections: yesterday's weird real-world query is tomorrow's golden-set case.
Guardrails, fallbacks, and budgets are product features
The last discipline is refusing to let the model be a single point of failure. Input guardrails catch prompt injection and off-topic abuse before the model sees them; output guardrails validate structure, check citations actually exist, and screen for leakage of personal data — which for our EU clients is not optional hygiene but GDPR line-item compliance. Behind them sit fallback chains: if the primary model times out or the output fails validation, retry once, then degrade to a simpler model or a template, then hand off to a human. The user sees a slower answer, not a broken feature.
Budgets belong in the same category. Cost and latency are product decisions that engineering merely enforces: a per-request token budget, a p95 latency target, and an alert threshold on cost per resolved ticket. Agentic features make this urgent — a retry loop that works fine in testing can triple token spend under production load, and without per-request cost tracing you find out at invoice time. Roll all of it out the way you'd roll out any risky change: canary to 5% of traffic, compare eval metrics and cost against the control group, then widen.
None of this is exotic. It's the same discipline that made ordinary software reliable — tests, monitoring, gradual rollouts — translated for systems that are probabilistic instead of deterministic. The teams on the right side of the 95% aren't the ones with the flashiest demos. They're the ones who can answer, with a number, the question every stakeholder eventually asks: is it actually working?