Production leaders do not need cheaper answers at the expense of trust. They need a repeatable way to send each question to the lowest-cost model that can answer it well, and to block any release that fails agreed quality thresholds. Model routing controls the economics; The release gate protects the outcome. Together, they separate agents that become more efficient over time from agents that merely become cheaper.
Part 1 of this series established that agents stall because of the operating model around them rather than the model itself and set out a reference architecture of five layers and three planes. Two pieces do the heavy lifting: The Artificial Intelligence (AI) gateway decides what each answer costs through routing, and the release plane decides whether an answer reaches a user through evaluation. This part goes inside both.
The principle is well established: Most questions an enterprise agent receives do not need frontier-level reasoning, so they should not pay frontier-level prices. The challenge is classifying questions within milliseconds; in a way an auditor can follow without a data scientist interpreting every decision.
KPI Partners' routing work began on a hybrid multimodal Retrieval-Augmented Generation (RAG) assistant that helps site engineers troubleshoot heavy machinery from equipment manuals, repair videos and technicians' audio logs. In that first build, routing across a pool of models cut cost by up to 20% against a single baseline model over roughly 1,000 queries, and latency improved as well. We then rebuilt the routing logic as a reusable reference implementation with nine models in two pools.
The path begins with a cache, not a model. An exact or semantically similar question, matched at a cosine similarity of 0.85 or above, is answered with no model call and no retrieval. The critical detail: The cache key carries the requester's access scope. A semantic cache that ignores identity can serve a finance manager's answer to an intern who asked something similar, turning a cost optimization into a data exposure.
Questions that miss the cache are classified for subject and complexity, and this is where our two designs part ways.
The results changed how we see the trade-off. In the manufacturing build, the dynamic approach was cheaper but slower, because decomposition and the extra classification tier added model calls. On a controlled suite of 150 questions across three scenarios, run on the same model pool, V2 with live, latency-aware model selection answered 35% to 39% faster than V1. Across the reference implementation's wider running traffic, the small-model pool handled 36% of questions for about 4% of total spend, and the cache answered roughly one question in five.
The right design depends on how teams work, not on technology. A Head of Data with deep, well-documented domain expertise gets an explainable system from V1, since every routing decision traces to a rule a business owner signed off. Teams fielding varied, compound questions get more from V2, if they accept a router that is harder to explain. Either way, cheap techniques go first, so simple rules catch the easy majority.
Two practices make the difference. First, the routing table is versioned and gated like a prompt, because changing which model answers a class of questions changes the agent's behavior, however harmless the pull request looks. Second, an optional compare mode re-answers a sample of questions through the largest model and lets a judge score both answers side by side. That shadow test turns a claimed saving into a measured one, which is the evidence finance will ask for, and the bridge to the second control: If a judge can score a routing decision, it can score every release.
The most common evaluation practice in the field is, at best, an informal "vibe check": One engineer asks five or six familiar questions, likes the answers, and declares the agent ready. That approach cannot tell you how often the agent fails, on which inputs, or whether the new version beats the last one, and those are the only three questions a release decision needs answered.
An engineer commits a change to instructions, tools, retrieval or routing to Git, and it becomes a numbered, immutable agent version, because production only runs saved versions. That version answers a fixed evaluation set, and automated judges grade each answer for correctness, groundedness and tool use. Every score must clear an agreed threshold, and a named reviewer signs off before the alias moves the version into production. If something goes wrong, one step moves the alias back.
From the top down, the architecture tells us which tests we need before any traffic exists: If the billing sub-agent calls a tool that reads customer spend, a deterministic test should prove the tool returns the right rows. From the bottom up, we analyze around a hundred real or Subject Matter Expert (SME) written traces to find failure modes nobody could have predicted, such as a text-to-Structured Query Language (SQL) step that keeps confusing two product acronyms. Because agents are nondeterministic, we run each input three or four times, so one lucky answer cannot pass for reliability.
Automated judges make this affordable, but only if they agree with the people who know the domain. We design the expert loop around one constraint: An SME's time is the scarcest resource in the program. Experts review only the hard cases, meaning conversations that users rated poorly or the judge scored low, sampled across the full range of inputs so the loudest users do not define quality for everyone else.
After each round, we recalibrate the judge wherever it disagrees with the experts, and the reviewed traces join the evaluation set. Within a few cycles, experts spend less time reviewing, the judge grades every release on its own, and the gate grows stricter exactly where the agent stumbled before.
Most programs skip this step. Before go-live, we seed a documented set of known regressions into a candidate version and measure the percentage the gate blocks. That percentage is the gate's catch rate. We also record which regressions escaped and use them to strengthen the evaluation set. A gate that has never demonstrated it can stop a bad release is not yet a control; It is a formality. Catch rate is one of the four numbers we provide at the end of every proof of value, and Part 3 explains the other three.
A release gate earns the name only when its catch rate has been measured.
Already running offline evaluations or a Large Language Model (LLM) judge? Keep them. The shift is in where they sit: From a dashboard checked after release to a threshold that blocks the release.
Routing and the release gate decide what reaches production, not how an agent behaves once there, whom it may act for, or how to grow from one agent to ten. Part 3 of this series covers the observability views each stakeholder actually uses, why a cheap answer can still be an expensive one, the two levels of permissions every agentic system needs, the four deployment patterns, and how KPI Partners delivers all of it in a twelve-week first engagement.