Not everything needs an agent
How much of your control flow you hand to inference time, and what it costs you.
Most of my work is client engagements building agentic systems across a range of domains, several of them regulated. Over the last year one pattern has become hard to ignore: the insistence that the system be agentic, applied more or less uniformly, regardless of what the problem turns out to require. The architecture arrives before the problem is characterised, and the requirements get fitted to it afterwards.
An example, a system to triage inbound service requests, around two thousand a day, assigning each one an urgency level and routing it to the right team. The design on the table was multi-agent: several specialised agents, with a supervisor coordinating them. I asked what the urgency assignment depended on, and the answer was a spreadsheet — eleven rows, with four predicates carrying most of the logic (service tier, whether the affected system was customer-facing, region, and time of day) and the rest covering escalation paths and on-call rotation.
That spreadsheet is a decision table, and its properties are worth stating explicitly. Coverage of the input space can be verified mechanically, as can overlap between rows. Evaluation takes microseconds. It is a text artifact, so it diffs under review and versions alongside the code. Given a stored input and a ruleset version, any past decision reproduces exactly, including the one that paged an engineer at 3am. The proposed replacement offered none of these properties.
Agentic architectures address a class of problem that was previously intractable: unbounded input formats, and tasks where the next step can only be determined from the results of the last one. That capability is real, and I rely on it. The failure mode is applying it outside that class. What follows is an attempt to make the boundary precise — what each technique guarantees, how to determine which guarantees a given problem requires, and what it costs when the two are mismatched.
TL;DR
The dial measures one thing: how much of your control flow is decided at inference time rather than when the system was written. Everything else — model choice, tool count, supervisor topology — is detail.
Generation is a capability, not a synonym for agency. Interpreting or producing unstructured content is something no rule, solver, or tabular model can do at all. Needing that is a reason to bring in a generative model. It is not, by itself, a reason to hand over control flow.
That handover moves you from testing to sampling. Fixed control flow can be enumerated and covered. Model-chosen control flow can only be sampled, because the path set is a function of the input distribution and the weights, and you have neither in a checkable form.
Reliability compounds multiplicatively across steps. Ten steps at 95% each lands at 60% end to end, and agent steps aren't independent.
The techniques you skipped offer guarantees, not just lower cost. A solver either satisfies every hard constraint or reports infeasibility. There is no third outcome where it returns something plausible.
Build mixtures. Model at the edges where the input is unbounded, deterministic code where the decision has to hold up, a person on anything you can't reverse.
What the dial is actually measuring
At the left end of the dial, every branch in the system is chosen when the system is written. At the right end, the branches are chosen during the run, by a model, based on what it has seen so far. Model size, tool count, whether there's a supervisor agent on top — none of that is the axis. The axis is how much of your control flow you have handed to inference time.
Two thresholds get crossed along that line, and they're worth separating. The first is where a generative model enters at all: to the left, the system operates on structured inputs and cannot read a paragraph or write one; from the middle rightward, it can. The second is where control flow moves to inference time. The thresholds are independent, and most of the confusion in this area comes from treating them as one. A language model reading a document inside a fixed pipeline has crossed the first and not the second.
Figure 1 — The same problem can be built at any point on this line. The question is which point it belongs at.
That framing is useful because it tells you exactly what you're trading, and the trade is not really about accuracy.
Control flow fixed at authoring time is a finite graph. You can enumerate the paths, write a test per path, measure branch coverage, and know what fraction of the system's behaviour you have actually exercised. When something changes, the diff shows you which paths moved.
Control flow chosen at inference time is not a finite graph you possess. The set of paths the system can take is a joint function of the input distribution and the model weights, and you have neither in a form you can check exhaustively. So you stop testing and start sampling: you assemble an eval set, run it, and get a point estimate of behaviour with a confidence interval attached. That is a legitimate and often necessary way to work. It is a different epistemic position from branch coverage, and the difference is the whole subject of this post.
Everything else follows. Moving right buys tolerance for input you couldn't enumerate. Moving left buys enumerability, per-decision cost measured in microseconds rather than seconds, and the ability to reconstruct a specific past decision exactly. You're always paying for one with the other. The mistake isn't landing on the right-hand side. It's landing there without noticing there was a trade.
Why the pull toward agents is rational
Before the criticism, the honest part.
The money has a name on it. Budget gets approved for an AI initiative, not for we are going to formalise the triage policy. I've watched an architecture get chosen by a line item. Nobody in that chain is being dishonest; the funding is real and it arrived with a label attached.
Demos. An agent demo is theatre — you watch it think, change its mind, go and fetch something. A decision table demo is a person scrolling a spreadsheet.
Prototypes lie about how finished they are, and there's a mechanical reason. A prompt-driven prototype carries almost no fixed cost: no schema, no state model, no error taxonomy, no idempotency story. That's why it reaches a working demo in an afternoon. What it also doesn't have is a defined failure surface — so you discover that surface one incident at a time in production, and each discovery becomes a new prompt clause, a validator, or a retry. The cost curve is flat and cheap at the start and then very long. A specified system front-loads the same cost: you argue about the schema and the state transitions before anything runs. That looks slow for three weeks and then stops surprising you. Demo-driven cultures systematically select for the first curve.
And the real one: people have seen agents do genuinely new things and they want that. The instinct is correct. It has just been generalised well past its evidence.
What the other techniques actually guarantee
The usual framing is that rules and classical models are cheaper. That undersells them. The real difference is that they come with guarantees, and guarantees are a category of thing a sampled system cannot offer you at any price.
Figure 2 — Two of these columns use a language model. Only one of them is an agent.
Decision tables and state machines give you completeness and consistency checking. Formalisms like DMN let you ask, mechanically, whether your rule set covers the whole input space and whether any two rows can fire on the same input with conflicting outcomes. Those are static properties, verified without running anything against real data. A finite state machine gives you the same for sequencing: the reachable states are enumerable, so "can this ever end up approved without a signature" is a question you answer by inspection rather than by testing and hoping. Evaluation is sub-millisecond, the artifact diffs in review, and every decision is reproducible from the input plus the ruleset version.
Solvers are the ones I see skipped most, and it's the most expensive miss. If you're assigning shifts, routing vehicles, packing containers, allocating theatre slots — hard constraints, an objective to minimise — that's a constraint programming or mixed-integer problem, and mature solvers have been eating problems like it for decades. The property that matters: a solver either returns a solution satisfying every hard constraint, or reports the problem infeasible. There is no third outcome where it hands back something that looks reasonable but has someone rostered on a double shift. It will also report the optimality gap, so you know how much better the theoretical best could be. Ask a language model for a roster and you get a plausible roster, with constraint violations distributed unpredictably through it and no gap to report.
Gradient-boosted trees on tabular features remain the right answer for high-volume scoring with a lot of labelled history. Not because they're cheaper per call — though they are, by four or five orders of magnitude — but because of what surrounds them. You can backtest on held-out time periods and see how the model would have behaved through a past regime change. You can check calibration, so a score of 0.8 means the thing happens about 80% of the time. You can set the operating threshold against an explicit cost matrix for false positives against false negatives. A risk function is going to ask for all three, and "we ran an eval set and it scored well" is not a substitute for any of them.
None of this makes those techniques better. It makes them different, in ways that map cleanly onto specific questions.
What none of them can do
Every technique in the previous section operates on structured input. A decision table needs its predicates already evaluated. A solver needs its constraints stated. A gradient-boosted model needs its features extracted. None of them can take a paragraph of prose, a scanned form, or a pasted stack trace and produce something to act on, and none of them can compose an explanation a person will read.
That is the capability generative models added, and it has no substitute. Where the task requires interpreting unstructured content or producing it, no amount of rule-writing gets you there. That question is settled before any of the six below are asked.
What it establishes is narrower than it's usually taken to be. It establishes that a generative model belongs somewhere in the system. It says nothing about whether that model should also be deciding what happens next.
Six questions, and how to actually answer them
These aren't clever. They're the ones that settle the argument, and each one has evidence you can go and collect rather than opinions you can trade.
They apply to the decision, not to the reading. Where the input is unstructured, a generative model is already doing the extraction — these questions are about what happens to the values it extracted.
Figure 3 — Answer these about the problem and the architecture mostly picks itself.
Can you write the rule down? The test isn't whether you personally can. It's whether the people who own the policy already have, somewhere. Ask for it. You will frequently be handed a spreadsheet, a runbook, or a slide that has been through three rounds of review. That artifact is a specification, and turning a reviewed specification into a prompt is a downgrade in every property except effort.
Must the same input give the same answer? Not usually the same. The same. And be careful with the obvious workaround here: setting temperature to zero does not give you determinism. Floating-point addition isn't associative, so parallel reductions on a GPU can return different results depending on how work is scheduled; kernel selection varies with batch shape; mixture-of-experts routing can depend on what else is in the batch alongside your request; and providers update weights behind a stable model alias. Greedy decoding narrows the distribution a great deal. It does not make the function reproducible. If you need bit-identical replay of a decision from eighteen months ago, that requirement is asking for code.
Do you have history? Thousands of past cases with known outcomes is a substantial asset, and a prompt is a poor way to spend it. Labelled history supports backtesting, calibration, and threshold tuning. Few-shot examples in a context window support none of those.
Do you know the steps before you start? This separates workflows from agents more cleanly than anything else, and it's measurable rather than philosophical. Instrument your existing prototype and log the sequence of tool calls per run. Then look at the distribution of distinct sequences. If ninety-eight of a hundred runs took the same path, the runtime branching you paid for is being exercised twice.
Can you undo a wrong answer? Classify your outputs by reversibility, not by importance. A wrong summary is an annoyance. A wrongly closed account, a wrongly dropped ticket, a page sent to an engineer at 3am — those have a person on the other end and a paper trail. Costly and irreversible means a human decides, with the machine assembling the evidence.
How often does it run, and how fast? Do the arithmetic before the architecture. Multiply your daily volume by the number of model calls per decision by the cost per call, then annualise it. Do the same for the latency budget: a six-call agent chain at a second a call is not going in front of a synchronous request path. This arithmetic has ended more of these debates for me than any argument about capability.
Two of those questions carry most of the weight, and they sit naturally on a pair of axes:
Figure 4 — How unbounded the input is, and whether you can take a wrong answer back.
Three of those four squares are not an agent. That's not a rhetorical trick — it's roughly the distribution I see in the field.
Signs you built a sequence and called it an agent
Some of these I've caught in other people's systems. Some I've shipped.
Your prompt has turned into a policy document. If your system prompt contains ALWAYS page the on-call engineer when the contract is Tier 1, you've encoded a requirement in a medium with no coverage metric, no completeness check, and no test that fails when the clause is accidentally dropped during an edit. Prompts are a fine way to describe a job. They're a bad place to store a requirement, because nothing tells you when the requirement stops being honoured.
The tools always fire in the same order. Go and measure it, per the fourth question. Sequence entropy near zero means you have a pipeline that occasionally has a bad day.
Figure 5 — Left: what it was designed as. Right: what a hundred runs actually looked like.
Temperature zero, one tool, output pinned to a schema. Constrained decoding guarantees your output parses. It guarantees nothing about whether the values in it are right — a well-formed JSON object with a wrong priority field is still a wrong decision, and now it's a wrong decision that passes validation. If what you wanted was a typed function call, you've bought it at roughly a million times the per-call cost.
Most of your engineering goes into suppressing variance. Retries, validators, critic passes, self-consistency voting across three samples. Every one of those is a legitimate technique and I use all of them. But notice what majority voting is: a 3× cost multiplier that reduces variance without addressing bias, so it cleans up sampling noise and leaves systematic errors exactly where they were.
The clearest way to see this is to draw the finished thing next to what it replaced.
Figure 6 — Every layer on the left exists to buy back a property the column on the right started with.
Read the layers from the outside in and ask what each is for. The schema validation, the range checks, the retries, the override table, the fallback — none of them add capability. Every one is there to constrain the output back toward a fixed set of answers. The override table is the most honest item in the diagram: it's a decision table, added late, to catch the cases the model kept getting wrong.
So the finished system contains the simple technique. It just also contains every layer around it, several model calls per decision, and a per-decision cost orders of magnitude higher — to arrive at the same answers with weaker guarantees. When that apparatus becomes most of the system, step back and look at what you've built: you selected the tool whose defining property is variability, then constructed a large machine to suppress it.
Nobody can reconstruct a specific past decision. Why was this one paged in March? gets asked in every domain I've worked in, always at the worst possible moment. Answering it requires the input, the code version, the ruleset version, and the timestamp — four things you can store. If your honest answer involves a model version the provider has since retired and a sampling path you didn't record, you don't have a decision system. You have something that produces decisions, which is not the same thing.
Why long chains degrade faster than people expect
This one deserves its own section because the arithmetic is unintuitive and it explains a lot of failed pilots.
If a chain has n steps and each step is independently correct with probability p, the whole run is correct with probability pn. That exponent does brutal work. At 95% per step — which sounds like a system in good health — ten steps land at 60%. At 90% per step, ten steps land at 35%.
Figure 7 — Per-step reliability, raised to the power of the number of steps.
Two things make the real picture worse than the curve.
Steps aren't independent. An early misreading gets carried forward as context and conditions everything downstream, so failures cluster rather than distributing evenly. The independence assumption behind that curve is optimistic.
And per-step reliability is usually estimated on an eval set that under-represents the tail — which is precisely the region where you introduced an agent in the first place, because the input was unbounded. So the p you measured is likely higher than the p you're operating at.
This is also the clearest argument for the composite architecture, and it isn't an aesthetic one. Every step you move out of the chain and into deterministic code sets that factor to 1. Not 0.99. Exactly 1, for that step, permanently. Shortening the probabilistic chain is the single highest-leverage reliability move available, and it costs nothing but the willingness to specify.
Where agents genuinely win
I'd rather not be read as a sceptic, because I'm not one. There are jobs where nothing else comes close, and they share a structure.
Input variety you can't enumerate — arbitrary phrasing, arbitrary formats, a long tail nobody will ever standardise. Work where the next step depends on what the last step found, which covers most of research, triage of genuinely novel cases, investigation, and debugging. Tasks where writing the specification costs more than doing the work, which is most one-off analysis. And integration against systems with no stable contract, where a brittle mapping breaks weekly and a model absorbing the drift is cheaper than maintaining it.
And open-ended reasoning over content, where the conclusion has to be assembled from several sources and nobody can say in advance which sources matter. Retrieval alone doesn't cover this. The value is in the model choosing what to look at next based on what it just read — which is the point where generation and agency finally do coincide.
The common structure: the branching factor is too large to enumerate, or the path can't be known until you're partway down it. That's the whole test. If the procedure can be written down, write it down. If it can't, that's what agents are for — and that's a real and valuable category, not a consolation prize.
The composite is the architecture, not the compromise
Take that triage problem from the top of this post. The version that works looks like this.
A model reads whatever arrives — an email with the real problem buried in the third paragraph, a half-filled form, a pasted stack trace — and emits a structured record against a fixed schema. That record crosses a boundary into a decision function: the eleven-row table, evaluated deterministically, same answer every time, versioned and diffable. A model then drafts the acknowledgement that goes back to the requester. Anything expensive or irreversible — the 3am page — stops at a person.
Figure 8 — Put the agent where the ambiguity is. Put the code where the accountability is.
Both edges are generative work: interpretation inbound, composition outbound. Neither is agentic, because the control flow is fixed. The model is there because nothing else can read a paragraph — not because the system needs to decide what to do next.
Notice what the model does there. It reads, and it writes. It does not decide. The decision sits in the middle, in ordinary code, where it can be covered by tests, explained from stored inputs, and changed deliberately rather than by rewording a paragraph and re-running an eval.
The boundary between those regions is the part worth designing carefully. It's a schema, and it should be strict — every field typed, every enum closed, out-of-range values rejected loudly rather than coerced. The model's job is to hit that schema; the decision function's job is to assume nothing beyond it. Get that contract right and you can change models without touching decision logic, and change decision logic without re-running a single eval.
There's a regression test hiding in that. Swap the model — different vendor, smaller variant, next quarter's version — and re-run your decision corpus. Extraction quality will shift. Your decisions should not. If they do, the model was making them, and you now know that on a Tuesday afternoon rather than during an audit.
None of this is elegant the way a single agent is elegant. It's four components with contracts between them, and the contracts are a real share of the work. That's normal. Most systems that survive contact with production look like that — several separate pieces, each doing what it's good at, connected deliberately.
What this sounds like in the room
I don't refuse the request. That never goes well, and it's usually not right — on day one the client understands their problem better than I do.
What I do is ask the six questions out loud, with whoever knows the process best, and write the answers where everyone can see them. Most of the time the answers do the arguing. The room arrives at "oh, that part is just the policy table" without my having to say it, which is a much better outcome than me saying it.
Then I show the composite and point out that the agentic parts are still agentic parts. This matters more than it should. The word agent is often load-bearing for reasons unrelated to architecture — someone promised it upward, it's in a board deck. That's fine, because the honest description of what I want to build is still an AI system. It just has its AI in the places where AI is the right answer.
The other thing I've learned to say early: the mixture usually ships sooner, because the deterministic middle is the part you can actually finish and test. I've never had a client disappointed that something arrived in March rather than November.
What it costs to get this wrong
The money is the smaller loss.
A pilot that doesn't reach production makes the next three proposals harder. I've watched an organisation conclude that AI doesn't work for us on the strength of a single project mis-scoped in its first week — where the technology was never the problem, the match was. Everything anyone tries there for the next two years starts from behind.
That's the real reason I've become fussy about this. Every problem handed to the wrong technique burns credibility the right problems needed.
Four things I'd keep
The dial measures how much control flow you've moved to inference time. That's the decision. Make it deliberately, and know that it moves you from covering paths to sampling them.
Keep two questions apart. Does this need a generative model and does this need an agent have different answers. The first asks whether the content is unstructured; the second asks whether the path is knowable in advance.
Count the steps in your probabilistic chain, then shorten it. Every step you move into deterministic code changes its factor to exactly 1. Nothing else buys reliability that cheaply.
Design the boundary, not just the components. A strict schema between the model and the decision function is what lets you swap either one without holding your breath.
The industry spent a couple of years finding out what agents can do, which was necessary and mostly fun. The next stretch is less exciting and more valuable: working out where they belong.
The views, opinions, and technical perspectives expressed in this article are entirely my own and do not represent, reflect, or imply the positions, strategies, or endorsements of my employer or any organization I am affiliated with.
