Inference
Running a trained model to produce output — the operational, per-request phase of AI, as opposed to training. Latency, throughput and cost per token are its economics.
The phase you actually pay for
Training is a capital event; inference is the operating cost, incurred on every single request forever. An agent that takes twelve model calls to finish a task costs twelve times a chatbot answering once — which is why the economics of agent systems are decided by inference efficiency far more than by model choice alone.
Three numbers that matter
Latency to first token, throughput once flowing, and cost per token. Different workloads weight them differently: an interactive assistant lives on the first, a nightly batch job on the second, a high-volume classification step on the third. Choosing a provider without knowing which one you are optimising is how teams end up unhappy with a perfectly good model.
Route by step, not by system
The same workflow can send its bulk classification to a cheap fast tier, its judgment call to a frontier model, and its private data step to a local one. Treating inference as a per-step routing decision rather than a platform-wide commitment is usually the largest single cost lever available.
In AGNT
AGNT points inference anywhere: frontier APIs, fast-inference clouds, or fully local through Ollama.
Inference — common questions
Why is my agent so much more expensive than a chatbot?
Because it makes many calls per task and carries a large context on each one. Prompt caching, tighter context and routing routine steps to cheaper models are the standard remedies.
Is local inference free?
Free of per-token charges, yes. You are paying in hardware and electricity instead, which for high-volume routine work is frequently the better deal.
Can I change providers without rebuilding workflows?
In AGNT, yes — workflows are provider-portable, so the model behind a step is configuration rather than architecture.