Research
Running agents at a fraction of the cost,with credentials the model never sees.
We compared Colmena —the engine that runs our agents— against five agent frameworks, with the same model and the same tasks, measured on the wire.
- 10–12×
- fewer input tokens over a ten-turn session
- ~7–8×
- lower cost for the same work, with the same model
- 0%
- credential leakage, against 100% in all five
Same model, same tasks, same proxy. The full methodology is published.
Colmena, the engine we wrote
What we compare is not a wrapper over another framework: it is a graph interpreter written in Rust that runs every agent. An agent is a JSON document —nodes and edges— and the engine interprets it. The three differences we measure follow from that decision.
01
The agent is configuration
There is no build and no deploy. Publishing means saving a document, and the same document runs whether it came from the canvas or the API.
02
Credentials never enter the transcript
Sensitive values are replaced by opaque identifiers and encrypted outside the conversation; the real value is injected only into the outbound call.
03
The engine manages the context
Tool schemas are requested when needed and bulky results are summarized before returning to the model, instead of piling up every turn.
The four problems we measured
Each one is an architectural decision you can verify, not a marketing claim.
01
The cost problem
The others pay for the whole conversation again on every turn
Frameworks resend the full transcript each turn, and with it every large tool output. Our engine keeps the context light by design: it manages documents and bulky results instead of nailing them to the history.
It is not a setting to configure or code anyone writes: it is what the engine does for every agent.
02
The trust problem
A customer credential should never reach the model
When an agent receives an API key mid-conversation, all five frameworks write it into the model transcript, following their own recommended pattern. The engine encrypts it to an opaque identifier and injects the real value only into the outbound call.
There is no leak to clean up afterwards: the secret never enters the transcript, the provider history, or a cache.
03
The maintenance problem
Security as configuration, not as code to maintain
On a refund authorization task with human approval, critic retry, credential masking and an escalation policy, all six pass the four criteria. What separates them is not capability but the authoring model.
Ours is not the shortest and we claim no advantage in lines. The difference is in kind: here you change a document the same engine reinterprets.
04
The deployment problem
An agent is a file you hand to the server
This is the sharpest architectural difference. In the other frameworks an agent is a program: to add or change one, someone edits code and redeploys the service. Here an agent is a document a generic server runs.
Plainly: this is an architecture and operating-model advantage, not a benchmark figure.
Why these numbers hold
Independent measurement
Tokens and cost captured by a proxy between each framework and the provider. Read off the traffic, never self-reported.
Identical conditions
Same model, same temperature, same inputs. Every competitor built the way its own documentation recommends.
Quality-gated
Every comparison first passes the same answer-quality check. No framework saves tokens by answering worse.
Reproducible
Every figure comes from a pinned build and pinned versions, with a shared script per task. Run it again and it returns the same.
What the benchmark does not win
The engine is not faster per session. Per-call latency is on par with the fastest competitor, but the very mechanisms that save tokens cost more round trips to the model. A deployment bound by the clock rather than by cost should weigh that trade.
Where we come from
The founding team published a dozen papers applying artificial intelligence to neuroscience, agriculture and environmental monitoring. Startti is that work continued outside the lab, which is why the benchmark ships the way a paper does: with the method, the data and the code.
Every figure comes from runs measured at the provider. Context tax: mean of N=12. Credential isolation: 3 seeds × 2 attack surfaces per framework. The full methodology and the reproduction are in the paper.
The full methodology is published.
The charts, the tables and the cases where the engine loses are on the product site.