Skip to content
Benchmarks

Measured, or not claimed.

Every number on this page was produced by a command in the repository. Where a figure is an estimate it says so, and where a comparison would be dishonest it is simply absent.

Recorded missions

Two fixtures, both built the same way: the tests exist, the implementation does not, and the test files sit in each task’s reference scope rather than its write scope. A worker can read what it must satisfy and cannot edit it, so the only route to a green suite is code that is actually correct.

forge-app
receipt-splitting library · committed plan
LVR-f8f72d56
tasks verified
4 / 4
proof checks
8 / 8
workers hired
7
cognitive handoffs
3
elapsed
149.6s
local / free calls
3 / 3
actual paid inference
$0.00
Handoffs
  • RATE_LIMIT · 453196 tokens (57% smaller)
  • TEST_FAILURE · 421220 tokens (48% smaller)
  • TEST_FAILURE · 464327 tokens (30% smaller)
arcade
gravity-arena prototype · committed plan
LVR-5d6aff86
tasks verified
4 / 4
proof checks
8 / 8
workers hired
6
cognitive handoffs
2
elapsed
423.0s
local / free calls
1 / 2
actual paid inference
$0.00
Handoffs
  • INVALID_OUTPUT · 467176 tokens (62% smaller)
  • PROVIDER_5XX · 1050165 tokens (84% smaller)
greeter
three small modules · plan written by a model
LVR-31eacf88
tasks verified
3 / 3
proof checks
6 / 6
workers hired
3
cognitive handoffs
0
elapsed
135.4s
local / free calls
0 / 3
actual paid inference
$0.00

RocketRide executed the cloud workers

Leverage decides which intelligence deserves a job. RocketRide runs it. A worker whose cost class is not local or host executes as a RocketRide pipeline, so the sponsor is on the critical path rather than beside it.

These are credit deltas against the real staging organisation, written by npm run verify:rocketride to demo/evidence/rocketride-run.json. A health check would prove nothing, so the figure that matters is the one at the bottom: the worker inside the pipeline returned output.

RocketRide staging execution evidence
Endpointhttps://staging.rocketride.ai
Balance before the script4188.6 / 5000
Balance after the script4186.3 / 5000
Balance moved2.30
Reported by the pipeline run2.30
Round trip19.4s
Engine tokens1.9
Worker output"READY"

One full mission ran this way. Mission LVR-bda3ba68 was started through the MCP tool leverage_run, not a script: 4/4 tasks verified across 8 proof checks in 311s, with 3 of 6 workers executing as RocketRide pipelines and $0.00 of paid inference.

Three things RocketRide's own docs get wrong
An LLM component wired to the control lane runs, consumes credits, and returns its input unchanged; the worker has to sit in the data lane. The credential field is apikey, though the server error asks for api_key. And the hackathon runs on staging, not the host the SDK defaults to. All three cost us a day and are written up in docs/ROCKETRIDE_FINDINGS.md.

Capability probe

A cold-start auction is blind: with no observations every candidate scores the prior, so the winner is effectively arbitrary. On the first real run the auction hired two models that return an empty response to any structured request, then burned three attempts discovering it.

So every reachable model is probed with small, real, executable tasks, verified by running what it wrote rather than reading it.

Capability probe results per model
ModelResultNote
ollama:samantha-v0.1:latest2/2
ollama:samantha-core:latest2/2
ollama:qwen3:4b-instruct0/2The operation was aborted due to timeout
ollama:qwen3.5:4b0/2The operation was aborted due to timeout
ollama:qwen3:4b0/2The operation was aborted due to timeout
ollama:qwen3:8b0/2The operation was aborted due to timeout
ollama:qwen2.5-coder:3b2/2
ollama:kodro-fast:latest0/2worker answer was not valid JSON: Expected property name or
ollama:kodro-coder:latest0/2no JSON object found in worker answer
ollama:kodro-tutor:latest1/2The operation was aborted due to timeout
ollama:robolearn-fast:latest0/2worker answer was not valid JSON: Expected double-quoted pro
ollama:gemma3:1b0/2worker answer was not valid JSON: Expected double-quoted pro
ollama:gemma3:4b2/2
ollama:llama3.2:3b0/2The operation was aborted due to timeout
pool:auto/best-coding2/2
pool:auto/best-reasoning2/2
pool:auto/best-fast2/2
pool:auto/coding:free0/2provider returned HTTP 429: {"error":{"message":"[429]: Felo
pool:auto/best-free2/2
pool:ghm/deepseek/deepseek-v3-03240/2provider returned HTTP 410: {"error":{"message":"[github-mod
pool:ghm/meta/llama-3.3-70b-instruct0/2provider returned HTTP 410: {"error":{"message":"[github-mod
pool:ghm/mistral-ai/codestral-25010/2provider returned HTTP 410: {"error":{"message":"[github-mod

8 of 22 models completed every probe. The failure modes were not subtle: timeouts, HTTP 500 from the local runtime, malformed output, and one genuine HTTP 429.

The probe is unstable between runs
Run twice against the same models, minutes apart, the results disagreed on three of them, and the committed probe file is the later of the two runs (the earlier one was overwritten, so only the disagreement, not its exact figures, is on record). That is what small models on free routes actually behave like, and it is the strongest argument for the architecture: a system that picks one model up front and trusts it is betting on a coin flip. Leverage assumes any worker may fail, verifies every result, and keeps the understanding when one does.

Estimated frontier-equivalent cost

The one derived number, and the one most likely to be abused. What it means, precisely: take the prompt and completion tokens observed during the run where the provider reported them, and counted from the text at 3.6 characters per token where it did not; price them at published frontier API rates; report the result.

src/core/budget.ts · FRONTIER_BASELINE
baseline   Claude Sonnet 4.5 published pricing
           $3.00 / 1M input · $15.00 / 1M output

forge-app  $0.0467
arcade     $0.0837
greeter    $0.0098

It is not a saving, not a charge, and not a claim about what a frontier agent would have spent solving the problem. That agent would plausibly have used a different number of tokens and far fewer attempts. It prices this workload at those rates and nothing more.

The control plane under load

The same scheduler that ran the missions above, driven over a 100-task graph (diamond, 4 per layer) with stub providers whose outages are scripted from a seeded generator. No provider is called and nothing is spent: this measures claiming, ordering, the auction and policy filter, the budget ledger, scoped writes, verification and handoff, and nothing about cloud throughput.

Synthetic control-plane stress test
seed 20260904 · 2026-09-05
COMPLETED
tasks completed
100 / 100
workers hired
116
outages scripted
16
handoffs
16
peak concurrent workers
4 of 8
events written
1696
duplicate claims
0
ordering violations
0
budget overshoots
0
paid candidates struck out
116
unexplained handoffs
0
elapsed
0.44s

What is deliberately absent

No baseline comparison. Running the same mission on a single frontier model needs a paid API key this build does not have. Rather than invent one, there isn’t one: no speedup multiple, no cost-reduction percentage, no “N× faster” anywhere in this repository.

One language, two fixtures. Nothing here generalises to a large polyglot repository and no such claim is made.

The benchmark plans are committed, not planner-generated. A benchmark whose task graph changes between runs measures the planner, not the workforce.

The rate limit in the canonical run is injected. It is deterministic, and labelled INJECTED in the event stream, the UI and here. What is being demonstrated is the recovery, not the coincidence.

Reproduce it

npm run fixture:reset
npm run mission -- --inject-429

npm run fixture:reset:arcade
npm run mission -- --arcade

npm run scale                        # the control-plane stress test above
npm run mission -- --repo=/abs/path/to/your/repo --goal="make test/ pass"

It will not reproduce identically. These are stochastic models on free routes and the number of attempts and handoffs varies. What is stable is the shape: every task is verified before it is accepted, and paid spend is $0.00 every time, because that part is policy rather than luck.