Measured, or not claimed.
Every number on this page was produced by a command in the repository. Where a figure is an estimate it says so, and where a comparison would be dishonest it is simply absent.
Recorded missions
Two fixtures, both built the same way: the tests exist, the implementation does not, and the test files sit in each task’s reference scope rather than its write scope. A worker can read what it must satisfy and cannot edit it, so the only route to a green suite is code that is actually correct.
- tasks verified
- 4 / 4
- proof checks
- 8 / 8
- workers hired
- 7
- cognitive handoffs
- 3
- elapsed
- 149.6s
- local / free calls
- 3 / 3
- actual paid inference
- $0.00
- RATE_LIMIT · 453 → 196 tokens (57% smaller)
- TEST_FAILURE · 421 → 220 tokens (48% smaller)
- TEST_FAILURE · 464 → 327 tokens (30% smaller)
- tasks verified
- 4 / 4
- proof checks
- 8 / 8
- workers hired
- 6
- cognitive handoffs
- 2
- elapsed
- 423.0s
- local / free calls
- 1 / 2
- actual paid inference
- $0.00
- INVALID_OUTPUT · 467 → 176 tokens (62% smaller)
- PROVIDER_5XX · 1050 → 165 tokens (84% smaller)
- tasks verified
- 3 / 3
- proof checks
- 6 / 6
- workers hired
- 3
- cognitive handoffs
- 0
- elapsed
- 135.4s
- local / free calls
- 0 / 3
- actual paid inference
- $0.00
RocketRide executed the cloud workers
Leverage decides which intelligence deserves a job. RocketRide runs it. A worker whose cost class is not local or host executes as a RocketRide pipeline, so the sponsor is on the critical path rather than beside it.
These are credit deltas against the real staging organisation, written by npm run verify:rocketride to demo/evidence/rocketride-run.json. A health check would prove nothing, so the figure that matters is the one at the bottom: the worker inside the pipeline returned output.
| Endpoint | https://staging.rocketride.ai |
|---|---|
| Balance before the script | 4188.6 / 5000 |
| Balance after the script | 4186.3 / 5000 |
| Balance moved | 2.30 |
| Reported by the pipeline run | 2.30 |
| Round trip | 19.4s |
| Engine tokens | 1.9 |
| Worker output | "READY" |
One full mission ran this way. Mission LVR-bda3ba68 was started through the MCP tool leverage_run, not a script: 4/4 tasks verified across 8 proof checks in 311s, with 3 of 6 workers executing as RocketRide pipelines and $0.00 of paid inference.
apikey, though the server error asks for api_key. And the hackathon runs on staging, not the host the SDK defaults to. All three cost us a day and are written up in docs/ROCKETRIDE_FINDINGS.md.Capability probe
A cold-start auction is blind: with no observations every candidate scores the prior, so the winner is effectively arbitrary. On the first real run the auction hired two models that return an empty response to any structured request, then burned three attempts discovering it.
So every reachable model is probed with small, real, executable tasks, verified by running what it wrote rather than reading it.
| Model | Result | Note |
|---|---|---|
| ollama:samantha-v0.1:latest | 2/2 | – |
| ollama:samantha-core:latest | 2/2 | – |
| ollama:qwen3:4b-instruct | 0/2 | The operation was aborted due to timeout |
| ollama:qwen3.5:4b | 0/2 | The operation was aborted due to timeout |
| ollama:qwen3:4b | 0/2 | The operation was aborted due to timeout |
| ollama:qwen3:8b | 0/2 | The operation was aborted due to timeout |
| ollama:qwen2.5-coder:3b | 2/2 | – |
| ollama:kodro-fast:latest | 0/2 | worker answer was not valid JSON: Expected property name or |
| ollama:kodro-coder:latest | 0/2 | no JSON object found in worker answer |
| ollama:kodro-tutor:latest | 1/2 | The operation was aborted due to timeout |
| ollama:robolearn-fast:latest | 0/2 | worker answer was not valid JSON: Expected double-quoted pro |
| ollama:gemma3:1b | 0/2 | worker answer was not valid JSON: Expected double-quoted pro |
| ollama:gemma3:4b | 2/2 | – |
| ollama:llama3.2:3b | 0/2 | The operation was aborted due to timeout |
| pool:auto/best-coding | 2/2 | – |
| pool:auto/best-reasoning | 2/2 | – |
| pool:auto/best-fast | 2/2 | – |
| pool:auto/coding:free | 0/2 | provider returned HTTP 429: {"error":{"message":"[429]: Felo |
| pool:auto/best-free | 2/2 | – |
| pool:ghm/deepseek/deepseek-v3-0324 | 0/2 | provider returned HTTP 410: {"error":{"message":"[github-mod |
| pool:ghm/meta/llama-3.3-70b-instruct | 0/2 | provider returned HTTP 410: {"error":{"message":"[github-mod |
| pool:ghm/mistral-ai/codestral-2501 | 0/2 | provider returned HTTP 410: {"error":{"message":"[github-mod |
8 of 22 models completed every probe. The failure modes were not subtle: timeouts, HTTP 500 from the local runtime, malformed output, and one genuine HTTP 429.
Estimated frontier-equivalent cost
The one derived number, and the one most likely to be abused. What it means, precisely: take the prompt and completion tokens observed during the run where the provider reported them, and counted from the text at 3.6 characters per token where it did not; price them at published frontier API rates; report the result.
baseline Claude Sonnet 4.5 published pricing
$3.00 / 1M input · $15.00 / 1M output
forge-app $0.0467
arcade $0.0837
greeter $0.0098It is not a saving, not a charge, and not a claim about what a frontier agent would have spent solving the problem. That agent would plausibly have used a different number of tokens and far fewer attempts. It prices this workload at those rates and nothing more.
The control plane under load
The same scheduler that ran the missions above, driven over a 100-task graph (diamond, 4 per layer) with stub providers whose outages are scripted from a seeded generator. No provider is called and nothing is spent: this measures claiming, ordering, the auction and policy filter, the budget ledger, scoped writes, verification and handoff, and nothing about cloud throughput.
- tasks completed
- 100 / 100
- workers hired
- 116
- outages scripted
- 16
- handoffs
- 16
- peak concurrent workers
- 4 of 8
- events written
- 1696
- duplicate claims
- 0
- ordering violations
- 0
- budget overshoots
- 0
- paid candidates struck out
- 116
- unexplained handoffs
- 0
- elapsed
- 0.44s
What is deliberately absent
No baseline comparison. Running the same mission on a single frontier model needs a paid API key this build does not have. Rather than invent one, there isn’t one: no speedup multiple, no cost-reduction percentage, no “N× faster” anywhere in this repository.
One language, two fixtures. Nothing here generalises to a large polyglot repository and no such claim is made.
The benchmark plans are committed, not planner-generated. A benchmark whose task graph changes between runs measures the planner, not the workforce.
The rate limit in the canonical run is injected. It is deterministic, and labelled INJECTED in the event stream, the UI and here. What is being demonstrated is the recovery, not the coincidence.
Reproduce it
npm run fixture:reset npm run mission -- --inject-429 npm run fixture:reset:arcade npm run mission -- --arcade npm run scale # the control-plane stress test above npm run mission -- --repo=/abs/path/to/your/repo --goal="make test/ pass"
It will not reproduce identically. These are stochastic models on free routes and the number of attempts and handoffs varies. What is stable is the shape: every task is verified before it is accepted, and paid spend is $0.00 every time, because that part is policy rather than luck.