v0.2.0 · open source · MIT

Put AI coding agents to the test. On your own code.

RepoBoss injects real bugs into your repository, hides the tests, lets an AI agent try to fix them, and judges every change against your protected test suite. No leaderboards built on somebody else's code: measured results on yours.

5languages verified end to end
17languages for bug injection
141automated tests passing
repoboss — real session

    
PythonJavaScriptTypeScriptRustC#CC++ JavaKotlinGoRubyPHPSwiftDart ScalaLuaElixirZig
How it works

A challenge is accepted only when it is provably fair.

Every number RepoBoss reports comes from running your real test suite. Nothing is estimated.

1

Snapshot

A read-only, content-addressed copy of your repository. Your files and your git index are never touched; secrets, symlinks and caches are left out.

2

Inject one bug

Located on the syntax tree (Python ast, tree-sitter for everything else) with exact spans and hashes, never search-and-replace.

3

Prove it

  • original passes
  • with the bug, tests fail
  • restored, byte-identical, passes

Compile errors, timeouts and missing dependencies are rejected.

4

The agent tries

Tests are hidden, build configuration is read-only, and the attempt runs isolated with no network by default.

5

Judge

Only allowed changes go into a clean tree with the protected tests. Outcome, patch, tokens, model version and time are stored.

Three kinds of bugs

Real mistakes, injected with surgical precision.

config_value

A constant drifts: the kind of bug that hides in plain sight.

src/shopcalc/settings.py
-TAX_RATE = 0.22
+TAX_RATE = 0.44

path_reference

A data file reference breaks. Only strings that point to real files qualify.

src/catalog.rs
-.join("catalog.csv")
+.join("catalogs.csv")

logic_condition

An off-by-one boundary inside a function. Generics are never touched.

src/pricing.js
-  if (quantity <= 0) {
+  if (quantity < 0) {
AI agents

A real coding agent, built in.

RepoBoss ships a tool-using agent loop. The model lists, reads and searches files, edits them through checked tools, then calls finish. Point it at a local model through Ollama, or at any OpenAI-compatible API.

  • Measured, not guessed. Tokens come from the API's own usage fields; cost only appears if you set prices.
  • Versioned. The exact model digest is recorded (e.g. qwen3:14b@bdbd181c33f2).
  • Transparent. Every step is saved in a transcript you can read back.
  • Your tool too. Any command-line agent plugs in with a documented protocol.
.repoboss/agents.toml
# a local model, no account needed
[agents.qwen]
type = "llm"
provider = "ollama"
model = "qwen3:14b"

# any OpenAI-compatible API
[agents.gpt]
type = "llm"
provider = "openai"
base_url = "https://api.openai.com/v1"
model = "gpt-4.1-mini"
api_key_env = "OPENAI_API_KEY"
terminal
$ repoboss run ch-60427357 --agent qwen --attempts 5
$ repoboss compare --challenge ch-60427357
Real results

qwen3:14b, on a desktop GPU, across five languages.

31 attempts on 15 challenges · Ollama · local runner · numbers copied from the RepoBoss database.

12of 31 attempts fixed
path_reference
9 / 10
logic_condition
3 / 11
config_value
0 / 10

Broken file names are visible in the code, and the model fixed them almost every time. A changed constant is not: with the tests hidden, the right value cannot be inferred. That is the kind of insight RepoBoss surfaces about your setup. A few attempts are descriptive, never a general ranking.

Languages

Bring almost any stack.

repoboss detect recognises npm/pnpm/yarn, Python, Cargo, Go, .NET, Maven, Gradle, Bundler, Composer, CMake, Mix and Dart projects.

Pythonpytest · end to end
JavaScriptnode:test · end to end
Rustcargo · end to end
C#.NET + xUnit · end to end
CCMake + CTest · end to end
TypeScriptjest · vitest
JavaMaven · Gradle · JUnit XML
Gogo test -json
KotlinGradle · JUnit XML
RubyRSpec · Minitest
PHPPHPUnit
C++ · Swift · Dart · Scala · Lua · Elixir · Zigtree-sitter grammars

End to end = verified with the real toolchain on the bundled examples. The others are covered by unit tests of the parsers and of bug injection.

Use it your way

Native Windows app, desktop app, web dashboard and terminal.

Every interface drives the same engine, so results are identical everywhere.

RepoBoss Windows app showing runs, a colored patch and the agent log
Native WPF app for Windows 10/11 (.NET 8): overview, challenges, new challenge, run agent, runs with colored diffs, comparison and settings.
RepoBoss desktop app built with Python and Tkinter
repoboss gui: a dark desktop app in pure Python (Tkinter) for Windows, macOS and Linux.
RepoBoss web dashboard with run details
repoboss dashboard: a local, read-only web dashboard with demo labels, isolation badges and patches.
RepoBoss in the terminal
The full CLI for scripts and CI, plus repoboss interactive: guided menus with no flags to remember.

SwiftUI app: source available

A native macOS app with the same sections is in macos/. It has not been built on a Mac yet, so it is published as source only. On macOS today, use repoboss gui, the dashboard or the terminal.

Security

Agents are untrusted code. RepoBoss treats them that way.

The Docker runner installs dependencies into an image, removes every source file from it, and runs each attempt in a fresh container: no network, CPU/memory/PID limits, all capabilities dropped, nothing from your machine mounted. Links coming back are rejected. A local runner exists for trusted code, and it says so on every run.

Honest status: the Docker runner is implemented and unit-tested, but its end-to-end run has not been executed yet on a machine with Docker. Docker reduces risk; it is not a perfect boundary.

Hidden testsnever in the agent's copy
Read-only configchanging it rejects the attempt
No shell stringsevery command is an argv list
Safe importshash-checked, never auto-executed
Honest metricsunknown values stay n/a, never 0
Demos labelledpipeline demos never pose as models
Quick start

From zero to your first verified challenge.

1
pip install git+https://github.com/diegobruvhrpiugvjrigjtibjte/repoboss
2
repoboss init --runner local
3
repoboss challenge create --repo path/to/your/project
4
repoboss run ch-XXXXXXXX --agent qwen --attempts 5
5
repoboss gui
FAQ

Questions people ask.

Is this a leaderboard of models?

No. RepoBoss measures agents on your repositories, in your environment. With a handful of attempts it shows descriptive numbers and says so; it never ranks models in general.

Can the agent cheat by editing the tests?

The tests are not in its copy, configuration files are read-only, and the evaluation rebuilds a clean tree from the original snapshot. Touching a protected path rejects the attempt outright.

Why are compile errors rejected as challenges?

A compiler points straight at the bug. A good challenge must build and fail a test, so the agent has to understand the code.

Does it support Claude Code, Codex or other hosted agents?

There is no native integration yet, because none has been verified against the real tool. Any agent that runs non-interactively can be plugged in as a command agent today.

Does it work without an AI account?

Yes. Use a local model through Ollama, or try the pipeline with the built-in demo adapters, which are clearly labelled and never presented as a model's result.

Stop guessing which agent works for you.

Measure it on your code.