Coder Eval

Compare models & configs — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt

Prerequisites: Python 3.13+, uv 0.8+, and the Claude CLI ( brew install claude ). Developed on macOS; CI runs on Linux.

The optional [uipath] extra ( uv sync --extra dev --extra uipath ) adds the in-host uipath SDK for local sandbox parity; it installs from public PyPI (no credentials required). Without it the framework runs end-to-end; uipath-dependent features fail at dispatch with a clear hint.

Using Coder Eval in CI or another project? Install the published package instead of cloning:

To add it as a project dependency instead: uv add coder-eval or pip install coder-eval . In a real CI gate, pin to a specific released version so a harness upgrade can't silently move your results. (The example tasks/ live in this repo — clone it or point the CLI at your own task files.) See Tutorial 02 — Running Coder Eval in CI for the full setup.