# Coder Eval

> Test that your Claude Code skills, MCP servers, and CLIs actually work when an agent uses them — sandboxed YAML suites, activation checks, A/B experiments, CI…

- **Website:** https://github.com/uipath/coder_eval
- **Pricing:** open source
- **Categories:** Developer Tools, Design
- **Tags:** developer-tools, design, ai-agents
- **Platforms:** macOS, Linux, CLI
- **Last verified:** 2026-09-10
- **Canonical page:** https://linkrena.com/tools/coder-eval

## About

Compare models & configs — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt

Prerequisites: Python 3.13+, uv 0.8+, and the Claude CLI ( brew install claude ). Developed on macOS; CI runs on Linux.

The optional [uipath] extra ( uv sync --extra dev --extra uipath ) adds the in-host uipath SDK for local sandbox parity; it installs from public PyPI (no credentials required). Without it the framework runs end-to-end; uipath-dependent features fail at dispatch with a clear hint.

Using Coder Eval in CI or another project? Install the published package instead of cloning:

To add it as a project dependency instead: uv add coder-eval or pip install coder-eval . In a real CI gate, pin to a specific released version so a harness upgrade can't silently move your results. (The example tasks/ live in this repo — clone it or point the CLI at your own task files.) See Tutorial 02 — Running Coder Eval in CI for the full setup.

## Related tools

- [Reachpad CLI](https://linkrena.com/tools/reachpad-cli): The reachpad CLI: workspaces where Claude, Codex and other coding agents build full-stack apps and share them by link, from your terminal.
- [Routebase](https://linkrena.com/tools/routebase): The single source of truth for your APIs: design, mock, test, and document APIs from one living spec.
- [Knowl Cloud](https://linkrena.com/tools/knowl-cloud): Knowl is an MCP memory server for Claude Code, Cursor and Codex.
- [Claude Skills Starter Kit](https://linkrena.com/tools/claude-skills-starter-kit): 20 free production-ready domain skills for Claude Code, Antigravity & AI agents — LangGraph, RAG, GEO/AEO, MCP, security - yevhens-hue/claude-skills-starter-kit
- [49 IDE](https://linkrena.com/tools/49-ide): Open-source 2D IDE for managing AI agents in native CLIs, terminal, gits, beads issues, and files across multiple projects and machines.
- [VTCode](https://linkrena.com/tools/vtcode): VT Code is an open-source Rust terminal coding agent.