Developer tools / Case study
PatchPilot
Trace a failing test to a repair you can verify.
- Type
- Open-source engineering
- Release
- v1.1.0
- Stack
- Python 3.12 · AST Analysis · MCP JSON-RPC 2.0 · Automated Repair
- Automated tests
- 92
System architecture
- Hybrid Stack-Trace & AST Call-Graph Fault Localizer (src/patchpilot/localizer.py)
- 8-Rule AST Anti-Reward-Hacking Guard (src/patchpilot/guard.py)
- Transactional Patch Sandbox & MCP Server (sandbox.py / server.py)
01 / Problem
Why it exists
Autonomous coding agents (SWE-agent, OpenHands, Devin-style repair loops) frequently flood context windows with entire repositories when a test fails, and often "reward-hack" test suites by deleting assertions, injecting pytest.skip, mutating expected literals, or swallowing exceptions (except Exception: pass).
02 / Design
How it works
Enforce a pre-execution AST structural comparison (RH001–RH008) paired with a SHA-256 transactional rollback workspace.
Hybrid Stack-Trace & AST Call-Graph Fault Localizer (src/patchpilot/localizer.py)
Parses tracebacks, indexes workspace AST symbols and caller/callee edges, and extracts minimal line-numbered suspect slices.
8-Rule AST Anti-Reward-Hacking Guard (src/patchpilot/guard.py)
Inspects before/after ASTs for test deletion/skipping (RH001), assertion weakening (RH002), exception swallowing (RH003), dangerous primitives (RH005), and protected path edits (RH008).
Transactional Patch Sandbox & MCP Server (sandbox.py / server.py)
Applies patches transactionally, runs verification suites, rolls back byte-for-byte on failure, and exposes 4 MCP tools.
AST Structural Diff Verification Against Coding-Agent Reward Hacking
- Context
- Evaluating coding agents solely by whether pytest exits with code 0 allows agents to pass by deleting failing assertions or catching Exception: pass.
- Decision
- Enforce a pre-execution AST structural comparison (RH001–RH008) paired with a SHA-256 transactional rollback workspace.
- Trade-offs
- Requires language-aware AST parsing prior to running tests, but adds only 0.098 ms P50 latency while eliminating reward-hacked green builds.
- Verification
- Verified across 92 pytest tests, unified git diff parsing, and Nexus Aegis G5 bridge.
Security and operational constraints
- Rule RH005 blocks injection of eval(), exec(), __import__(), os.system(), and subprocess(..., shell=True) inside agent-generated patches.
- Rule RH008 prevents autonomous modification of protected CI workflows (.github/workflows/*), build configs (pyproject.toml), and test harnesses (conftest.py).
- Every repair attempt produces a SHA-256 signed RepairReceipt binding pre-state hashes, post-state hashes, and guard verdicts.
- Any patch that reduces assertion counts, introduces assert True, or adds pytest.skip/xfail in a test file is deterministically rejected before test execution.
- Failed verification or guard rejection restores all modified, created, or deleted files to their exact pre-transaction SHA-256 digests.
- Protected configuration and CI workflow paths cannot be modified unless explicitly allowlisted.
03 / Evidence
How to verify it
Repository-reported / Fault Localization & Anti-Cheat Gate
100% Top-1 · -96.6% Tokens · 92 Tests100% Top-1 fault localization across 25 scenarios and 100% recall (0% FPR) blocking 20 reward-hacking patches (RH001–RH008)
Reproduce from the command line
git clone https://github.com/codesphere-u/patchpilot.git
cd patchpilot && pytest -v
python -m patchpilot.cli demo && python -m patchpilot.cli benchmark