All work

Developer tools / Case study

AgentFlight

Replay an AI agent’s failure in a test.

A recorder and replay tool for comparing AI agent runs without repeating live model or tool calls.

🚀 Launch Live Mission Control
Type
Open-source engineering
Release
v1.4.0
Stack
Python 3.12 · MCP Schema Linter · Deterministic Replay · OTel W3C Export
Automated tests
167
incident.flight.json

Recorded tool call · payment-api · 15m

{
  "p99_latency_ms": 1840.0,
  "anomaly": true
}

✓ Matching input replays the recorded response

What if the call asks for 30 minutes?
TrajectoryDivergenceError
window: expected "15m", got "30m"
AgentFlight replay · Simulated incident from the project README

01 / Problem

Why it exists

Testing multi-step tool-calling agents against live LLMs in CI is slow, flaky, and expensive, while poorly structured MCP tool schemas ("MCP tool smells") cause agents to misinvoke tools or enter runaway A→B→A→B retry loops.

02 / Design

How it works

Combine static 10-rule MCP schema linting (TS001–TS010) with SHA-256 hash-chained trajectory cassettes for sub-millisecond deterministic replay.

  1. Black-Box Flight Recorder & Cassette Replayer (src/agentflight/recorder.py)

    Records LLM turns, tool calls, and MCP RPCs into SHA-256 hash-chained cassettes and replays them deterministically in CI with structural divergence diffs.

  2. 10-Rule MCP Tool-Schema Ergonomics Linter (src/agentflight/linter.py)

    Audits MCP and function-calling schemas for TS001–TS010 smells, computing an Agent Ergonomics Score (0–100) and token overhead estimate.

  3. Real-Time Loop Circuit Breaker & Trajectory Evaluator (guard.py / eval.py)

    Halts exact repeats, A↔B ping-pong cycles, error storms, and budget overruns; computes Tool Selection Precision/Recall/F1 against golden trajectories.

Canonical Input-Hash Cassette Replay & Static MCP Schema Linting
Context
Running live LLM evaluations on every pull request is slow, non-deterministic, and misses structural flaws in MCP tool definitions until runtime.
Decision
Combine static 10-rule MCP schema linting (TS001–TS010) with SHA-256 hash-chained trajectory cassettes for sub-millisecond deterministic replay.
Trade-offs
Requires re-recording golden cassettes when intentional prompt or tool-schema changes alter agent trajectories, but makes CI regression testing 10,000x faster and $0 cost.
Verification
Verified across 167 pytest tests (90%+ coverage), live OpenAny provider integration, and W3C OTel trace export.
Security and operational constraints
  • Tamper-evident SHA-256 hash chain across every recorded TrajectorySpan (verify_cassette detects any payload or sequence mutation).
  • Rule TS010 flags destructive MCP tools (delete_*, drop_*, exec_*, transfer_*) that lack explicit confirmation or dry-run safeguards.
  • Hard token and USD cost circuit breakers prevent denial-of-wallet attacks from runaway agent loops.
  • Any modification to a recorded span payload, latency, or order in a .flight.jsonl cassette causes verify_cassette() to fail.
  • CassetteReplayer(strict_order=True) raises TrajectoryDivergenceError with a key-by-key diff the instant an agent deviates from the golden trajectory.
  • TrajectoryCircuitBreaker deterministically trips before an agent exceeds max_tokens, max_cost_usd, or period-2/3 oscillation thresholds.

03 / Evidence

How to verify it

Repository-reported / Deterministic Replay & Ergonomics Gate

2.83 µs/span · 167 Tests

353,356 spans/sec deterministic $0 CI replay + Trajectory Diff Analyzer + W3C OTel exporter + 10-rule MCP linter

Reproduce from the command line
git clone https://github.com/codesphere-u/agentflight.git
cd agentflight && pytest -v
python -m agentflight.cli lint && python -m agentflight.cli benchmark
View source on GitHub🚀 Launch Live Mission Control
Next case study · AI security AgentShield Check permissions before an agent acts.