AI verification / Case study
TruthGate
Check an agent’s claims against sources and code.
- Type
- Open-source engineering
- Release
- v1.0.0
- Stack
- Python 3.12 · AST Grounding · NLI Entailment · MCP JSON-RPC 2.0
- Automated tests
- 75
System architecture
- Atomic Claim & Citation Decomposer (truthgate/decomposer.py)
- Deterministic Citation-Span Entailment Verifier (truthgate/entailment.py)
- Workspace AST Symbol Grounder & Repair Compiler (truthgate/grounding.py, corrector.py)
01 / Problem
Why it exists
RAG and coding agents frequently cite valid source IDs while inverting negation polarity ("not permitted" -> "permitted"), shifting numeric or currency magnitudes by 10x, or referencing file paths, line ranges (#L90-L120), and function names that do not exist in the workspace.
02 / Design
How it works
Combine algebraic negation/antonym polarity checks, numeric token conservation, and live AST symbol table inspection into a sub-millisecond verification gate.
Atomic Claim & Citation Decomposer (truthgate/decomposer.py)
Splits agent markdown into atomic propositions and extracts citations, numeric/currency tokens, negation polarity, and code references.
Deterministic Citation-Span Entailment Verifier (truthgate/entailment.py)
Checks source existence (TG001), entity support (TG002), negation/antonym polarity conservation (TG003), and exact numeric conservation (TG004).
Workspace AST Symbol Grounder & Repair Compiler (truthgate/grounding.py, corrector.py)
Verifies file paths (TG006), line bounds (TG007), and AST symbols (TG008), compiling a minimal self-correction prompt when violations occur.
Deterministic Entailment & AST Symbol Grounding Over LLM-as-a-Judge
- Context
- Using a second LLM call to audit agent responses adds 2–5 seconds of latency, doubles API cost, and frequently misses subtle line-number or numeric-drift bugs.
- Decision
- Combine algebraic negation/antonym polarity checks, numeric token conservation, and live AST symbol table inspection into a sub-millisecond verification gate.
- Trade-offs
- Focuses on structural, numeric, polarity, citation, and AST grounding errors rather than open-ended stylistic critique, executing 10,000x faster (0.115 ms P50).
- Verification
- Verified across 75 pytest tests and a 40-scenario benchmark (100% precision, 100% recall, 0% FPR).
Security and operational constraints
- Workspace symbol grounder enforces strict path-traversal containment inside the workspace root.
- Deterministic verification runs 100% locally in 0.115 ms P50 without sending proprietary source documents to external LLM judges.
- Zero false-positive guarantee verified on grounded RAG and coding responses.
- Any cited source ID missing from the context corpus deterministically triggers TG001 (PHANTOM_CITATION).
- Any numeric or currency value in a cited claim that drifts from the source document triggers TG004 (NUMERIC_DRIFT).
- Any file link or symbol reference failing filesystem/AST resolution triggers TG006–TG008 before the response reaches the user.
03 / Evidence
How to verify it
Repository-reported / Hallucination Recall & Latency
100% Recall · 0% FPR · 0.115 ms40/40 exact accuracy across 30 adversarial hallucination scenarios (TG001–TG008) and 10 grounded controls
Reproduce from the command line
git clone https://github.com/codesphere-u/truthgate.git
cd truthgate && pytest -v
python -m truthgate.cli demo && python -m truthgate.cli benchmark