Public engineering case study
A reproducible workbench for measuring where document-grounded AI systems break.
FAULTLINE replaces subjective evaluation with seeded environments, independent oracles, labelled fault injection, reconstructable traces, explicit replay boundaries, and statistical tests that keep reliability claims honest.
302
passing tests
18
completed increments
6
labelled fault classes
4,000
traced Q1 runs
System architecture
Each layer creates the evidence required by the next. Faults are known before detection, traces survive failure, and evaluation is scored against truth outside the detector.
Byte-reproducible corpora, questions, and ground truth.
Typed tool contracts and structural termination guarantees.
Six deterministic fault families with out-of-band labels.
Crash-complete spans, redaction, reconstruction, and replay boundaries.
Independent oracles, per-class scoring, intervals, and paired tests.
Headline experiment · Day 10
Injected semantic corruption
expected value = 40
received value = 80
schema_valid = true
invariant_ok = true
oracle_verdict = "wrong"
classifier = "ok"
Shape validation and a narrow invariant both passed. Only the independent oracle knew that the value was wrong.
40%
In the committed Day 10 experiment, the detector caught 3 of 5 wrong-data positives: recall 0.60, precision 1.00, with 2 false negatives. Explicit provider failures were detected with recall 1.00.
2 / 5
semantic faults escaped
0
false positives
Detector evaluation · Day 15
The full labelled set contains 44 samples. Overall recall was 61.5%, but the useful result is per-class: deterministic failures and semantic failures do not have the same detection surface.
F1Schema corruption
66.7% recall
F2Latency budget
33.3% recall
F3Wrong data
50% recall
F4Provider error
100% recall
F5Context corruption
50% recall
F6Loop exhaustion
100% recall
Two more findings
97.2% → 28.8%
Across 500 trials per depth, measured success fell to 28.8% at eight required hops while naive constant-rate compounding predicted 79.7%. The naive curve left the measured 95% interval at hop three.
Open Q1 results0 / 9 → 6 / 9
Repair-retry recovered six paired scenarios that failed without recovery. McNemar's exact p-value was 0.03125. Persistent faults still exhausted safely inside attempts, cost, and latency ceilings.
Open recovery evidenceReproduce it
A fresh clone can run the full gate or rebuild the specific evidence shown on this page. The repository pins dependencies and keeps every result under a versioned evidence directory.
Review every evidence target$ git clone https://github.com/samirsawarkar/faultline-ai-reliability.git
$ cd faultline-ai-reliability
$ make venv && make test
$ make day10-contracts
$ make day15-q2
$ make day18-recovery
Honest boundary
The subjects are deterministic stand-ins. They validate the harness, measurement design, and failure accounting—not the reliability of a production model. Recorded provider calls can replay exactly; live provider re-execution cannot be claimed byte-reproducible.
Read the full replay boundaryWhy this matters
Reliability engineering starts by making the ruler trustworthy. FAULTLINE makes the ruler reproducible, exposes where deterministic detection stops working, and preserves the evidence needed to challenge every conclusion.
Evaluate the work
Samir Sawarkar