LLM Eval Gauntlet
Invariant-checked evaluation suites for tool-calling agents that run in CI and fail the build when an agent invents a number, leaks its prompt or obeys an injection.
- 1The problem
Agents that quote prices or touch money can reply with a figure the business never approved. Unit tests on the pricing engine don’t catch it, because the engine is right; the model is what drifts.
- 2Our contribution
Designed and built end to end by Kumar Sahil.
- 3What we built
Two layers on every case: per-case expectations (right tool, right arguments, expected totals, required refusals) and universal invariants on every reply (number provenance against tool results, canary-based prompt-leak detection, forged control markers, injection refusal). Text, JSON and JUnit reports with a non-zero exit code for CI.
- 4The result
Released as open source. Read the code, tests and design notes on GitHub.
Other systems we’ve built
Reply Classifier Cascade
A cost-ordered classifier for English and Dutch cold-email replies that decides which replies become CRM deals, built so a false “hot deal” needs several layers to agree.Read the case studyLLM Gateway
A provider-neutral LLM gateway: code asks for a job, the gateway routes it through a chain of models with safe failover, a deadline per call, and the cost of every call written to a ledger.Read the case studyFind the workflow worth automating first.
Tell us about one process that costs your team hours every week. We will tell you honestly whether AI or automation is the right fix, and what a first build would take.