# LLM evals-as-code · the CI gate we run on every RAG deploy

> The eval harness we actually ship: metrics, regression-blocking thresholds, gold-set curation, and the eval-as-code CI gate that catches regressions early.

An eval you run manually before each release isn't an eval · it's hope. The RAG systems that actually hold up in production have evals-as-code: a fixed gold set, metric classes in CI, regression blocking, and diff reporting per pull request.

## Five metric classes we measure

- Faithfulness · did the answer stay inside the retrieved context?
- Context precision · did retrieval pull the right chunks?
- Answer relevance · did the answer address the question?
- Bias · did neutral context produce a neutral answer?
- Injection resistance · did the system resist 80+ known attack patterns?

## Gold set discipline

Version the gold set in the repo · not a spreadsheet, not a shared Notion page. Any change to it requires a PR with reviewer sign-off. The eval harness pins to a specific gold-set version so you can compare eval runs across time fairly.

## Regression blocking

CI fails the build when any metric drops more than 5 points from the baseline. Baseline updates only merge when a human explicitly approves · not automatically on every improvement. Stops drift where 'small' regressions compound.

```python
if current_score < baseline_score - 0.05:
    raise AssertionError(
        f'Regression · {metric}: {current_score:.3f} < {baseline_score:.3f} - 0.05'
    )
```

## Weekly canary against live model

Beyond PR-blocking evals, we run the same suite weekly against the live production traffic · catches drift between what CI saw at merge time and what happens in the wild (query distribution shift, prompt-template drift, data-staleness).

> The fastest evals-as-code win: point promptfoo at 20 of your real customer prompts + what you'd accept as the right answer. Ship to CI. Before that, any LLM quality conversation is vibes.

## What to take away

- Evals that run manually are not evals. The workflow we ship: 5 metric classes in CI, a gold set under version control, regression-blocking thresholds, a diff report per PR, and a weekly 'canary' against the live model.

## Tags

AI, LLM, RAG, Eval, CI, Production

## We build this for clients

https://dfieldsolutions.com/en/services/ai-automation

## More from the lab

- https://dfieldsolutions.com/en/lab/llm-eval-harness-200-lines-typescript.md — Build an LLM Eval Harness in 200 Lines of TS
- https://dfieldsolutions.com/en/lab/shipping-ai-agents-that-work.md — Shipping AI agents that actually work in production
- https://dfieldsolutions.com/en/lab/rag-three-failure-modes-diagnostic-table.md — RAG's three failure modes (and the diagnostic table)
- https://dfieldsolutions.com/en/lab/llm-prompt-caching-production-savings.md — LLM prompt caching in production · a 60-80% cost cut

---

Source: https://dfieldsolutions.com/en/lab/llm-evals-as-code-production
DField Solutions — Torony köz 5-1.a, 2120 Dunakeszi, Hungary — +36 20 722 0002 — dezso@dfieldsolutions.com
Booking: see https://dfieldsolutions.com/en/contact