Operations

Deployment

Evo.ai ships as a Docker stack (the app plus Qdrant, with Ollama behind an optional profile) and deploys with the shared tooling:

zdeploy evoai

The server's .env (including a generated SECRET_KEY) lives on the host and is preserved across deploys. In production, run with DEV_MODE=false and a reachable EVOPLATFORM_JWKS_URL so tokens are verified and tenants isolated — see Multi-tenancy & security.

Running fully local (Ollama) keeps every prompt and retrieved chunk on the box; pointing a tenant at a cloud provider sends its prompts (including retrieved context) to that provider. Choose per tenant via model configuration.

Quality gates

Two gates are meant to run after every change to the retrieval or answer path, before shipping.

Latency gate

scripts/run_latency.py times representative questions per pipeline path (gate, analytics count, field lookup, vector, two-turn follow-up), averaged over several runs, against a committed baseline (eval/latency_baseline.json). Any category more than 25% over its baseline fails and needs review. After an intentional performance change, re-baseline with --update-baseline. Baselines are environment-specific.

python scripts/run_latency.py --token <svc-key> --tenant <site-uuid>

LLM-as-judge suite

scripts/run_judge.py runs a growing set of answer-quality cases (eval/judge.jsonl) — every closed defect adds one, tagged with its issue. Each case runs deterministic asserts first (regex must / must-not, envelope checks), then an LLM judge scores the rubric (PASS/FAIL + reason). --skip-judge runs just the deterministic layer without a judge key.

python scripts/run_judge.py --token <svc-key> --tenant <site-uuid>

Trend reporting

Every gate run is recorded to eval/qa_history.db (SQLite) with timestamp, git commit, and per-metric results. scripts/qa_report.py renders eval/qa_report.html — dependency-free inline-SVG trend charts (latency per path, judge pass rate) plus a table of recent runs. Pass --no-record for throwaway runs.

python scripts/qa_report.py

Retrieval eval

Separately, scripts/run_eval.py scores retrieval against a golden Q&A set — hit rate, MRR, and keyword coverage — and is worth running on any retrieval change:

cp eval/golden.example.jsonl eval/golden.jsonl   # then edit with real Q&A
python scripts/run_eval.py eval/golden.jsonl --min-hit-rate 0.7

Add --min-hit-rate as a CI gate to fail the build when retrieval regresses.