phoenix-evals
Build, run, and validate evaluators for AI/LLM applications with Phoenix, in Python or TypeScript (including Vercel AI SDK apps) — code-based checks and LLM-as-a-judge, run over exported traces or dataframes or as Phoenix experiments, gate them in CI, and validate judges against human-labeled data.
Install
git clone https://github.com/Arize-ai/phoenix /tmp/phoenix && ln -s /tmp/phoenix/.agents/skills/phoenix-evals ~/.claude/skills/phoenix-evalsFrom README
Phoenix Evals Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans. Quick Reference | Task | Files | | ---- | ----- | | | -python, -typescript | | Decide what to evaluate | evaluators-overview | | Choose a judge model | fundamentals-model-selection | | Use pre-built evaluators | evaluators-pre-built | | Build code evaluator | evaluators-code-python, evaluators-code-typescript | | Build LLM evaluator | evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates | | Batch evaluate DataFrame | evaluate-dataframe-python | | Run experiment | experiments-running-python, experiments-running-typescript | | Run evals in a test runner (CI gate) | integrations-pytest, integrations-vitest-jest | | Create dataset | experiments-datasets-python, experiments-datasets-typescript | | Generate synthetic data | experiments-synthetic-python, experiments-synthetic-typescript | | Validate evaluator accuracy | validation...