phoenix-evals
Build, run, and validate evaluators for AI/LLM applications with Phoenix, in Python or TypeScript (including Vercel AI SDK apps) — code-based checks and LLM-as-a-judge, run over exported traces or dataframes or as Phoenix experiments, gate them in CI, and validate judges against human-labeled data.
Install
With your agent
Install OMGSkills skill "arize-ai/phoenix:phoenix-evals" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/Arize-ai/phoenix /tmp/phoenix && ln -s /tmp/phoenix/.agents/skills/phoenix-evals ~/.claude/skills/phoenix-evalsFrom README
Phoenix Evals Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans. Quick Reference | Task | Files | | ---- | ----- | | | -python, -typescript | | Decide what to evaluate | evaluators-overview | | Choose a judge model | fundamentals-model-selection | | Use pre-built evaluators | evaluators-pre-built | | Build code evaluator | evaluators-code-python, evaluators-code-typescript | | Build LLM evaluator | evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates | | Batch evaluate DataFrame | evaluate-dataframe-python | | Run experiment | experiments-running-python, experiments-running-typescript | | Run evals in a test runner (CI gate) | integrations-pytest, integrations-vitest-jest | | Create dataset | experiments-datasets-python, experiments-datasets-typescript | | Generate synthetic data | experiments-synthetic-python, experiments-synthetic-typescript | | Validate evaluator accuracy | validation...