arize-evaluator
Handles LLM-as-judge and code evaluator workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.
Install
git clone https://github.com/Arize-ai/arize-skills /tmp/arize-skills && ln -s /tmp/arize-skills/skills/arize-evaluator ~/.claude/skills/arize-evaluatorFrom README
Arize Evaluator Skill SPACE — --space flags accept a space name (e.g., my-workspace) or a base64 space ID (e.g., U3BhY2U6...). Find yours with ax spaces list. This skill covers designing, creating, and running evaluators on Arize — both LLM-as-judge (template) evaluators and code evaluators (deterministic, no LLM required). An evaluator defines the judge; a task is how you run it against real data. --- Prerequisites Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront. If an ax command fails, troubleshoot based on the error: command not found or version error → see references/ax- .md 401 Unauthorized / missing API key → run ax profiles show to inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it.
More from this repo
arize-admin
Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the...
arize-ai-provider-integration
Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and...
arize-annotation
Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human...