build-a-scorer
Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI workflow, including MemAlign and domain-expert labels. Build a small set of atomic quality criteria with the cheapest reliable implementation.
Install
With your agent
Install OMGSkills skill "mlflow/skills:build-a-scorer" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/mlflow/skills /tmp/skills && ln -s /tmp/skills/build-a-scorer ~/.claude/skills/build-a-scorerFrom README
build-a-scorer Help the user get from zero to shipped evaluation prototype. Not 100% correctness on the first pass — a stable, understandable scorer suite they can run, inspect, and iterate on. Your primary job is understanding, and the user is the expert on what counts as wrong. Do not start from "which MLflow scorer should I use?" North star: Understand the app well enough to define a small set of atomic, durable quality criteria, then implement each criterion with the cheapest reliable scorer. Doctrine Prototype first. Ship a useful v1 that runs now. Mark v2 upgrades explicitly. Small suite. 3-5 high-signal scorers, hard cap. Adding one means dropping a weaker one. One scorer, one criterion. If a criterion contains "and", split it. No "overall quality" judge. Cheapest reliable implementation. Code/rules beat built-ins beat hand-written LLM judges. See Phase 4 — the rule most often skipped, and skipping it costs real money at scale.
More from this repo
agent-evaluation
Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving...
analyzing-mlflow-session
Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction.
analyzing-mlflow-trace
Analyzes a single MLflow trace to answer a user query about it.