# build-a-scorer

> Claude/Codex skill by [@mlflow](https://omgskills.com/library/mlflow/index.md) · 82 stars · Updated 2026-10-23

Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI workflow, including MemAlign and domain-expert labels. Build a small set of atomic quality criteria with the cheapest reliable implementation.

## Install

```sh
git clone https://github.com/mlflow/skills /tmp/skills && ln -s /tmp/skills/build-a-scorer ~/.claude/skills/build-a-scorer
```

## From README

build-a-scorer Help the user get from zero to shipped evaluation prototype. Not 100% correctness on the first pass — a stable, understandable scorer suite they can run, inspect, and iterate on. Your primary job is understanding, and the user is the expert on what counts as wrong. Do not start from "which MLflow scorer should I use?" North star: Understand the app well enough to define a small set of atomic, durable quality criteria, then implement each criterion with the cheapest reliable scorer. Doctrine Prototype first. Ship a useful v1 that runs now. Mark v2 upgrades explicitly. Small suite. 3-5 high-signal scorers, hard cap. Adding one means dropping a weaker one. One scorer, one criterion. If a criterion contains "and", split it. No "overall quality" judge. Cheapest reliable implementation. Code/rules beat built-ins beat hand-written LLM judges. See Phase 4 — the rule most often skipped, and skipping it costs real money at scale.

## Links

- HTML page: https://omgskills.com/library/mlflow/skills/build-a-scorer/
- GitHub: https://github.com/mlflow/skills
- Author: [@mlflow](https://omgskills.com/library/mlflow/index.md)

## More from this repo

- [agent-evaluation](https://omgskills.com/library/mlflow/skills/agent-evaluation/index.md) — Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving...
   - [@mlflow](https://omgskills.com/library/mlflow/index.md) · 82 stars
- [analyzing-mlflow-session](https://omgskills.com/library/mlflow/skills/analyze-mlflow-chat-session/index.md) — Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction.
   - [@mlflow](https://omgskills.com/library/mlflow/index.md) · 82 stars
- [analyzing-mlflow-trace](https://omgskills.com/library/mlflow/skills/analyze-mlflow-trace/index.md) — Analyzes a single MLflow trace to answer a user query about it.
   - [@mlflow](https://omgskills.com/library/mlflow/index.md) · 82 stars