# output-eval-validate-judge

> Claude/Codex skill by [@growthxai](https://omgskills.com/library/growthxai/index.md) · 443 stars · Updated 2026-10-22

Validate LLM judges against human labels using TPR/TNR metrics and train/dev/test splits. Use after writing a judge prompt to verify it agrees with human judgment.

Tags: agents, ai, ai-agents, ai-sdk, claude, claude-code, javascript, llm, nodejs, open-source, reliability, temporal, typescript, workflow-engine, workflows

## Install

```sh
git clone https://github.com/growthxai/output /tmp/output && ln -s /tmp/output/coding_assistants/claude/plugins/outputai/skills/output-eval-validate-judge ~/.claude/skills/output-eval-validate-judge
```

## From README

Validating LLM Judges Overview An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite. Prerequisites A judge .prompt file — Written following output-eval-judge-prompt 100 human-labeled traces — With binary pass/fail labels for the failure mode this judge targets. Aim for 50 pass and 50 fail. Minimum: 20 pass and 20 fail. Labels stored in dataset YAML — Each dataset has groundtruth.evals. .verdict: pass or fail This process applies only to LLM-based judges. For code-based Verdict. evaluators, write unit tests instead.

## Links

- HTML page: https://omgskills.com/library/growthxai/output/coding_assistants/claude/plugins/outputai/skills/output-eval-validate-judge/
- GitHub: https://github.com/growthxai/output
- Author: [@growthxai](https://omgskills.com/library/growthxai/index.md)

## More from this repo

- [output-build-workflow](https://omgskills.com/library/growthxai/output/coding_assistants/claude/plugins/outputai/skills/output-build-workflow/index.md) — Implement an Output SDK workflow from a plan document.
   - [@growthxai](https://omgskills.com/library/growthxai/index.md) · 443 stars
- [output-credentials-edit](https://omgskills.com/library/growthxai/output/coding_assistants/claude/plugins/outputai/skills/output-credentials-edit/index.md) — View, edit, and set encrypted credentials in an Output.ai project.
   - [@growthxai](https://omgskills.com/library/growthxai/index.md) · 443 stars
- [output-credentials-env-vars](https://omgskills.com/library/growthxai/output/coding_assistants/claude/plugins/outputai/skills/output-credentials-env-vars/index.md) — Wire encrypted credentials to environment variables using the credential: convention.
   - [@growthxai](https://omgskills.com/library/growthxai/index.md) · 443 stars