output-eval-validate-judge
Validate LLM judges against human labels using TPR/TNR metrics and train/dev/test splits. Use after writing a judge prompt to verify it agrees with human judgment.
Install
With your agent
Install OMGSkills skill "growthxai/output:coding_assistants/claude/plugins/outputai/skills/output-eval-validate-judge" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/growthxai/output /tmp/output && ln -s /tmp/output/coding_assistants/claude/plugins/outputai/skills/output-eval-validate-judge ~/.claude/skills/output-eval-validate-judgeFrom README
Validating LLM Judges Overview An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite. Prerequisites A judge .prompt file — Written following output-eval-judge-prompt 100 human-labeled traces — With binary pass/fail labels for the failure mode this judge targets. Aim for 50 pass and 50 fail. Minimum: 20 pass and 20 fail. Labels stored in dataset YAML — Each dataset has groundtruth.evals. .verdict: pass or fail This process applies only to LLM-based judges. For code-based Verdict. evaluators, write unit tests instead.
More from this repo
output-build-workflow
Implement an Output SDK workflow from a plan document.
output-credentials-edit
View, edit, and set encrypted credentials in an Output.ai project.
output-credentials-env-vars
Wire encrypted credentials to environment variables using the credential: convention.