output-eval-audit
Audit an existing eval suite for trustworthiness. Use when inheriting evals, suspecting evals miss real failures, or after significant pipeline changes.
Install
With your agent
Install OMGSkills skill "growthxai/output:coding_assistants/claude/plugins/outputai/skills/output-eval-audit" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/growthxai/output /tmp/output && ln -s /tmp/output/coding_assistants/claude/plugins/outputai/skills/output-eval-audit ~/.claude/skills/output-eval-auditFrom README
Auditing an Eval Suite Overview Audit your eval suite to determine whether it actually catches real failures. This skill provides a structured diagnostic that identifies gaps in error analysis, evaluator design, judge validation, and dataset coverage, with concrete remediation steps for each finding. When to Use Inheriting an eval suite from another team or developer Suspecting that evals pass but production quality is poor After switching models, rewriting prompts, or changing pipeline logic Periodic health check (quarterly or after major releases) Step 1: Gather Artifacts Read the eval infrastructure files for the workflow being audited: Inventory what exists: | Artifact | File(s) | Count | |----------|---------|-------| | Evaluators | tests/evals/evaluators.ts | ? | | Eval workflow | tests/evals/workflow.ts | ? entries in evals array | | Judge prompts | tests/evals/.prompt | ? | | Datasets | tests/datasets/.yml | ? | | Datasets with groundtruth | ? of above | ?
More from this repo
output-build-workflow
Implement an Output SDK workflow from a plan document.
output-credentials-edit
View, edit, and set encrypted credentials in an Output.ai project.
output-credentials-env-vars
Wire encrypted credentials to environment variables using the credential: convention.