output-eval-error-analysis
Systematically review workflow traces to identify failure modes before building evaluators. Use when starting an eval project, after significant pipeline changes, or when production quality drops.
Install
With your agent
Install OMGSkills skill "growthxai/output:coding_assistants/claude/plugins/outputai/skills/output-eval-error-analysis" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/growthxai/output /tmp/output && ln -s /tmp/output/coding_assistants/claude/plugins/outputai/skills/output-eval-error-analysis ~/.claude/skills/output-eval-error-analysisFrom README
Error Analysis for Workflow Evaluation Overview Review real workflow traces and categorize how your workflow fails before writing any evaluators. Evaluators built without error analysis target generic qualities ("is this good?") instead of the specific ways your workflow actually breaks. This skill walks you through the process. When to Use Starting a new eval project for an existing workflow Production quality has dropped and you need to understand why After significant prompt, model, or pipeline changes Before building your first evaluator for a workflow Step 1: Collect Traces Gather 50-100 representative workflow executions. More traces = more reliable failure categories. From recent runs List recent workflow executions and pull their traces: From production (bulk download) Download production traces directly into dataset YAML files: This creates YAML files in tests/datasets/ with the input and lastoutput fields populated from real executions.
More from this repo
output-build-workflow
Implement an Output SDK workflow from a plan document.
output-credentials-edit
View, edit, and set encrypted credentials in an Output.ai project.
output-credentials-env-vars
Wire encrypted credentials to environment variables using the credential: convention.