output-eval-dataset-design

From README

Designing Eval Datasets Overview Diverse datasets catch more failures. This skill teaches dimension-based dataset design — systematically varying inputs along axes that target failure-prone regions of your workflow. The output is a set of YAML dataset files ready for output workflow test. When to Use Bootstrapping an eval dataset for a new workflow Existing datasets only cover happy paths Real production traces are sparse (fewer than 50) Stress-testing specific failure hypotheses from error analysis When NOT to Use You already have 100+ representative real traces — use stratified sampling from those instead of generating synthetic data You haven't done error analysis yet — do that first (output-eval-error-analysis) so your dimensions target real failure modes, not guesses Step 1: Define Dimensions Identify 3+ axes of input variation that target anticipated failure modes. Each dimension should vary one aspect of the input that you expect to influence output quality.