output-dev-eval-testing
Create offline evaluation tests for Output SDK workflows using @outputai/evals. Use when implementing test evaluators with verify(), creating dataset YAML files, building eval workflows, or running workflow tests via CLI.
Install
With your agent
Install OMGSkills skill "growthxai/output:coding_assistants/claude/plugins/outputai/skills/output-dev-eval-testing" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/growthxai/output /tmp/output && ln -s /tmp/output/coding_assistants/claude/plugins/outputai/skills/output-dev-eval-testing ~/.claude/skills/output-dev-eval-testingFrom README
Offline Evaluation Testing Overview The @outputai/evals package provides an offline evaluation framework for testing workflow quality using datasets and evaluators. This is complementary to the runtime evaluator() from @outputai/core: | Aspect | Runtime Evaluators (@outputai/core) | Offline Eval Tests (@outputai/evals) | |--------|----------------------------------------|------------------------------------------| | When | During workflow execution | After execution, at test time | | Where | evaluators.ts in workflow folder | tests/evals/ in workflow folder | | Purpose | Live quality scoring with confidence | Dataset-driven pass/fail verification | | Triggered by | Workflow orchestration | output workflow test CLI command | | Returns | EvaluationBooleanResult, etc. | Verdict helpers (pass/partial/fail) | Use offline eval testing when you want to validate workflow behavior against known datasets, build regression test suites, or assess subjective quality with LLM judges.
More from this repo
output-build-workflow
Implement an Output SDK workflow from a plan document.
output-credentials-edit
View, edit, and set encrypted credentials in an Output.ai project.
output-credentials-env-vars
Wire encrypted credentials to environment variables using the credential: convention.