model-evaluation
Generates python code that evaluates SageMaker models. Supports two evaluation types: LLM-as-Judge and Custom Scorer. Use when the user says "evaluate my model", "run a benchmark", "test model performance", "how did my model perform", "compare models", or other similar requests.
Install
With your agent
Install OMGSkills skill "awslabs/agent-plugins:plugins/sagemaker-ai/skills/model-evaluation" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/awslabs/agent-plugins /tmp/agent-plugins && ln -s /tmp/agent-plugins/plugins/sagemaker-ai/skills/model-evaluation ~/.claude/skills/model-evaluation
From README
Model Evaluation Generate code that evaluates a SageMaker model. Prerequisites The SDK environment has been verified (SDK version, region, execution role). If not done, activate the sdk-getting-started skill first. Principles One thing at a time. Each response advances exactly one decision. Never combine multiple questions in a single turn. Confirm before proceeding. Wait for the user to agree before moving to the next step. Don't read files until you need them. Only read reference files when you've reached the step that requires them. Don't ask what you already know. If the answer is in conversation history, workflowstate.json, plan.md, or any file you've already read — use it. Confirm if unsure, but don't re-ask. No narration. Share outcomes and ask questions. Keep responses short. No repetition. If you said something before a tool call, don't repeat it after. Scope This skill supports the evaluation feature for SageMaker Serverless Model Customization.
More from this repo
amplify-workflow
Used to build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first).
api-gateway
Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket).
aws-lambda
Design, build, deploy, test, and debug serverless applications with AWS Lambda.
