agent-evaluation
Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong/incomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).
Install
With your agent
Install OMGSkills skill "mlflow/skills:agent-evaluation" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/mlflow/skills /tmp/skills && ln -s /tmp/skills/agent-evaluation ~/.claude/skills/agent-evaluationFrom README
Agent Evaluation with MLflow Comprehensive guide for evaluating GenAI agents with MLflow. Use this skill for the complete evaluation workflow or individual components - tracing , environment configuration, dataset creation, scorer definition, or evaluation execution. Each section can be used independently based on your needs. ⛔ CRITICAL: Must Use MLflow APIs DO NOT create custom evaluation frameworks. You MUST use MLflow's native APIs: Datasets: Use mlflow.genai.datasets.createdataset() - NOT custom test case files Scorers: Use MLflow built-ins, @scorer, or mlflow.genai.judges.makejudge() - NOT a custom evaluation framework Evaluation: Use mlflow.genai.evaluate() - NOT custom evaluation loops Scripts: Use the provided scripts/ directory templates - NOT custom evaluation/ directories Why? MLflow tracks everything (datasets, scorers, traces, results) in the experiment. Custom frameworks bypass this and lose all observability.
More from this repo
analyzing-mlflow-session
Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction.
analyzing-mlflow-trace
Analyzes a single MLflow trace to answer a user query about it.
build-a-scorer
Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI...