huggingface-community-evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.
Install
With your agent
Install OMGSkills skill "huggingface/skills:huggingface-community-evals" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/huggingface/skills /tmp/skills && ln -s /tmp/skills/skills/huggingface-community-evals ~/.claude/skills/huggingface-community-evalsFrom README
Overview This skill is for running evaluations against models on the Hugging Face Hub on local hardware. It covers: inspect-ai with local inference lighteval with local inference choosing between vllm, Hugging Face Transformers, and accelerate smoke tests, task selection, and backend fallback strategy It does not cover: Hugging Face Jobs orchestration model-card or model-index edits README table extraction Artificial Analysis imports .evalresults generation or publishing PR creation or community-evals automation If the user wants to run the same eval remotely on Hugging Face Jobs, hand off to the hugging-face-jobs skill and pass it one of the local scripts in this skill. If the user wants to publish results into the community evals workflow, stop after generating the evaluation run and hand off that publishing step to /code/community-evals. All paths below are relative to the directory containing this SKILL.md.
More from this repo
hf-cli
Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers,...
hf-mcp
Use Hugging Face Hub via MCP server tools.
huggingface-best
Used when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to...