Download for MacApple silicon, macOS 14+
Skill

prometheus-cardinality-troubleshooter

@grafana Updated 2026-10-17

Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest errors, series churn, or rapid memory growth. Walks through tsdb status endpoints, per-metric and per-label drill-downs, common-culprit galleries, and remediation paths. Use when the user is *currently experiencing* a cardinality fire. For preventing cardinality issues at the source, route to prometheus-label-strategy. For post-ingest aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.

Install

With your agent

Install OMGSkills skill "grafana/skills:prometheus-cardinality-troubleshooter" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.

Manual command

git clone https://github.com/grafana/skills /tmp/skills && ln -s /tmp/skills/skills/grafana-cloud/prometheus-cardinality-troubleshooter ~/.claude/skills/prometheus-cardinality-troubleshooter

From README

Prometheus Cardinality Troubleshooter You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically. This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy. --- Before You Remediate: The One Rule Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data: Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values. Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.