prometheus-cardinality-troubleshooter
Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest errors, series churn, or rapid memory growth. Walks through tsdb status endpoints, per-metric and per-label drill-downs, common-culprit galleries, and remediation paths. Use when the user is *currently experiencing* a cardinality fire. For preventing cardinality issues at the source, route to prometheus-label-strategy. For post-ingest aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.
Install
git clone https://github.com/grafana/skills /tmp/skills && ln -s /tmp/skills/skills/grafana-cloud/prometheus-cardinality-troubleshooter ~/.claude/skills/prometheus-cardinality-troubleshooter
From README
Prometheus Cardinality Troubleshooter You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically. This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy. --- Before You Remediate: The One Rule Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data: Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values. Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.
More from this repo
adaptive-metrics
Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules —...
admin
Used to manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO/SAML/OAuth/GitHub auth,...
admission-control
Used when the user asks to "write a validator", "add validation", "implement admission control", "write a mutating...
