prometheus-cardinality-troubleshooter
Diagnostic guide for active Prometheus cardinality problems — slow queries, OOMing Prometheus, high Grafana Cloud Active Series or DPM bills, "too many samples" ingest errors, series churn, or rapid memory growth. Walks through tsdb status endpoints, per-metric and per-label drill-downs, common-culprit galleries, and remediation paths. Use when the user is *currently experiencing* a cardinality fire. For preventing cardinality issues at the source, route to prometheus-label-strategy. For post-ingest aggregation, route to adaptive-metrics. For DPM-specific analysis, route to dpm-finder.
Install
With your agent
Install OMGSkills skill "grafana/skills:prometheus-cardinality-troubleshooter" into this agent's normal skills directory: call the OMGSkills MCP get_install_plan tool for your agent and run the script it returns (it installs only the pinned commit and verifies SKILL.md), and stop if MCP or the pin is unavailable. Do not overwrite or change existing skills without asking me.
Manual command
git clone https://github.com/grafana/skills /tmp/skills && ln -s /tmp/skills/skills/grafana-cloud/prometheus-cardinality-troubleshooter ~/.claude/skills/prometheus-cardinality-troubleshooter
From README
Prometheus Cardinality Troubleshooter You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically. This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy. --- Before You Remediate: The One Rule Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data: Counter resets from different series get merged → rate() and increase() return garbage, often absurdly high values. Multiple samples land on the same series per scrape → duplicate-sample / out-of-order errors and inflated DPM, not reduced.
More from this repo
k6-docs
Used to write or review k6 documentation across the three k6 repositories - k6-DefinitelyTyped (TypeScript types),...
adaptive-metrics
Cut Grafana Cloud Metrics cost by shrinking active-series count with Adaptive Metrics aggregation rules —...
admin
Used to manage Grafana Cloud accounts — organizations, stacks, RBAC roles and assignments, SSO/SAML/OAuth/GitHub auth,...
