feat: add grafana dashboarding and promql agent skills
This commit is contained in:
Moritz Graf
@@ -0,0 +1,103 @@
|
||||
---
|
||||
name: promql
|
||||
license: Apache-2.0
|
||||
description: Write, validate, and optimize PromQL for Prometheus / Grafana Mimir / Grafana Cloud Metrics. Covers `rate` vs `irate` vs `increase`, label matchers and regex, `sum / avg / topk / by / without` aggregation, classic + native `histogram_quantile`, ratios with divide-by-zero guards, `absent` / `changes` for staleness, time offsets and `predict_linear`, recording-rule naming, SLO + burn-rate math, and a cardinality-hunting playbook. Use when writing a metric query, fixing wrong p95s, building an error-budget alert, debugging "query is slow", finding the noisy label that blew up cardinality, or migrating a dashboard query to a recording rule — even when the user says "calculate the error rate", "p99 latency", "sum by service", "why is this query slow", or "what's filling Mimir" without naming PromQL.
|
||||
---
|
||||
|
||||
# PromQL Query Patterns
|
||||
|
||||
> **Docs**: https://prometheus.io/docs/prometheus/latest/querying/basics/
|
||||
|
||||
PromQL returns either an **instant vector**, a **range vector**, or a **scalar**.
|
||||
|
||||
**Golden rule:** `rate()` / `increase()` require a range vector ≥ 4× the scrape interval. 60s scrape → use `[5m]` minimum.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A Prometheus / Mimir / Grafana Cloud endpoint to query (`/api/v1/query` or via Grafana Explore)
|
||||
- The PromQL pattern library in [`references/patterns.md`](references/patterns.md)
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### 1. Write + validate a query
|
||||
|
||||
```bash
|
||||
# 0. Point at your Prometheus/Mimir. For Grafana Cloud, use the metrics endpoint
|
||||
# and add basic auth (-u "<metrics_user>:<token>") to each curl below.
|
||||
PROM=http://localhost:9090 # or https://prometheus-prod-XX.grafana.net/api/prom
|
||||
|
||||
# 1. Sketch the query — for "5xx error rate per service":
|
||||
EXPR='sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)'
|
||||
|
||||
# 2. Validate syntax + that the metric/labels exist
|
||||
curl -sG --data-urlencode "query=${EXPR}" \
|
||||
"$PROM/api/v1/query" | jq '.status, (.data.result|length)'
|
||||
# Expect: "success" and result count > 0. If 0 — check label spelling and scrape activity:
|
||||
curl -sG --data-urlencode "match[]=http_requests_total" "$PROM/api/v1/series" | jq '.data | length'
|
||||
|
||||
# 3. Sanity-check the magnitude — open Grafana Explore, paste the expr,
|
||||
# confirm the values look right against a known ground truth (k6 run, log count, etc.)
|
||||
```
|
||||
|
||||
### 2. Common patterns to copy
|
||||
|
||||
**Per-status request rate** (aggregate AFTER rate):
|
||||
|
||||
```promql
|
||||
sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
|
||||
```
|
||||
|
||||
**p95 latency** (must keep `le` in the inner aggregation):
|
||||
|
||||
```promql
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
|
||||
```
|
||||
|
||||
**Error rate with divide-by-zero guard:**
|
||||
|
||||
```promql
|
||||
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
|
||||
/ (sum(rate(http_requests_total[5m])) > 0)
|
||||
```
|
||||
|
||||
Full library (recording rules, SLO burn-rate, offsets, cardinality hunt, native histograms): [`references/patterns.md`](references/patterns.md).
|
||||
|
||||
### 3. Convert a slow dashboard query into a recording rule
|
||||
|
||||
```yaml
|
||||
# 1. Pick the slow expression, give it a recording-rule name
|
||||
groups:
|
||||
- name: http_request_rates
|
||||
interval: 1m
|
||||
rules:
|
||||
- record: job:http_request_duration_p95:rate5m
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
|
||||
```
|
||||
|
||||
```bash
|
||||
# 2. After rules load, verify the new metric exists
|
||||
curl -sG --data-urlencode "query=job:http_request_duration_p95:rate5m" \
|
||||
"$PROM/api/v1/query" | jq '.data.result | length' # → > 0
|
||||
|
||||
# 3. Verify it matches the original expression for at least one sample window
|
||||
# (Both queries should produce the same value at the same timestamp.)
|
||||
|
||||
# 4. Replace the dashboard panel expression with the recording-rule metric.
|
||||
```
|
||||
|
||||
## Common bugs
|
||||
|
||||
- `histogram_quantile` returns NaN → forgot `by (le)` in the inner aggregation
|
||||
- "No data" → check the metric exists (`/api/v1/series`) and the window ≥ 4× scrape interval
|
||||
- Wrong rate magnitude → counter was aggregated before `rate()` (always `rate()` first)
|
||||
- Query timeout → series count too high; use `topk(...)` + a recording rule + drop high-cardinality labels (see [`references/patterns.md`](references/patterns.md))
|
||||
|
||||
## Resources
|
||||
|
||||
- [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
- [Operators](https://prometheus.io/docs/prometheus/latest/querying/operators/)
|
||||
- [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)
|
||||
- [Grafana Mimir](https://grafana.com/docs/mimir/latest/)
|
||||
@@ -0,0 +1,174 @@
|
||||
# PromQL pattern library
|
||||
|
||||
## Rate, counter, and aggregation
|
||||
|
||||
```promql
|
||||
# rate() — per-second average
|
||||
rate(http_requests_total[5m])
|
||||
|
||||
# CORRECT: rate first, then aggregate
|
||||
sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
|
||||
|
||||
# WRONG: never sum() a raw counter before rate()
|
||||
sum(http_requests_total) by (status_code) # do NOT then rate()
|
||||
|
||||
# increase() — total count over window
|
||||
increase(http_requests_total[1h])
|
||||
```
|
||||
|
||||
`rate` vs `irate`:
|
||||
- `rate()` smooths the full window — use for dashboards + alerts
|
||||
- `irate()` uses the last two samples — for spike detection only, never alerting
|
||||
|
||||
## Label matchers
|
||||
|
||||
```promql
|
||||
http_requests_total{job="api", status_code="200"}
|
||||
http_requests_total{status_code=~"5.."}
|
||||
http_requests_total{status_code!~"2.."}
|
||||
http_requests_total{env=~"staging|production"}
|
||||
```
|
||||
|
||||
## Aggregation
|
||||
|
||||
```promql
|
||||
sum(rate(http_requests_total[5m])) by (service)
|
||||
avg(node_cpu_seconds_total{mode="idle"}) by (instance)
|
||||
|
||||
# Top 5 by request rate
|
||||
topk(5, sum(rate(http_requests_total[5m])) by (service))
|
||||
|
||||
# Count of distinct label values
|
||||
count(count(up) by (job)) by ()
|
||||
```
|
||||
|
||||
`by` keeps only listed labels; `without` drops only listed labels.
|
||||
|
||||
## Histogram quantiles
|
||||
|
||||
```promql
|
||||
# Classic histogram
|
||||
histogram_quantile(0.99,
|
||||
sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le))
|
||||
|
||||
# Multi-service comparison — must include le AND service
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
|
||||
|
||||
# Native histograms (Prometheus 2.40+)
|
||||
histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m])))
|
||||
```
|
||||
|
||||
Forgetting `by (le)` is the most common bug — `histogram_quantile` returns NaN or wrong values.
|
||||
|
||||
## Ratios and error rates
|
||||
|
||||
```promql
|
||||
# Error fraction
|
||||
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
|
||||
/ sum(rate(http_requests_total[5m]))
|
||||
|
||||
# Success percentage
|
||||
(1 - sum(rate(http_requests_total{status_code=~"5.."}[5m]))
|
||||
/ sum(rate(http_requests_total[5m]))) * 100
|
||||
|
||||
# Avoid divide-by-zero
|
||||
sum(rate(errors_total[5m]))
|
||||
/ (sum(rate(requests_total[5m])) > 0)
|
||||
```
|
||||
|
||||
## Absence and staleness
|
||||
|
||||
```promql
|
||||
absent(up{job="api"}) # metric disappeared
|
||||
changes(up{job="api"}[5m]) == 0 # value not changing → stale exporter
|
||||
count_over_time(up{job="api"}[5m]) > 0 # at least one sample in window
|
||||
```
|
||||
|
||||
## Time offsets
|
||||
|
||||
```promql
|
||||
# Current vs 1h ago
|
||||
rate(http_requests_total[5m])
|
||||
- rate(http_requests_total[5m] offset 1h)
|
||||
|
||||
# Day-over-day
|
||||
rate(http_requests_total[5m])
|
||||
/ rate(http_requests_total[5m] offset 1d)
|
||||
|
||||
# Predict 2h ahead from 1h trend (linear regression)
|
||||
predict_linear(node_filesystem_avail_bytes[1h], 2 * 3600)
|
||||
```
|
||||
|
||||
## Recording rules
|
||||
|
||||
Naming: `<aggregation_level>:<metric_name>:<operation_and_window>`
|
||||
|
||||
```yaml
|
||||
groups:
|
||||
- name: http_request_rates
|
||||
interval: 1m
|
||||
rules:
|
||||
- record: job:http_requests_total:rate5m
|
||||
expr: sum(rate(http_requests_total[5m])) by (job)
|
||||
|
||||
- record: job:http_errors:ratio5m
|
||||
expr: |
|
||||
sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)
|
||||
/ sum(rate(http_requests_total[5m])) by (job)
|
||||
|
||||
- record: job:http_request_duration_p95:rate5m
|
||||
expr: |
|
||||
histogram_quantile(0.95,
|
||||
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
|
||||
```
|
||||
|
||||
## SLO + burn-rate
|
||||
|
||||
```promql
|
||||
# Availability SLO over 30 days
|
||||
1 - (sum(increase(http_requests_total{status_code=~"5.."}[30d]))
|
||||
/ sum(increase(http_requests_total[30d])))
|
||||
|
||||
# 1h burn-rate vs target (replace 0.999 with your SLO target)
|
||||
( sum(rate(http_requests_total{status_code=~"5.."}[1h]))
|
||||
/ sum(rate(http_requests_total[1h])) )
|
||||
/ (1 - 0.999)
|
||||
```
|
||||
|
||||
## Cardinality controls
|
||||
|
||||
```promql
|
||||
# Top 10 metrics by series count
|
||||
topk(10, count by (__name__)({__name__=~".+"}))
|
||||
|
||||
# Series count for one metric
|
||||
count(http_requests_total)
|
||||
|
||||
# Cardinality of one label
|
||||
count(count by (user_id)(http_requests_total))
|
||||
```
|
||||
|
||||
Drop a high-cardinality label at scrape time (Alloy):
|
||||
|
||||
```alloy
|
||||
prometheus.scrape "api" {
|
||||
targets = [...]
|
||||
rule { source_labels = ["user_id"]
|
||||
action = "labeldrop" }
|
||||
}
|
||||
```
|
||||
|
||||
## Other common patterns
|
||||
|
||||
```promql
|
||||
# Service availability (alert)
|
||||
avg_over_time(up{job="api"}[5m]) < 0.9
|
||||
|
||||
# Saturation — disk full in <4h
|
||||
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
|
||||
|
||||
# Throughput spike — current rate > 3x 1h average
|
||||
rate(http_requests_total[5m])
|
||||
> 3 * avg_over_time(rate(http_requests_total[5m])[1h:5m])
|
||||
```
|
||||
Reference in New Issue
Block a user