feat: add grafana dashboarding and promql agent skills

2026-07-03 12:48:42 +02:00 · 2026-07-03 12:48:42 +02:00 · 304ca52cca
parent 3f7c501340
commit 304ca52cca
13 changed files with 2463 additions and 0 deletions
--- a/.agents/skills/dashboarding/SKILL.md
+++ b/.agents/skills/dashboarding/SKILL.md
@ -0,0 +1,132 @@
+---
+name: dashboarding
+license: Apache-2.0
+description: Build, modify, and ship Grafana dashboards as JSON via the HTTP API — panel types (timeseries / stat / gauge / table / heatmap / logs / traces / node-graph), `gridPos` 24-column layout, units, thresholds, template + datasource + chained variables, transformations (`organize` / `calculateField` / `filterByValue`), panel + dashboard links with `${__field.labels.x}` / `${__from}`, and Loki/Prometheus annotations. Use when scripting dashboard creation, writing the dashboard JSON for a new service, adding a `$job` dropdown variable, computing an "Error %" column with a transformation, overlaying deploys as annotations, or pushing a dashboard via `POST /api/dashboards/db` — even when the user says "create a dashboard for this metric", "add a service dropdown", "show errors as percentage", "overlay our deploys", or "export the dashboard JSON" without naming the API or schema. After every API push, verify with the returned `version` plus a GET on the dashboard UID.
+---
+
+# Grafana Dashboard Authoring
+
+> **Docs**: https://grafana.com/docs/grafana/latest/dashboards/
+
+Dashboards are JSON. Author once, push via API, share by `uid`.
+
+## Prerequisites
+
+- Grafana stack (OSS, Enterprise, or Cloud) reachable from your machine
+- API token with `dashboards:write` (`Authorization: Bearer <token>`)
+- `jq` for inspecting responses
+- The JSON-schema cheat sheet in [`references/json-schema.md`](references/json-schema.md)
+
+## Common Workflows
+
+### 1. Push a new dashboard via the API + verify
+
+```bash
+# 1. Build the payload — wrap the dashboard JSON, set folder, mark overwrite
+cat > /tmp/dash.json <<'JSON'
+{
+  "dashboard": {
+    "uid": "demo-svc-v1",
+    "title": "Demo Service",
+    "schemaVersion": 41,
+    "tags": ["demo"],
+    "time": { "from": "now-1h", "to": "now" },
+    "templating": { "list": [] },
+    "panels": [{
+      "id": 1, "type": "timeseries", "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 24, "h": 8 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{
+        "expr": "sum(rate(http_requests_total[5m])) by (status_code)",
+        "legendFormat": "{{status_code}}", "refId": "A"
+      }],
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] }
+    }]
+  },
+  "folderUid": "",
+  "overwrite": true,
+  "message": "initial push"
+}
+JSON
+
+# 2. Validate the JSON BEFORE you send it (catches trailing-comma typos)
+jq empty /tmp/dash.json && echo "json ok"
+
+# 3. POST
+RESP=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  "$GRAFANA/api/dashboards/db" -d @/tmp/dash.json)
+echo "$RESP" | jq '{status, uid, url, version}'
+# Expect: status="success", url="/d/demo-svc-v1/...", version=1 (incremented on each push)
+
+# 4. Verify the round-trip — read it back and confirm one panel + the expected title
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" \
+  | jq '{title: .dashboard.title, panels: (.dashboard.panels | length)}'
+# Expect: {"title":"Demo Service","panels":1}
+
+# 5. Open the dashboard in a browser — confirm the panel renders with data.
+```
+
+### 2. Add a `$job` template variable to an existing dashboard
+
+```bash
+# 1. Fetch existing dashboard
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" > /tmp/dash.json
+
+# 2. Edit templating.list — append:
+#   { "name":"job", "type":"query",
+#     "datasource":{"type":"prometheus","uid":"prometheus"},
+#     "query":{"query":"label_values(up, job)","refId":"A"},
+#     "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+#  (Use jq, an editor, or the Grafana UI — schema in references/json-schema.md.)
+
+# 3. Update the panel expr to use the variable: rate(http_requests_total{job=~"$job"}[5m])
+
+# 4. POST it back with overwrite: true. Verify the variable appears in the UI dropdown.
+```
+
+### 3. Compute an "Error %" column with a transformation
+
+```json
+{
+  "id": "calculateField",
+  "options": {
+    "alias": "Error %", "mode": "reduceRow",
+    "reduce": { "reducer": "last" },
+    "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }
+}
+```
+
+Add this to the panel's `transformations: []`. Verify in the UI panel inspector — the new field should appear and update with the variable selection.
+
+Full schema (panels, units, all transformations, annotations, links): [`references/json-schema.md`](references/json-schema.md).
+
+## API reference
+
+```bash
+# Get
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/<uid>" | jq '.dashboard'
+
+# Search
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/search?query=kubernetes&type=dash-db" | jq '.[] | {uid,title,folderTitle}'
+
+# Create folder
+curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" "$GRAFANA/api/folders" \
+  -d '{"uid":"platform-team","title":"Platform Team"}'
+```
+
+For dashboards embedded in app plugins, use `@grafana/scenes` (skill `grafana-o11y:grafana-scenes`).
+
+## Resources
+
+- [Dashboard JSON model](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/)
+- [HTTP API — dashboards](https://grafana.com/docs/grafana/latest/developers/http_api/dashboard/)
+- [Panel types](https://grafana.com/docs/grafana/latest/panels-visualizations/)
+- [Variables](https://grafana.com/docs/grafana/latest/dashboards/variables/)
+- [Transformations](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/)
--- a/.agents/skills/dashboarding/references/json-schema.md
+++ b/.agents/skills/dashboarding/references/json-schema.md
@ -0,0 +1,157 @@
+# Dashboard + panel JSON schema
+
+## Dashboard root
+
+```json
+{
+  "title": "My Dashboard",
+  "uid": "my-dashboard-v1",
+  "tags": ["service", "production"],
+  "time":     { "from": "now-1h", "to": "now" },
+  "refresh":  "30s",
+  "timezone": "browser",
+  "schemaVersion": 41,
+  "templating":  { "list": [] },
+  "annotations": { "list": [] },
+  "panels": []
+}
+```
+
+- `uid` — stable identifier; keep short
+- `schemaVersion` — `41` for Grafana 11+
+- `time.from` / `to` — relative (`now-1h`) or absolute ISO
+- `refresh` — `"30s"`, `"1m"`, `"5m"`, `""` (off)
+
+## Panel
+
+```json
+{
+  "id": 1,
+  "type": "timeseries",
+  "title": "Request Rate",
+  "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+  "datasource": { "type": "prometheus", "uid": "${datasource}" },
+  "targets": [{
+    "expr": "sum(rate(http_requests_total{job=\"$job\"}[5m])) by (status_code)",
+    "legendFormat": "{{status_code}}",
+    "refId": "A"
+  }],
+  "fieldConfig": {
+    "defaults": {
+      "unit": "reqps",
+      "thresholds": { "mode": "absolute", "steps": [
+        { "color": "green",  "value": null },
+        { "color": "yellow", "value": 1000 },
+        { "color": "red",    "value": 5000 }
+      ]}
+    },
+    "overrides": []
+  },
+  "options": {
+    "legend":  { "calcs": ["mean","max","last"], "displayMode": "table", "placement": "bottom" },
+    "tooltip": { "mode": "multi", "sort": "desc" }
+  }
+}
+```
+
+`gridPos`: 24-column grid. Widths: full=24, half=12, third=8, quarter=6. Height 1 unit ≈ 30 px.
+
+## Panel types
+
+| Panel | Use case |
+|---|---|
+| **Time series** | Any metric over time |
+| **Stat** | Single value + sparkline |
+| **Gauge** | % or value against min/max |
+| **Bar gauge** | Side-by-side comparison |
+| **Table** | Multi-column data |
+| **Heatmap** | Distribution over time |
+| **Logs** | Loki log streams |
+| **Traces** | Tempo trace search |
+| **Text** | Markdown docs |
+| **Candlestick** | OHLC / min-max-avg |
+| **Node graph** | Service dependency graph |
+
+## Useful units
+
+```
+reqps        requests/sec
+ops          ops/sec
+Bps          bytes/sec
+percentunit  0.0-1.0 as %
+bytes        bytes (auto-scales)
+decbytes     decimal bytes (1 KB = 1000 B)
+ms / s       milliseconds / seconds
+dtdurationms 1h 2m 3s
+short        compact (1.2k, 3.4M)
+none         raw number
+```
+
+## Template variables
+
+```json
+{ "name":"job", "type":"query", "datasource":{"type":"prometheus","uid":"prometheus"},
+  "query":{"query":"label_values(up, job)","refId":"A"},
+  "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+
+{ "name":"cluster", "type":"constant", "query":"production", "label":"Cluster" }
+
+{ "name":"datasource", "type":"datasource", "pluginId":"prometheus",
+  "includeAll":false, "label":"Prometheus" }
+```
+
+Multi-value variables expand to a regex OR: `$job=["api","worker"]` → `job=~"api|worker"`.
+Chained: `label_values(kube_pod_info{namespace=\"$namespace\"}, pod)`.
+
+## Transformations
+
+```json
+"transformations": [
+  { "id": "merge", "options": {} },
+  { "id": "organize", "options": {
+      "renameByName":  { "Value #A": "Request Rate", "Value #B": "Error Rate" },
+      "excludeByName": { "Time": true }
+  }},
+  { "id": "calculateField", "options": {
+      "alias": "Error %", "mode": "reduceRow",
+      "reduce": { "reducer": "last" },
+      "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }},
+  { "id": "filterByValue", "options": {
+      "filters": [{ "fieldName":"Error %", "config":{ "id":"greater", "options":{ "value":0.01 }}}],
+      "type": "include", "match": "any"
+  }}
+]
+```
+
+Common IDs: `merge`, `organize`, `rename`, `calculateField`, `filterByValue`, `groupBy`, `sortBy`, `limit`, `labelsToFields`, `seriesToRows`, `partitionByValues`.
+
+## Links & annotations
+
+```json
+"links": [
+  { "title":"Go to details", "url":"/d/details?var-service=${__field.labels.service}", "targetBlank":false },
+  { "title":"Runbook",        "url":"https://wiki.example.com/runbook/${job}", "icon":"external link",
+    "targetBlank":true, "type":"link" }
+]
+```
+
+Built-in vars: `${__value.raw}`, `${__field.labels.job}`, `${__url.params}`, `${__from}` / `${__to}` (Unix ms).
+
+Loki annotation:
+
+```json
+{ "datasource":{"type":"loki","uid":"loki"},
+  "expr":"{job=\"deployments\"} |= \"deployed\"",
+  "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"{{service}} deployed", "textFormat":"{{version}} by {{author}}" }
+```
+
+Prometheus annotation:
+
+```json
+{ "datasource":{"type":"prometheus","uid":"prometheus"},
+  "expr":"changes(kube_deployment_status_observed_generation{namespace=\"production\"}[5m]) > 0",
+  "step":"60s", "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"Deploy: {{deployment}}" }
+```
--- a/.agents/skills/grafana-oss/SKILL.md
+++ b/.agents/skills/grafana-oss/SKILL.md
@ -0,0 +1,153 @@
+---
+name: grafana-oss
+license: Apache-2.0
+description: Configure Grafana OSS — provisions dashboards from YAML, sets up data sources (Prometheus / Loki / Tempo / Pyroscope), writes dashboard JSON with template variables, builds panel queries, assigns built-in roles (Viewer / Editor / Admin / GrafanaAdmin), mints service-account tokens, edits grafana.ini server config, creates annotations, installs plugins via provisioning, and validates each step with a health-check curl. Use when building dashboards, configuring data sources, setting up provisioning YAML, picking a panel type, writing template variables, managing users and roles, configuring SMTP/OAuth in grafana.ini, creating annotations via API, troubleshooting why a provisioned dashboard isn't showing up, or running Grafana OSS locally — even when the user says "set up a Prometheus data source", "provision dashboards from git", "make a service account", or "configure SSO in OSS" without saying "Grafana OSS".
+---
+
+# Grafana OSS
+
+> **Docs**: https://grafana.com/docs/grafana/latest.md
+
+## Common Workflows
+
+### Provisioning dashboards from disk
+
+1. Drop dashboard JSON file(s) under `/var/lib/grafana/dashboards/`
+2. Add a provider in `provisioning/dashboards/default.yaml` (see [§ Dashboard provisioning](#dashboard-provisioning) below)
+3. Restart Grafana so the provider config is loaded
+4. **Verify the dashboard landed**:
+   ```bash
+   curl https://grafana.example.com/api/dashboards/uid/<uid> \
+     -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+   ```
+   Returns the title → success. 404 → provisioning didn't pick it up; check Grafana server logs (`journalctl -u grafana-server | grep -i provisioning`) for parse errors.
+
+### Provisioning data sources
+
+1. Write `provisioning/datasources/datasources.yaml` (see [§ Data source provisioning](#data-source-provisioning) below)
+2. Restart Grafana
+3. **Health-check the data source via API**:
+   ```bash
+   curl https://grafana.example.com/api/datasources/uid/<uid>/health \
+     -H "Authorization: Bearer <token>"
+   # { "status": "OK", "message": "..." } → working
+   # { "status": "ERROR", ... } → URL unreachable or auth misconfigured
+   ```
+
+### Creating a service account + token
+
+1. Provision via YAML or `POST /api/serviceaccounts` (full API in [references/api.md § Users + service accounts](references/api.md#users--service-accounts))
+2. Mint a token via `POST /api/serviceaccounts/{id}/tokens`
+3. **Verify the token works**:
+   ```bash
+   curl https://grafana.example.com/api/org \
+     -H "Authorization: Bearer <new-token>"
+   # 200 + org JSON → token + role assignment work
+   # 401 → token wrong; 403 → role wrong
+   ```
+
+## Dashboard provisioning
+
+```yaml
+# provisioning/dashboards/default.yaml
+apiVersion: 1
+providers:
+  - name: default
+    folder: MyFolder
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+For the dashboard JSON shape itself (panels, queries, template variables), see [references/dashboard-json.md](references/dashboard-json.md).
+
+## Data source provisioning
+
+```yaml
+# provisioning/datasources/datasources.yaml
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    jsonData:
+      timeInterval: 15s
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+
+  - name: Tempo
+    type: tempo
+    access: proxy
+    url: http://tempo:3200
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki_uid
+        tags: [{ key: "service.name", value: "app" }]
+      serviceMap:
+        datasourceUid: prometheus_uid
+      nodeGraph:
+        enabled: true
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+```
+
+## RBAC (built-in roles)
+
+| Role | Permissions |
+|------|-------------|
+| **Viewer** | Read dashboards, alerts |
+| **Editor** | Create/edit dashboards, alerts |
+| **Admin** | Manage data sources, users, plugins |
+| **GrafanaAdmin** | Server-wide admin (superuser) |
+
+Service-account provisioning:
+
+```yaml
+# provisioning/access-control/service_accounts.yaml
+apiVersion: 1
+serviceAccounts:
+  - name: ci-reader
+    orgId: 1
+    role: Viewer
+    tokens:
+      - name: ci-token
+        # expires: optional ISO 8601 timestamp; omit for no-expiry tokens
+```
+
+(Custom RBAC roles with fine-grained permissions are Enterprise / Cloud only — see the `grafana-cloud/admin` skill if you need those.)
+
+## Plugin provisioning
+
+```yaml
+# provisioning/plugins/plugins.yaml
+apiVersion: 1
+apps:
+  - type: grafana-pyroscope-app
+    disabled: false
+    jsonData:
+      backendUrl: http://pyroscope:4040
+```
+
+After restart, verify via `GET /api/plugins/<plugin-id>/health`.
+
+## References
+
+- [`references/dashboard-json.md`](references/dashboard-json.md) — full dashboard JSON model + template variables + common problems (uid uniqueness, gridPos arithmetic, datasource uid matching)
+- [`references/dashboards.md`](references/dashboards.md) — dashboard workflows, settings, variables, annotations, sharing, versions, playlists, and provisioning-as-code
+- [`references/datasources.md`](references/datasources.md) — data source setup and query examples for Prometheus, Loki, Tempo, SQL, CloudWatch, and plugins
+- [`references/panel-types.md`](references/panel-types.md) — panel-type table + decision guide for picking the right one
+- [`references/panels.md`](references/panels.md) — panel editor, visualization options, field config, transformations, query options, inspection, and performance tips
+- [`references/alerting.md`](references/alerting.md) — alerting concepts, contact points, notification policies, templates, silences, and common rule examples
+- [`references/api.md`](references/api.md) — full Grafana OSS API reference (dashboards, data sources, users, service accounts, annotations) with verification curls and common failure modes
+- [`references/config.md`](references/config.md) — `grafana.ini` server / database / SMTP / auth / security / feature-toggle config + restart-required issues
--- a/.agents/skills/grafana-oss/references/alerting.md
+++ b/.agents/skills/grafana-oss/references/alerting.md
@ -0,0 +1,425 @@
+# Grafana Alerting - Detailed Reference
+
+## Overview
+
+Grafana Alerting is a unified alerting system for monitoring metrics and logs across multiple data sources. It fires notifications when conditions are breached.
+
+Key capabilities:
+- Query multiple data sources in a single alert rule
+- Multi-dimensional alerts (one rule creates many alert instances)
+- Flexible notification routing via policies
+- Silences and mute timings for planned maintenance
+- Alert history and state tracking
+
+---
+
+## Core Concepts
+
+### Alert Rule
+Defines what to monitor and when to fire. Contains: queries, a condition (threshold), an evaluation interval, and a pending period.
+
+### Alert Instance
+When a rule produces multi-dimensional data, it creates one alert instance per unique label combination. Example: an alert on `cpu_usage{host=~".*"}` creates one instance per host.
+
+### Alert States
+
+| State | Description |
+|-------|-------------|
+| **Normal** | Query running; condition not met |
+| **Pending** | Condition met but pending period not yet elapsed |
+| **Firing** | Condition met + pending period elapsed; notifications sent |
+| **Resolved** | Previously firing alert returned to normal |
+| **No Data** | Query returned no data (configurable behavior) |
+| **Error** | Query failed with an error (configurable behavior) |
+
+### Evaluation Group
+Alert rules are organized into evaluation groups. All rules in a group share the same evaluation interval and are evaluated sequentially.
+
+### Pending Period
+How long the condition must be continuously met before firing.
+- `0s` = fire immediately
+- `5m` = must be in breach for 5 continuous minutes
+
+### Keep Firing For
+How long an alert continues firing after the condition resolves (prevents brief recovery from clearing the alert).
+
+---
+
+## Alert Rule Types
+
+### Grafana-Managed Rules (Recommended)
+- Stored in Grafana's database
+- Can query any data source
+- Support multi-dimensional alerting
+- Support expressions (math, reduce, threshold)
+
+### Data Source-Managed Rules (Prometheus/Mimir/Loki)
+- Rules stored in the external system
+- Evaluated by the external system
+- Grafana provides UI to manage them
+- Useful when migrating from Prometheus alerting
+
+---
+
+## Creating Grafana-Managed Alert Rules
+
+Navigate to: **Alerting > Alert rules > New alert rule**
+
+### Step 1: Define query and condition
+
+Write one or more queries (labeled A, B, C...).
+
+Example with Prometheus:
+```promql
+# Query A: CPU usage per host
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Expression types you can add after queries:
+- **Math**: `$A > 80` or `($A + $B) / 2`
+- **Reduce**: Collapse time series to single value (Last, Mean, Sum, Max, Min)
+- **Resample**: Change time resolution
+- **Classic conditions**: Multiple threshold conditions with AND/OR
+- **Threshold**: Set the firing threshold
+
+### Step 2: Set evaluation behavior
+
+| Setting | Description |
+|---------|-------------|
+| Folder | Organize rules; folder is the RBAC boundary |
+| Evaluation group | Group name (all rules share this group's interval) |
+| Evaluation interval | How often the rule is evaluated (e.g., 1m) |
+| Pending period | How long condition must hold before firing (e.g., 5m) |
+| Keep firing for | How long alert stays firing after recovery |
+
+### Step 3: Configure labels and notifications
+
+**Labels** - Key-value pairs attached to alert instances. Used for routing and grouping:
+```
+severity=critical
+team=infrastructure
+service=database
+environment=production
+```
+
+**Annotations** - Context included in notification messages:
+```
+Summary: CPU usage above 80% on {{ $labels.instance }}
+Description: CPU usage is {{ $values.A.Value | humanize }}% on {{ $labels.instance }}
+Runbook URL: https://runbooks.company.com/cpu-high
+```
+
+**Notification settings**:
+- Set a **Contact point** to send directly, bypassing the routing policy tree
+- Or leave empty to use the notification policy routing tree
+
+### Step 4: No data and error handling
+
+| Situation | Options |
+|-----------|---------|
+| No data | Alerting, OK, No Data state, Keep last state |
+| Query error | Alerting, OK, Error state, Keep last state |
+
+---
+
+## Contact Points
+
+Contact points are notification destinations.
+
+Navigate to: **Alerting > Contact points**
+
+### Supported integrations
+
+| Integration | Notes |
+|-------------|-------|
+| Email | Requires SMTP config in grafana.ini |
+| Slack | Webhook URL or API token + channel |
+| PagerDuty | Integration key |
+| OpsGenie | API key |
+| VictorOps (Splunk) | API key |
+| Microsoft Teams | Incoming webhook URL |
+| Discord | Webhook URL |
+| Telegram | Bot token + chat ID |
+| Webhook | HTTP POST to custom URL |
+| Alertmanager | Forward to external Alertmanager |
+| Pushover | User key + API token |
+| LINE | LINE Notify token |
+
+### Email configuration in grafana.ini
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@company.com
+password = app-password
+from_address = alerts@company.com
+from_name = Grafana Alerts
+```
+
+### Webhook payload format
+Grafana POSTs a JSON payload to webhook endpoints:
+```json
+{
+  "receiver": "webhook-receiver",
+  "status": "firing",
+  "alerts": [
+    {
+      "status": "firing",
+      "labels": { "alertname": "HighCPU", "instance": "server1" },
+      "annotations": { "summary": "CPU above 80%" },
+      "startsAt": "2024-01-15T10:00:00Z",
+      "generatorURL": "http://grafana/alerting/..."
+    }
+  ],
+  "groupLabels": { "alertname": "HighCPU" },
+  "externalURL": "http://grafana"
+}
+```
+
+---
+
+## Notification Policies
+
+Notification policies route alerts to contact points based on label matchers.
+
+Navigate to: **Alerting > Notification policies**
+
+### Default policy
+The root policy - all alerts reach here if no specific policy matches.
+
+### Child policies
+Add policies that match specific labels:
+```
+Match labels:
+  severity = critical    -> contact: pagerduty
+  team = infrastructure  -> contact: slack-infra
+  environment = staging  -> contact: email-dev
+```
+
+### Policy settings
+
+| Setting | Description |
+|---------|-------------|
+| Contact point | Where to send matching alerts |
+| Continue matching | If true, also evaluate subsequent sibling policies |
+| Group by | Labels used for batching alerts into single notifications |
+| Group wait | Wait before sending first notification for a new group (default: 30s) |
+| Group interval | Wait before sending updates for an existing group (default: 5m) |
+| Repeat interval | Wait before re-sending for still-firing alerts (default: 4h) |
+
+### Grouping
+When multiple alerts have the same "group by" labels, they are batched into a single notification. This prevents notification storms.
+
+Example:
+- 50 hosts alert on HighCPU simultaneously
+- Group by: `[alertname, datacenter]`
+- Result: 1 notification per datacenter containing all affected hosts
+
+---
+
+## Notification Templates
+
+Customize notification message format using Go templating.
+
+Navigate to: **Alerting > Contact points > Notification templates**
+
+### Built-in variables
+```
+{{ $labels }}          # Alert labels as map
+{{ $values }}          # Query values map
+{{ $labels.instance }} # Specific label value
+{{ $values.A.Value }}  # Specific query value
+{{ $status }}          # firing or resolved
+{{ $startsAt }}        # When alert started firing
+```
+
+### Example Slack template
+```
+{{ define "slack_message" }}
+{{ if eq .Status "firing" }}:red_circle:{{ else }}:large_green_circle:{{ end }} *{{ .Labels.alertname }}*
+
+*Status:* {{ .Status }}
+*Severity:* {{ .Labels.severity }}
+
+{{ range .Alerts }}
+*Instance:* {{ .Labels.instance }}
+*Value:* {{ .Values.A.Value | humanize }}
+*Summary:* {{ .Annotations.summary }}
+{{ end }}
+{{ end }}
+```
+
+### Humanize functions
+```
+{{ $value | humanize }}           # "12.3k"
+{{ $value | humanize1024 }}       # "12.3Ki"
+{{ $value | humanizeBytes }}      # "12.3 kB"
+{{ $value | humanizeDuration }}   # "3h 2m 1s"
+{{ $value | humanizePercentage }} # "12.3%"
+```
+
+---
+
+## Silences
+
+Silences temporarily suppress alert notifications without stopping alert evaluation.
+
+Navigate to: **Alerting > Silences**
+
+### Create a silence
+1. Click **Add silence**
+2. Set start time and end time (or duration)
+3. Add label matchers:
+   ```
+   alertname = HighCPU
+   instance =~ ".*staging.*"   # Regex match
+   severity != critical         # Negative match
+   ```
+4. Add a comment explaining why
+5. Save
+
+### Use cases
+- Planned maintenance windows
+- Known issues being investigated
+- Silencing noisy alerts during deploys
+
+From a firing alert detail view: click **Silence** to pre-populate label matchers.
+
+---
+
+## Mute Timings
+
+Recurring schedules when notifications are suppressed (unlike silences which are one-time).
+
+Navigate to: **Alerting > Mute timings**
+
+### Example mute timings
+```
+Name: no-alerts-weekends
+  Weekdays: Saturday, Sunday
+
+Name: business-hours-only
+  Weekdays: Monday-Friday
+  Times: 09:00-17:00
+```
+
+Attach to notification policies in the policy settings.
+
+---
+
+## RBAC for Alerting
+
+Default permissions by role:
+
+| Action | Viewer | Editor | Admin |
+|--------|--------|--------|-------|
+| View alert rules | Yes | Yes | Yes |
+| Create/edit alert rules | No | Yes | Yes |
+| Delete alert rules | No | No | Yes |
+| Manage contact points | No | No | Yes |
+| Manage notification policies | No | No | Yes |
+| Create silences | No | Yes | Yes |
+
+---
+
+## grafana.ini Alerting Configuration
+
+```ini
+[unified_alerting]
+# Enable unified alerting (default: true since Grafana 9)
+enabled = true
+
+# Maximum alert instances a single rule can produce
+max_annotations_to_keep = 100
+
+# Evaluation timeout
+evaluation_timeout = 30s
+
+# Minimum evaluation interval (prevent too-frequent evaluation)
+min_interval = 10s
+
+[smtp]
+# Required for email contact points
+enabled = true
+host = smtp.example.com:587
+```
+
+---
+
+## Common Alert Rule Examples
+
+### CPU usage above threshold
+```promql
+# Query A
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+# Threshold: A > 80
+# Pending: 5m
+# Labels: severity=warning
+```
+
+### Memory usage
+```promql
+# Query A
+(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
+# Threshold: A > 90
+```
+
+### HTTP error rate
+```promql
+# Query A: error requests
+sum(rate(http_requests_total{status=~"5.."}[5m]))
+# Query B: total requests
+sum(rate(http_requests_total[5m]))
+# Expression C (Math): $A / $B * 100
+# Threshold: C > 5
+```
+
+### Disk space
+```promql
+# Query A
+(1 - (node_filesystem_free_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) * 100
+# Threshold: A > 85
+```
+
+### Service down (no data = firing)
+```promql
+# Query A
+up{job="my-service"}
+# Threshold: A < 1
+# No data handling: Alerting (treat absence as firing)
+```
+
+### Loki log error rate
+```logql
+# Query A
+sum(rate({job="api"} |= "ERROR" [5m]))
+# Threshold: A > 10
+```
+
+---
+
+## Connecting Alerts to Dashboards
+
+### Create alert from panel
+1. Open panel editor
+2. Click **Alert** tab
+3. Click **Create alert rule from this panel**
+4. Pre-populates query from the panel
+
+### Link alert to dashboard panel
+In the alert rule definition, set **Dashboard** and **Panel** to link to a specific visualization. The alert state badge appears on the panel and clicking it goes to the alert rule.
+
+---
+
+## High Availability (HA) Alerting
+
+```ini
+[unified_alerting]
+ha_peers = grafana-1:9094,grafana-2:9094,grafana-3:9094
+ha_advertise_address = ${POD_IP}:9094
+ha_peer_timeout = 15s
+ha_gossip_interval = 200ms
+ha_push_pull_interval = 60s
+```
+
+Multiple Grafana instances share state to avoid duplicate notifications.
--- a/.agents/skills/grafana-oss/references/api.md
+++ b/.agents/skills/grafana-oss/references/api.md
@ -0,0 +1,107 @@
+# Grafana OSS API
+
+Base URL: `https://your-grafana.example.com/api/`. Auth: service account token (`Authorization: Bearer <token>`).
+
+## Contents
+
+- [Dashboards](#dashboards)
+- [Data sources](#data-sources)
+- [Users + service accounts](#users--service-accounts)
+- [Annotations](#annotations)
+
+## Dashboards
+
+```bash
+# Search
+GET /api/search?query=service&type=dash-db&folderIds=1
+
+# Get by UID
+GET /api/dashboards/uid/{uid}
+
+# Create / update (overwrite: true replaces existing)
+POST /api/dashboards/db
+Body: { "dashboard": {...}, "folderUID": "...", "overwrite": true }
+
+# Delete
+DELETE /api/dashboards/uid/{uid}
+```
+
+After provisioning a dashboard via YAML, verify it landed:
+```bash
+curl https://grafana.example.com/api/dashboards/uid/<uid> \
+  -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+# Should print the dashboard's title. 404 = not provisioned correctly.
+```
+
+## Data sources
+
+```bash
+# List
+GET /api/datasources
+
+# Get by UID
+GET /api/datasources/uid/{uid}
+
+# Create
+POST /api/datasources
+Body: { "name": "...", "type": "...", "url": "...", "access": "proxy" }
+
+# Health-check (good post-provision validation)
+GET /api/datasources/uid/{uid}/health
+# Returns { "status": "OK" | "ERROR", "message": "..." }
+```
+
+## Users + service accounts
+
+```bash
+# List org users
+GET /api/org/users
+
+# List service accounts
+GET /api/serviceaccounts/search?perpage=100&page=1
+
+# Create service account
+POST /api/serviceaccounts
+Body: { "name": "ci-reader", "role": "Viewer", "isDisabled": false }
+
+# Mint a token
+POST /api/serviceaccounts/{id}/tokens
+Body: { "name": "ci-token", "secondsToLive": 0 }   # 0 = no expiry
+
+# Verify a token works
+curl https://grafana.example.com/api/org \
+  -H "Authorization: Bearer <new-token>"
+# 200 + org JSON = token + role assignment work.
+```
+
+## Annotations
+
+```bash
+# Create
+curl -X POST https://grafana.example.com/api/annotations \
+  -H 'Authorization: Bearer <token>' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "dashboardUID": "service-overview",
+    "panelId": 1,
+    "time": 1706745600000,
+    "timeEnd": 1706749200000,
+    "tags": ["deploy", "v2.0"],
+    "text": "Deployed v2.0"
+  }'
+
+# Find (by tag)
+GET /api/annotations?tags=deploy&from=1706745600000&to=1706832000000
+
+# Delete
+DELETE /api/annotations/{id}
+```
+
+## Common failure modes
+
+| Symptom | Likely cause |
+|---|---|
+| `401 Unauthorized` | Token expired or wrong stack URL — check the Authorization header |
+| `403 Forbidden` on dashboard create | Service account lacks Editor role on the target folder |
+| `412 Precondition Failed` on POST `/dashboards/db` | UID exists and you didn't set `overwrite: true` |
+| Data source health check `ERROR` | Network unreachable (check `url` field) or credentials wrong |
--- a/.agents/skills/grafana-oss/references/config.md
+++ b/.agents/skills/grafana-oss/references/config.md
@ -0,0 +1,100 @@
+# grafana.ini configuration
+
+Reference for `grafana.ini` server-side settings.
+
+## Contents
+
+- [Server + database](#server--database)
+- [Alerting](#alerting)
+- [SMTP](#smtp)
+- [Auth (OAuth example)](#auth-oauth-example)
+- [Security](#security)
+- [Feature toggles](#feature-toggles)
+
+## Server + database
+
+```ini
+[server]
+http_port = 3000
+domain = grafana.example.com
+root_url = https://grafana.example.com/
+
+[database]
+type = postgres
+host = postgres:5432
+name = grafana
+user = grafana
+password = secret
+```
+
+After changing these, restart Grafana and verify:
+- `curl http://localhost:3000/api/health` returns `{"database": "ok", ...}`
+- `curl http://localhost:3000/api/admin/settings | jq '.database'` shows the new config
+
+## Alerting
+
+```ini
+[alerting]
+enabled = true
+
+[unified_alerting]
+enabled = true
+```
+
+## SMTP
+
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@example.com
+password = yourpassword
+from_address = alerts@example.com
+```
+
+Verify by sending a test email via the Alerting → Contact points UI.
+
+## Auth (OAuth example)
+
+```ini
+[auth.generic_oauth]
+enabled = true
+name = Okta
+client_id = your_client_id
+client_secret = your_secret
+auth_url = https://your-org.okta.com/oauth2/v1/authorize
+token_url = https://your-org.okta.com/oauth2/v1/token
+api_url = https://your-org.okta.com/oauth2/v1/userinfo
+scopes = openid profile email groups
+```
+
+For SAML and GitHub OAuth, see the [grafana-cloud/admin skill](../../../grafana-cloud/admin/references/sso.md). The configs are the same in OSS.
+
+## Security
+
+```ini
+[security]
+admin_user = admin
+admin_password = secret
+allow_embedding = true       # required for embedding dashboards in iframes
+```
+
+`admin_password` is only consulted on first startup. To change later, use `grafana-cli admin reset-admin-password <new-password>`.
+
+## Feature toggles
+
+```ini
+[feature_toggles]
+enable = publicDashboards
+```
+
+Multiple toggles are space-separated:
+```ini
+enable = publicDashboards correlations grafanaApiServer
+```
+
+## Common problems
+
+- **`grafana.ini` changes need a restart** — config is read at startup, not live-reloaded
+- **`root_url` matters for OAuth callbacks** — if you set `domain` but not `root_url`, OAuth redirect URIs may not match
+- **Sections are global** — `[server]`, `[database]`, etc. apply at the process level, not per-org
--- a/.agents/skills/grafana-oss/references/dashboard-json.md
+++ b/.agents/skills/grafana-oss/references/dashboard-json.md
@ -0,0 +1,81 @@
+# Dashboard JSON model + template variables
+
+## Minimal dashboard JSON
+
+```json
+{
+  "title": "Service Overview",
+  "uid": "service-overview",
+  "time": { "from": "now-1h", "to": "now" },
+  "refresh": "30s",
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus" },
+          "expr": "rate(http_requests_total{job=\"$job\"}[5m])",
+          "legendFormat": "{{method}} {{status}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 1000 }
+            ]
+          }
+        }
+      }
+    }
+  ]
+}
+```
+
+## Template variables
+
+```json
+{
+  "templating": {
+    "list": [
+      {
+        "name": "namespace",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prom" },
+        "definition": "label_values(kube_pod_info, namespace)",
+        "includeAll": true,
+        "multi": true
+      },
+      {
+        "name": "env",
+        "type": "custom",
+        "query": "production,staging,dev",
+        "current": { "value": "production" }
+      },
+      {
+        "name": "interval",
+        "type": "interval",
+        "query": "1m,5m,15m,1h",
+        "auto": true
+      }
+    ]
+  }
+}
+```
+
+Reference variables in queries with `$variable`:
+```promql
+rate(http_requests_total{namespace="$namespace"}[$interval])
+```
+
+## Common problems
+
+- **`uid` must be unique across the org** — if you POST a dashboard with an existing UID and `overwrite: false`, Grafana returns 412 Precondition Failed
+- **`gridPos`** uses a 24-column grid; `w + x` must be ≤ 24
+- **`datasource.uid` in `targets`** must match an existing data source UID; misspell it and panels render as "Datasource not found"
+- **Template variable `query` field** is data-source-specific syntax (PromQL `label_values(...)`, LogQL `{label="…"}`, SQL, etc.)
--- a/.agents/skills/grafana-oss/references/dashboards.md
+++ b/.agents/skills/grafana-oss/references/dashboards.md
@ -0,0 +1,215 @@
+# Grafana Dashboards - Detailed Reference
+
+## What is a Dashboard?
+
+A Grafana dashboard is a set of one or more panels organized into rows, providing an at-a-glance view of related information. Dashboards connect to data sources, display data through panels, and support interactive filtering via variables.
+
+## Core Components
+
+- **Panels**: Containers that display visualizations; each combines a query + a visualization type
+- **Rows**: Horizontal groupings of panels; can be collapsed for organization
+- **Data Sources**: Connections to databases, APIs, or services that panels query
+- **Variables**: Dropdown selectors at the top of a dashboard for dynamic filtering
+- **Annotations**: Markers overlaid on graphs to indicate events
+
+---
+
+## Creating Dashboards
+
+### From scratch
+1. Click **Dashboards** in the left sidebar
+2. Click **New > New dashboard**
+3. Click **Add visualization** to add your first panel
+4. Select a data source, write a query, choose visualization type
+5. Click **Apply** to save the panel to the dashboard
+6. Click the save icon (or Ctrl+S) to save the dashboard
+
+### From a template / Import
+- Click **Dashboards > New > Import**
+- Paste a dashboard JSON, enter a Grafana.com dashboard ID, or upload a JSON file
+- Grafana.com hosts thousands of community dashboards (grafana.com/grafana/dashboards)
+
+---
+
+## Dashboard Settings
+
+Access via the gear icon at the top-right of any dashboard.
+
+| Setting | Description |
+|---------|-------------|
+| General | Name, description, tags, folder, editable flag |
+| Annotations | Configure annotation queries that overlay events on panels |
+| Variables | Add/edit template variables (dropdowns at top) |
+| Links | Add links to other dashboards or external URLs |
+| JSON Model | View/edit raw JSON for the entire dashboard |
+| Versions | Browse and restore prior versions |
+| Time options | Default time range, auto-refresh intervals, timezone |
+
+---
+
+## Time Range Controls
+
+- **Time picker** (top-right): Select absolute or relative ranges (Last 6 hours, Last 7 days, etc.)
+- **Auto-refresh**: Set to Off, 5s, 10s, 30s, 1m, 5m, etc.
+- **Zoom**: Click-drag on any time series to zoom in
+
+### Common relative time shortcuts
+```
+now-5m    last 5 minutes
+now-1h    last 1 hour
+now-24h   last 24 hours
+now-7d    last 7 days
+now/d     today so far
+now-1d/d  yesterday
+```
+
+---
+
+## Annotations
+
+Annotations are event markers overlaid on time series panels.
+
+### Types
+- **Native annotations**: Manually add a note to a specific time directly in the dashboard
+- **Query annotations**: Pull events from a data source and display as markers
+
+### Adding a manual annotation
+- Hold Ctrl (or Cmd on Mac) and click a time series panel
+- Type a description; optionally set a time range
+
+---
+
+## Library Panels
+
+Library panels are reusable panel definitions shared across multiple dashboards.
+
+- **Create**: Panel menu (3-dot) > "Create library panel"
+- **Use**: Add a panel > "Add from panel library"
+- **Update**: Edit the source panel; all dashboards using it reflect the change
+- **Unlink**: Detach to make it independent in a specific dashboard
+
+---
+
+## Panel Layout
+
+- Drag panel corners to resize
+- Drag panel header to move
+- **Add > Row** to insert collapsible rows
+- Panel menu > **Duplicate** to clone within the same dashboard
+
+---
+
+## Dashboard Versions
+
+- Access via Dashboard Settings > Versions
+- See who saved what and when
+- Compare two versions (diff view)
+- Restore any previous version
+
+---
+
+## JSON Model
+
+Every dashboard is stored as a JSON document.
+
+Key top-level JSON fields:
+```json
+{
+  "title": "My Dashboard",
+  "uid": "abc123",
+  "tags": ["production", "infrastructure"],
+  "time": { "from": "now-6h", "to": "now" },
+  "refresh": "30s",
+  "panels": [],
+  "templating": { "list": [] },
+  "annotations": { "list": [] }
+}
+```
+
+---
+
+## Sharing Dashboards
+
+### Share link
+- Click **Share** icon > **Link** tab
+- Toggle "Lock time range" to embed the current time window
+- Toggle "Include template variable values"
+
+### Snapshot
+- **Share > Snapshot**: Creates a read-only, public snapshot
+- Contains rendered data at share time - no live data source access needed
+- Can expire (1 hour, 1 day, 7 days, or never)
+
+### Export / Import JSON
+- **Share > Export**: Download the dashboard JSON
+- Import at **Dashboards > New > Import**
+- Useful for version control, migration, sharing with the community
+
+### Embed
+- **Share > Embed**: Generates an iframe HTML snippet
+- Requires anonymous access in Grafana config (OSS/Enterprise only)
+
+### Public dashboards (Grafana 10+)
+- Make a dashboard publicly accessible with no login
+- Enable per-dashboard in Share > Public dashboard
+
+---
+
+## Playlists
+
+Playlists cycle through dashboards automatically at a configurable interval.
+
+- Create at **Dashboards > Playlists > New playlist**
+- Add dashboards by name or tag
+- Set interval (e.g., 5 minutes)
+- Append `?kiosk=1` to URL for kiosk/TV mode
+
+---
+
+## Best Practices
+
+1. **Organize with folders**: Use folders as RBAC permission boundaries
+2. **Use variables**: Make dashboards reusable across environments/services
+3. **Limit panels per dashboard**: Aim for <20 panels for performance
+4. **Use library panels**: Standardize common panels across teams
+5. **Tag dashboards**: Consistent tags for searchability
+6. **Version control**: Export JSON and store in Git
+7. **Use rows**: Collapse related panels to organize complex dashboards
+8. **Template your queries**: Use variables so one dashboard covers many targets
+
+---
+
+## Keyboard Shortcuts
+
+| Key | Action |
+|-----|--------|
+| `Ctrl+S` | Save dashboard |
+| `e` | Open panel editor (when panel focused) |
+| `v` | Toggle panel fullscreen |
+| `d r` | Refresh all panels |
+| `d s` | Dashboard settings |
+| `d k` | Toggle kiosk mode |
+| `?` | Show all shortcuts |
+
+---
+
+## Provisioning Dashboards (as Code)
+
+```yaml
+# /etc/grafana/provisioning/dashboards/default.yaml
+apiVersion: 1
+
+providers:
+  - name: default
+    orgId: 1
+    folder: Infrastructure
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: false
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+Place dashboard JSON files in `/var/lib/grafana/dashboards/`. Subdirectories become folders when `foldersFromFilesStructure: true`.
--- a/.agents/skills/grafana-oss/references/datasources.md
+++ b/.agents/skills/grafana-oss/references/datasources.md
@ -0,0 +1,462 @@
+# Grafana Data Sources - Detailed Reference
+
+## Overview
+
+Data sources are connections between Grafana and the systems storing your data. Grafana ships with built-in support for many popular data sources and supports additional sources via plugins.
+
+- Only **Organization Admins** can add or remove data sources
+- Each data source has its own query editor
+- Data sources can be used in: Dashboard panels, Explore, Alert rules, Annotations
+
+---
+
+## Managing Data Sources
+
+### Add a data source
+1. Go to **Connections > Data sources** (or **Configuration > Data sources** in older versions)
+2. Click **Add new data source**
+3. Search for and select the data source type
+4. Fill in connection details (URL, credentials, etc.)
+5. Click **Save & Test** to verify the connection
+
+### Data source settings (common to all types)
+| Setting | Description |
+|---------|-------------|
+| Name | Display name used in dashboards |
+| Default | If checked, pre-selected in new panels |
+| HTTP URL | The endpoint for the data source server |
+| Auth | Basic auth, TLS, bearer token, API key options |
+
+---
+
+## Prometheus
+
+Native support - no plugin required.
+
+### Configuration
+```ini
+URL: http://prometheus:9090
+HTTP method: POST (recommended, supports longer queries)
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Scrape interval | Default: 15s - should match your Prometheus scrape interval |
+| Query timeout | Default: 60s |
+| Exemplars | Enable to link metrics to traces |
+| Ruler URL | For Prometheus-managed alert rules |
+
+### Query editor
+- **Metrics browser**: Browse available metrics with autocomplete
+- **Query type**: Range query (time series) or Instant query (single value)
+- **Legend**: Customize series labels using `{{label_name}}` syntax
+
+### Template variables with Prometheus
+```
+# Variable type: Query
+# Query examples:
+label_values(metric_name, label_name)    # All values of a label for a metric
+label_values(label_name)                  # All values across all metrics
+metrics(prefix)                           # All metric names matching prefix
+query_result(promql_expression)           # PromQL result as variable values
+```
+
+### Exemplars
+When enabled, Prometheus exemplars link high-cardinality trace IDs to metric data points. Requires a Tempo data source for trace drill-through.
+
+---
+
+## Loki (Log Aggregation)
+
+### Configuration
+```ini
+URL: http://loki:3100
+Maximum lines: 1000
+```
+
+### Derived fields
+Extract values from log lines and link to other systems.
+
+Example - extract trace ID and link to Tempo:
+```
+Name: TraceID
+Regex: traceID=(\w+)
+URL: (internal link to Tempo data source)
+```
+
+### Query editor (LogQL)
+```logql
+# Basic log stream selector
+{job="nginx", namespace="production"}
+
+# Filter by text
+{job="nginx"} |= "error"
+{job="nginx"} != "debug"
+
+# Regex filter
+{job="nginx"} |~ "status=5\d\d"
+
+# Parse and filter structured logs
+{job="api"} | json | level="error"
+{job="api"} | logfmt | duration > 1s
+
+# Metrics from logs
+rate({job="nginx"} |= "error" [5m])
+sum(rate({job="nginx"}[5m])) by (status_code)
+```
+
+### Template variables with Loki
+```
+label_names()                          # All label names
+label_values(label_name)               # All values for a label
+label_values({job="nginx"}, pod)       # Label values filtered by stream selector
+```
+
+---
+
+## Tempo (Distributed Tracing)
+
+### Configuration
+```ini
+URL: http://tempo:3200
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Trace to logs | Link traces to Loki logs via trace ID |
+| Trace to metrics | Link trace spans to Prometheus metrics |
+| Service graph | Enable service dependency graph |
+| Node graph | Show trace as node graph |
+
+### TraceQL (query language)
+```traceql
+# Find traces with error spans
+{status=error}
+
+# Filter by service and duration
+{.service.name="frontend" && duration > 1s}
+
+# Structural queries
+{.http.url=~"/api/.*"} >> {status=error}
+```
+
+---
+
+## Alertmanager
+
+For connecting to an external Alertmanager.
+
+```ini
+URL: http://alertmanager:9093
+# Implementation: Prometheus, Mimir, or Cortex
+```
+
+---
+
+## Elasticsearch / OpenSearch
+
+### Configuration
+```ini
+URL: http://elasticsearch:9200
+Index name: logs-*
+Time field name: @timestamp
+Elasticsearch version: 8.x
+```
+
+---
+
+## MySQL
+
+Built-in support for MySQL 5.7+ and compatible databases (MariaDB, Percona, Amazon Aurora MySQL, Azure Database for MySQL, Google Cloud SQL MySQL).
+
+### Configuration
+```ini
+Host: mysql:3306
+Database: mydb
+User: grafana
+Password: secret
+Max open connections: 100
+Max idle connections: 100
+Connection max lifetime: 14400
+```
+
+### Time series queries
+```sql
+SELECT
+  UNIX_TIMESTAMP(time_col) as time_sec,
+  value_col as value,
+  name_col as metric
+FROM my_table
+WHERE $__timeFilter(time_col)
+ORDER BY time_col ASC
+```
+
+### Macros
+| Macro | Description |
+|-------|-------------|
+| `$__time(column)` | Converts column to Unix timestamp |
+| `$__timeFilter(column)` | Adds WHERE clause for dashboard time range |
+| `$__timeFrom()` | Start of dashboard time range as Unix timestamp |
+| `$__timeTo()` | End of dashboard time range as Unix timestamp |
+| `$__timeGroup(column, interval)` | Groups by time interval |
+| `$__timeGroupAlias(column, interval)` | As above, aliases as "time" |
+| `$__unixEpochFilter(column)` | Time filter for Unix epoch columns |
+| `$__interval` | Auto-calculated interval for current time range |
+
+### Example time series query
+```sql
+SELECT
+  $__timeGroup(created_at, $__interval) AS time,
+  status,
+  count(*) AS cnt
+FROM orders
+WHERE $__timeFilter(created_at)
+GROUP BY 1, 2
+ORDER BY 1
+```
+
+### Annotations
+```sql
+SELECT
+  UNIX_TIMESTAMP(time) AS time,
+  title AS text,
+  tags
+FROM events
+WHERE $__timeFilter(time)
+```
+
+---
+
+## PostgreSQL
+
+### Configuration
+```ini
+Host: postgres:5432
+Database: mydb
+User: grafana
+Password: secret
+SSL mode: disable / require / verify-ca / verify-full
+```
+
+### PostgreSQL time series
+```sql
+SELECT
+  time_bucket('$__interval', time) AS time,
+  avg(value)
+FROM metrics
+WHERE time BETWEEN $__timeFrom() AND $__timeTo()
+GROUP BY 1
+ORDER BY 1
+```
+
+---
+
+## Microsoft SQL Server
+
+### Configuration
+```ini
+Host: sqlserver:1433
+Database: mydb
+User: grafana
+Password: secret
+Encrypt: false / true / disable
+```
+
+---
+
+## InfluxDB
+
+Supports InfluxDB 1.x (InfluxQL) and InfluxDB 2.x / 3.x (Flux).
+
+### InfluxDB 1.x (InfluxQL)
+```ini
+URL: http://influxdb:8086
+Database: telegraf
+```
+
+Query example:
+```sql
+SELECT mean("value") FROM "measurement"
+WHERE $timeFilter
+GROUP BY time($interval), "host"
+```
+
+### InfluxDB 2.x (Flux)
+```ini
+URL: http://influxdb:8086
+Organization: myorg
+Token: <api-token>
+Default bucket: mydata
+```
+
+Flux query example:
+```flux
+from(bucket: "mydata")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r._measurement == "cpu")
+  |> filter(fn: (r) => r._field == "usage_idle")
+  |> aggregateWindow(every: v.windowPeriod, fn: mean)
+```
+
+---
+
+## AWS CloudWatch
+
+### Configuration
+```ini
+# Auth options:
+# - AWS SDK Default (instance role, ~/.aws/credentials, env vars)
+# - Access & secret key (explicit credentials)
+# - Assume role ARN
+
+Default region: us-east-1
+```
+
+Required IAM permissions:
+- `cloudwatch:GetMetricData`
+- `cloudwatch:ListMetrics`
+- `logs:*` (for CloudWatch Logs)
+
+### CloudWatch Logs Insights
+```
+fields @timestamp, @message
+| filter level = "ERROR"
+| sort @timestamp desc
+| limit 100
+```
+
+---
+
+## Azure Monitor
+
+Requires Azure App Registration with:
+- Tenant ID, Client ID, Client Secret
+- `Monitoring Reader` role on target subscriptions/resource groups
+
+---
+
+## Google Cloud Monitoring
+
+Uses Google Cloud service account credentials (JSON key file or workload identity).
+
+---
+
+## TestData (built-in)
+
+A built-in data source for generating test/demo data without a real backend.
+
+Use cases: Testing visualizations, demo dashboards, development without a real data source.
+Scenarios: Random walk, CSV metric values, streaming data, etc.
+
+---
+
+## Graphite
+
+```ini
+URL: http://graphite:8080
+```
+
+Query examples:
+```
+target(servers.*.cpu)
+averageSeries(servers.*.cpu)
+groupByNode(servers.*.cpu, 1, 'averageSeries')
+```
+
+---
+
+## Jaeger (Tracing)
+
+```ini
+URL: http://jaeger:16686
+```
+
+---
+
+## Zipkin (Tracing)
+
+```ini
+URL: http://zipkin:9411
+```
+
+---
+
+## Pyroscope (Continuous Profiling)
+
+```ini
+URL: http://pyroscope:4040
+```
+
+---
+
+## Data Source Permissions (Enterprise)
+
+By default, all org users can query any data source. With RBAC:
+- Assign specific users/teams to specific data sources
+- Configure at **Data source settings > Permissions**
+
+---
+
+## Provisioning Data Sources (as Code)
+
+```yaml
+# /etc/grafana/provisioning/datasources/prometheus.yaml
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    version: 1
+    editable: false
+    jsonData:
+      timeInterval: "15s"
+      queryTimeout: "60s"
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    jsonData:
+      maxLines: 1000
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: "traceID=(\\w+)"
+          name: TraceID
+          url: "${__value.raw}"
+
+  - name: MySQL Production
+    type: mysql
+    url: mysql:3306
+    database: mydb
+    user: grafana
+    secureJsonData:
+      password: "$GRAFANA_MYSQL_PASSWORD"
+    jsonData:
+      maxOpenConns: 100
+      maxIdleConns: 100
+      connMaxLifetime: 14400
+```
+
+Place YAML files in the provisioning directory; Grafana loads on startup and watches for changes.
+
+---
+
+## Plugin Data Sources
+
+Install additional data sources via:
+- **Connections > Add new connection** (search the plugin catalog)
+- `grafana-cli plugins install <plugin-id>`
+- Docker: set `GF_INSTALL_PLUGINS` environment variable
+
+Popular plugin data sources:
+- `grafana-opensearch-datasource` - OpenSearch
+- `grafana-bigquery-datasource` - Google BigQuery
+- `grafana-mongodb-datasource` - MongoDB (Enterprise)
+- `grafana-splunk-datasource` - Splunk
+- `grafana-datadog-datasource` - Datadog
--- a/.agents/skills/grafana-oss/references/panel-types.md
+++ b/.agents/skills/grafana-oss/references/panel-types.md
@ -0,0 +1,29 @@
+# Panel types
+
+| Panel | Use Case |
+|-------|----------|
+| **Time series** | Line/bar charts over time (default for metrics) |
+| **Stat** | Single value with color thresholds |
+| **Gauge** | Radial gauge for current value |
+| **Bar gauge** | Horizontal bars for comparisons |
+| **Table** | Tabular data, sortable columns |
+| **Logs** | Log stream viewer (Loki) |
+| **Traces** | Trace visualization (Tempo) |
+| **Heatmap** | Distribution over time |
+| **Histogram** | Value distribution |
+| **Pie chart** | Part-to-whole ratios |
+| **Geomap** | Geographic data |
+| **Canvas** | Custom SVG-based layouts |
+| **Node graph** | Service/topology graphs |
+| **Flame graph** | CPU/memory profiling |
+| **Text** | Markdown/HTML content |
+| **Alert list** | Show firing alerts |
+
+## Picking a panel type
+
+Default is **Time series** for almost any metric-over-time visualization. Switch from the default only when:
+
+- The user wants **one number, not a series** → Stat / Gauge / Bar gauge
+- The data is **categorical, not temporal** → Pie chart / Bar gauge / Table
+- The data is **logs/traces/profiles** → Logs / Traces / Flame graph (per data type)
+- The data is **2D distribution** (e.g. histogram-over-time) → Heatmap
--- a/.agents/skills/grafana-oss/references/panels.md
+++ b/.agents/skills/grafana-oss/references/panels.md
@ -0,0 +1,325 @@
+# Grafana Panels and Visualizations - Detailed Reference
+
+## What is a Panel?
+
+A panel is the basic building block of a Grafana dashboard. Each panel combines:
+- A **query** (or multiple queries) to a data source
+- A **visualization type** to display the data
+- **Field configuration** (units, thresholds, color mappings)
+- Optional **transformations** to reshape the data before display
+
+---
+
+## Panel Editor
+
+Open the panel editor by clicking a panel's title > Edit, or clicking **Add visualization** for a new panel.
+
+### Panel Editor Layout
+
+**Top bar**: Back to dashboard, Discard changes, Save dashboard
+
+**Center**: Visualization preview (live update as you configure)
+
+**Toggle**: "Table view" - shows raw query results as a table for debugging
+
+**Right sidebar tabs**:
+1. **Query** - configure data sources and write queries
+2. **Transform** - apply data transformations
+3. **Alert** - create alert rules from this panel
+4. Below tabs: visualization-specific options and field config
+
+### Query Tab
+
+- **Data source selector**: Choose which data source to query
+- **Query editor**: Data-source-specific interface (PromQL for Prometheus, LogQL for Loki, SQL for databases)
+- **Query options**:
+  - Max data points: Limit data points fetched
+  - Min interval: Minimum auto-calculated interval
+  - Interval: Override the auto-calculated interval
+  - Relative time: Override dashboard time range for this panel only
+  - Time shift: Shift the time range (e.g., to compare to last week)
+- **+ Query**: Add multiple queries (labeled A, B, C...)
+- **Expression**: Add server-side math expressions combining query results
+
+---
+
+## Visualization Types
+
+### Time Series (default)
+Best for: Metrics over time, continuous data
+- Renders as lines, points, or bars
+- Supports multiple series
+- Configurable line width, fill, point size
+- Supports thresholds as colored background regions
+- Supports annotations overlay
+- **Graph styles**: Lines, Bars, Points; can mix per series
+- **Stacking**: None, Normal, 100%
+- **Axis**: Left Y, Right Y, hidden
+- Default visualization - supports alerting
+
+### Stat
+Best for: Single important metric (KPI display)
+- Shows one or more values as large text
+- Supports sparkline background
+- Color modes: value, background, none
+- Can show last value, mean, sum, etc.
+- Great for dashboards that need quick status overviews
+
+### Bar Chart
+Best for: Comparing categorical data
+- Horizontal or vertical orientation
+- Grouped or stacked
+- Supports labels on bars
+
+### Gauge
+Best for: Showing a value relative to min/max range
+- Circular gauge with arc
+- Configurable thresholds set colors
+- Shows current value prominently
+
+### Bar Gauge
+Best for: Multiple metrics as horizontal/vertical bars
+- Useful for comparing many items
+- Supports thresholds for color coding
+- Modes: gradient, retro LCD, basic
+
+### Table
+Best for: Tabular data, multi-column metrics
+- Supports sorting by column
+- Column width customization
+- Cell display modes: color text, color background, gradient gauge
+- Pagination for large datasets
+- Can embed sparklines in cells
+
+### Heatmap
+Best for: Distribution over time, histogram-over-time
+- X axis: time; Y axis: buckets; Color: density/value
+- Supports pre-bucketed data (Prometheus histogram) and raw values
+- Tooltip shows exact bucket counts
+
+### Histogram
+Best for: Value distribution analysis
+- Groups values into buckets
+- Can combine multiple series
+- Configurable bucket size
+
+### Pie Chart
+Best for: Proportional data, parts-of-whole
+- Pie or donut style
+- Labels: name, value, percentage
+
+### Logs
+Best for: Log data from Loki, Elasticsearch, etc.
+- Displays raw log lines with timestamp
+- Log level coloring (info/warn/error)
+- Search/filter within results
+- Deduplication and time wrapping
+- Prettify JSON option
+
+### Traces
+Best for: Distributed tracing visualization (Tempo)
+- Renders trace spans as a waterfall/Gantt chart
+- Shows service name, operation, duration
+- Click to drill into trace details
+
+### Flame Graph
+Best for: CPU profiling data (Pyroscope)
+- Visualizes call stacks by CPU time
+- Click to zoom into subtrees
+
+### Node Graph
+Best for: Service dependency maps, network topology
+- Renders nodes and edges
+- Node color/size configurable by metric
+
+### Geomap
+Best for: Geographic data visualization
+- Layers: markers, heatmap, route
+- Multiple base map options (OpenStreetMap, CARTO, etc.)
+- Supports GeoJSON data
+
+### Canvas
+Best for: Custom layouts, process diagrams, status boards
+- Drag-and-drop element placement
+- Elements: text, metric value, rectangle, ellipse, icon, image, connections
+- Dynamic data binding per element
+
+### State Timeline
+Best for: State changes over time (on/off, OK/warn/crit)
+- Horizontal bands showing state duration
+- Each series = one row
+- Color per state value
+
+### Status History
+Best for: Periodic state checks over time
+- Grid: Y=services, X=time buckets
+- Color per state
+
+### XY Chart
+Best for: Correlation between two metrics
+- Scatter plot
+- X and Y axis from different fields
+- Bubble size from a third field
+
+### Candlestick
+Best for: Financial OHLC data
+- Open/High/Low/Close representation
+- Volume bars
+
+### Text
+Best for: Documentation panels, headers
+- Renders Markdown or HTML
+
+### Alert List
+Best for: Dashboard overview of current alert states
+
+### Dashboard List
+Best for: Navigation panels linking to other dashboards
+
+---
+
+## Field Configuration (Standard Options)
+
+Available for most visualizations under the visualization options panel.
+
+| Option | Description |
+|--------|-------------|
+| Unit | Display unit (bytes, seconds, %, requests/sec, etc.) |
+| Min / Max | Override auto-detected min/max for scales |
+| Decimals | Number of decimal places |
+| Display name | Override the series/field name |
+| Color scheme | Fixed, thresholds-based, palette, etc. |
+| No value | Text to display when value is null |
+
+### Thresholds
+Define color-coded boundaries:
+- **Absolute**: Fixed numeric values (e.g., >90 = red, >70 = yellow, else green)
+- **Percentage**: Relative to min/max
+
+Example threshold config:
+```
+Base (default): Green
+70: Yellow (warn)
+90: Red (crit)
+```
+
+### Value Mappings
+Transform raw values into human-readable labels or colors:
+- **Value to text**: e.g., `1` -> "OK", `0` -> "Down"
+- **Range to text**: e.g., `0-50` -> "Low"
+- **Regex to text**: Match patterns
+
+### Data Links
+Create clickable links from panel values:
+- Link to other dashboards with variable values interpolated
+- Link to external systems (e.g., Kibana, PagerDuty)
+- Use `${__value.raw}` and `${__field.name}` in URLs
+
+---
+
+## Field Overrides
+
+Apply specific field options to individual series/columns rather than all data:
+
+1. Click **+ Add field override** in the Overrides section
+2. Choose override target:
+   - Fields with name (exact match)
+   - Fields with name matching regex
+   - Fields with type (number, string, time)
+   - Fields returned by query (A, B, C...)
+3. Add properties to override (unit, color, alias, thresholds, etc.)
+
+Example: In a table with columns `cpu_idle` and `cpu_used`, set `cpu_used` to show as percentage and color by threshold while leaving `cpu_idle` with default styling.
+
+---
+
+## Transformations
+
+Transformations reshape query data before rendering. Apply multiple in sequence.
+
+Access: Panel editor > Transform tab > Add transformation
+
+### Most-Used Transformations
+
+| Transformation | Description |
+|----------------|-------------|
+| **Reduce** | Collapse a time series to a single value (last, mean, sum, max, min) |
+| **Filter by name** | Keep only specific fields/columns |
+| **Filter by value** | Filter rows where a field matches a condition |
+| **Organize fields** | Rename, reorder, or hide fields |
+| **Merge** | Combine multiple query results into one table |
+| **Join by field** | SQL-style join on a common field (e.g., time) |
+| **Group by** | Group rows and aggregate (count, sum, mean, etc.) |
+| **Sort by** | Sort rows by a field |
+| **Limit** | Keep only first N rows |
+| **Add field from calculation** | Add a new column computed from existing columns |
+| **Convert field type** | Change a field's data type |
+| **Rename by regex** | Batch rename fields using regex |
+| **Extract fields** | Parse JSON or regex from a string field |
+| **Labels to fields** | Convert label key/values into separate columns |
+| **Rows to fields** | Pivot: turn row values into column headers |
+| **Prepare time series** | Normalize time series format |
+| **Time series to table** | Convert time series format to table format |
+
+Enable "Debug" toggle on any transformation to see input/output for troubleshooting.
+
+---
+
+## Query Options
+
+### Multiple queries
+Add multiple queries (A, B, C...) to one panel. They are overlaid in the visualization. Use this to compare metrics or show multiple services on one graph.
+
+### Expressions (server-side)
+Server-side expressions allow math across query results:
+- **Math**: `$A + $B`, `$A / $B * 100`
+- **Reduce**: Collapse a series to scalar
+- **Resample**: Change time resolution of a series
+- **Classic conditions**: Threshold logic (used in alerting)
+
+Example - calculate error rate as percentage:
+```
+Query A: total_requests{job="api"}
+Query B: error_requests{job="api"}
+Expression C (Math): $B / $A * 100
+Display C as percentage
+```
+
+### Important query variables
+These variables are available in queries:
+
+| Variable | Description |
+|----------|-------------|
+| `$__interval` | Auto-calculated interval based on time range and resolution |
+| `$__rate_interval` | Interval suitable for rate() functions (at least 4x scrape interval) |
+| `$__from` | Start of the current time range (ms epoch) |
+| `$__to` | End of the current time range (ms epoch) |
+| `$__range` | Duration of current time range (e.g., "6h") |
+| `$__range_s` | Duration in seconds |
+| `$__range_ms` | Duration in milliseconds |
+
+Example PromQL using interval variable:
+```promql
+rate(http_requests_total[${__rate_interval}])
+```
+
+---
+
+## Panel Inspect
+
+Click panel menu (3-dot) > **Inspect** to access:
+- **Data**: Raw table view of the data powering the panel
+- **Stats**: Query performance (time, row count)
+- **JSON**: Panel JSON model
+- **Query**: Equivalent of query inspector showing raw query and response
+
+---
+
+## Performance Tips
+
+1. Limit max data points to avoid over-fetching
+2. Use recording rules in Prometheus for expensive queries
+3. Set longer min intervals for historical dashboards
+4. Use `$__interval` variable in queries to align with time resolution
+5. Use `$__rate_interval` instead of hardcoded intervals for rate() queries
+6. Avoid too many panels on one dashboard (aim for <20)
--- a/.agents/skills/promql/SKILL.md
+++ b/.agents/skills/promql/SKILL.md
@ -0,0 +1,103 @@
+---
+name: promql
+license: Apache-2.0
+description: Write, validate, and optimize PromQL for Prometheus / Grafana Mimir / Grafana Cloud Metrics. Covers `rate` vs `irate` vs `increase`, label matchers and regex, `sum / avg / topk / by / without` aggregation, classic + native `histogram_quantile`, ratios with divide-by-zero guards, `absent` / `changes` for staleness, time offsets and `predict_linear`, recording-rule naming, SLO + burn-rate math, and a cardinality-hunting playbook. Use when writing a metric query, fixing wrong p95s, building an error-budget alert, debugging "query is slow", finding the noisy label that blew up cardinality, or migrating a dashboard query to a recording rule — even when the user says "calculate the error rate", "p99 latency", "sum by service", "why is this query slow", or "what's filling Mimir" without naming PromQL.
+---
+
+# PromQL Query Patterns
+
+> **Docs**: https://prometheus.io/docs/prometheus/latest/querying/basics/
+
+PromQL returns either an **instant vector**, a **range vector**, or a **scalar**.
+
+**Golden rule:** `rate()` / `increase()` require a range vector ≥ 4× the scrape interval. 60s scrape → use `[5m]` minimum.
+
+## Prerequisites
+
+- A Prometheus / Mimir / Grafana Cloud endpoint to query (`/api/v1/query` or via Grafana Explore)
+- The PromQL pattern library in [`references/patterns.md`](references/patterns.md)
+
+## Common Workflows
+
+### 1. Write + validate a query
+
+```bash
+# 0. Point at your Prometheus/Mimir. For Grafana Cloud, use the metrics endpoint
+#    and add basic auth (-u "<metrics_user>:<token>") to each curl below.
+PROM=http://localhost:9090   # or https://prometheus-prod-XX.grafana.net/api/prom
+
+# 1. Sketch the query — for "5xx error rate per service":
+EXPR='sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)'
+
+# 2. Validate syntax + that the metric/labels exist
+curl -sG --data-urlencode "query=${EXPR}" \
+  "$PROM/api/v1/query" | jq '.status, (.data.result|length)'
+# Expect: "success" and result count > 0. If 0 — check label spelling and scrape activity:
+curl -sG --data-urlencode "match[]=http_requests_total" "$PROM/api/v1/series" | jq '.data | length'
+
+# 3. Sanity-check the magnitude — open Grafana Explore, paste the expr,
+#    confirm the values look right against a known ground truth (k6 run, log count, etc.)
+```
+
+### 2. Common patterns to copy
+
+**Per-status request rate** (aggregate AFTER rate):
+
+```promql
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+```
+
+**p95 latency** (must keep `le` in the inner aggregation):
+
+```promql
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+```
+
+**Error rate with divide-by-zero guard:**
+
+```promql
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / (sum(rate(http_requests_total[5m])) > 0)
+```
+
+Full library (recording rules, SLO burn-rate, offsets, cardinality hunt, native histograms): [`references/patterns.md`](references/patterns.md).
+
+### 3. Convert a slow dashboard query into a recording rule
+
+```yaml
+# 1. Pick the slow expression, give it a recording-rule name
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+```bash
+# 2. After rules load, verify the new metric exists
+curl -sG --data-urlencode "query=job:http_request_duration_p95:rate5m" \
+  "$PROM/api/v1/query" | jq '.data.result | length'   # → > 0
+
+# 3. Verify it matches the original expression for at least one sample window
+# (Both queries should produce the same value at the same timestamp.)
+
+# 4. Replace the dashboard panel expression with the recording-rule metric.
+```
+
+## Common bugs
+
+- `histogram_quantile` returns NaN → forgot `by (le)` in the inner aggregation
+- "No data" → check the metric exists (`/api/v1/series`) and the window ≥ 4× scrape interval
+- Wrong rate magnitude → counter was aggregated before `rate()` (always `rate()` first)
+- Query timeout → series count too high; use `topk(...)` + a recording rule + drop high-cardinality labels (see [`references/patterns.md`](references/patterns.md))
+
+## Resources
+
+- [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
+- [Operators](https://prometheus.io/docs/prometheus/latest/querying/operators/)
+- [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)
+- [Grafana Mimir](https://grafana.com/docs/mimir/latest/)
--- a/.agents/skills/promql/references/patterns.md
+++ b/.agents/skills/promql/references/patterns.md
@ -0,0 +1,174 @@
+# PromQL pattern library
+
+## Rate, counter, and aggregation
+
+```promql
+# rate() — per-second average
+rate(http_requests_total[5m])
+
+# CORRECT: rate first, then aggregate
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+
+# WRONG: never sum() a raw counter before rate()
+sum(http_requests_total) by (status_code)   # do NOT then rate()
+
+# increase() — total count over window
+increase(http_requests_total[1h])
+```
+
+`rate` vs `irate`:
+- `rate()` smooths the full window — use for dashboards + alerts
+- `irate()` uses the last two samples — for spike detection only, never alerting
+
+## Label matchers
+
+```promql
+http_requests_total{job="api", status_code="200"}
+http_requests_total{status_code=~"5.."}
+http_requests_total{status_code!~"2.."}
+http_requests_total{env=~"staging|production"}
+```
+
+## Aggregation
+
+```promql
+sum(rate(http_requests_total[5m])) by (service)
+avg(node_cpu_seconds_total{mode="idle"}) by (instance)
+
+# Top 5 by request rate
+topk(5, sum(rate(http_requests_total[5m])) by (service))
+
+# Count of distinct label values
+count(count(up) by (job)) by ()
+```
+
+`by` keeps only listed labels; `without` drops only listed labels.
+
+## Histogram quantiles
+
+```promql
+# Classic histogram
+histogram_quantile(0.99,
+  sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le))
+
+# Multi-service comparison — must include le AND service
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+
+# Native histograms (Prometheus 2.40+)
+histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m])))
+```
+
+Forgetting `by (le)` is the most common bug — `histogram_quantile` returns NaN or wrong values.
+
+## Ratios and error rates
+
+```promql
+# Error fraction
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / sum(rate(http_requests_total[5m]))
+
+# Success percentage
+(1 - sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+     / sum(rate(http_requests_total[5m]))) * 100
+
+# Avoid divide-by-zero
+sum(rate(errors_total[5m]))
+  / (sum(rate(requests_total[5m])) > 0)
+```
+
+## Absence and staleness
+
+```promql
+absent(up{job="api"})                  # metric disappeared
+changes(up{job="api"}[5m]) == 0        # value not changing → stale exporter
+count_over_time(up{job="api"}[5m]) > 0 # at least one sample in window
+```
+
+## Time offsets
+
+```promql
+# Current vs 1h ago
+rate(http_requests_total[5m])
+  - rate(http_requests_total[5m] offset 1h)
+
+# Day-over-day
+rate(http_requests_total[5m])
+  / rate(http_requests_total[5m] offset 1d)
+
+# Predict 2h ahead from 1h trend (linear regression)
+predict_linear(node_filesystem_avail_bytes[1h], 2 * 3600)
+```
+
+## Recording rules
+
+Naming: `<aggregation_level>:<metric_name>:<operation_and_window>`
+
+```yaml
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_requests_total:rate5m
+        expr:   sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_errors:ratio5m
+        expr: |
+          sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)
+          / sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+## SLO + burn-rate
+
+```promql
+# Availability SLO over 30 days
+1 - (sum(increase(http_requests_total{status_code=~"5.."}[30d]))
+     / sum(increase(http_requests_total[30d])))
+
+# 1h burn-rate vs target (replace 0.999 with your SLO target)
+( sum(rate(http_requests_total{status_code=~"5.."}[1h]))
+  / sum(rate(http_requests_total[1h])) )
+  / (1 - 0.999)
+```
+
+## Cardinality controls
+
+```promql
+# Top 10 metrics by series count
+topk(10, count by (__name__)({__name__=~".+"}))
+
+# Series count for one metric
+count(http_requests_total)
+
+# Cardinality of one label
+count(count by (user_id)(http_requests_total))
+```
+
+Drop a high-cardinality label at scrape time (Alloy):
+
+```alloy
+prometheus.scrape "api" {
+  targets = [...]
+  rule { source_labels = ["user_id"]
+         action        = "labeldrop" }
+}
+```
+
+## Other common patterns
+
+```promql
+# Service availability (alert)
+avg_over_time(up{job="api"}[5m]) < 0.9
+
+# Saturation — disk full in <4h
+predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
+
+# Throughput spike — current rate > 3x 1h average
+rate(http_requests_total[5m])
+  > 3 * avg_over_time(rate(http_requests_total[5m])[1h:5m])
+```