feat: add grafana dashboarding and promql agent skills

2026-07-03 12:48:42 +02:00
parent 3f7c501340
commit 304ca52cca
13 changed files with 2463 additions and 0 deletions
@@ -0,0 +1,425 @@
+# Grafana Alerting - Detailed Reference
+
+## Overview
+
+Grafana Alerting is a unified alerting system for monitoring metrics and logs across multiple data sources. It fires notifications when conditions are breached.
+
+Key capabilities:
+- Query multiple data sources in a single alert rule
+- Multi-dimensional alerts (one rule creates many alert instances)
+- Flexible notification routing via policies
+- Silences and mute timings for planned maintenance
+- Alert history and state tracking
+
+---
+
+## Core Concepts
+
+### Alert Rule
+Defines what to monitor and when to fire. Contains: queries, a condition (threshold), an evaluation interval, and a pending period.
+
+### Alert Instance
+When a rule produces multi-dimensional data, it creates one alert instance per unique label combination. Example: an alert on `cpu_usage{host=~".*"}` creates one instance per host.
+
+### Alert States
+
+| State | Description |
+|-------|-------------|
+| **Normal** | Query running; condition not met |
+| **Pending** | Condition met but pending period not yet elapsed |
+| **Firing** | Condition met + pending period elapsed; notifications sent |
+| **Resolved** | Previously firing alert returned to normal |
+| **No Data** | Query returned no data (configurable behavior) |
+| **Error** | Query failed with an error (configurable behavior) |
+
+### Evaluation Group
+Alert rules are organized into evaluation groups. All rules in a group share the same evaluation interval and are evaluated sequentially.
+
+### Pending Period
+How long the condition must be continuously met before firing.
+- `0s` = fire immediately
+- `5m` = must be in breach for 5 continuous minutes
+
+### Keep Firing For
+How long an alert continues firing after the condition resolves (prevents brief recovery from clearing the alert).
+
+---
+
+## Alert Rule Types
+
+### Grafana-Managed Rules (Recommended)
+- Stored in Grafana's database
+- Can query any data source
+- Support multi-dimensional alerting
+- Support expressions (math, reduce, threshold)
+
+### Data Source-Managed Rules (Prometheus/Mimir/Loki)
+- Rules stored in the external system
+- Evaluated by the external system
+- Grafana provides UI to manage them
+- Useful when migrating from Prometheus alerting
+
+---
+
+## Creating Grafana-Managed Alert Rules
+
+Navigate to: **Alerting > Alert rules > New alert rule**
+
+### Step 1: Define query and condition
+
+Write one or more queries (labeled A, B, C...).
+
+Example with Prometheus:
+```promql
+# Query A: CPU usage per host
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Expression types you can add after queries:
+- **Math**: `$A > 80` or `($A + $B) / 2`
+- **Reduce**: Collapse time series to single value (Last, Mean, Sum, Max, Min)
+- **Resample**: Change time resolution
+- **Classic conditions**: Multiple threshold conditions with AND/OR
+- **Threshold**: Set the firing threshold
+
+### Step 2: Set evaluation behavior
+
+| Setting | Description |
+|---------|-------------|
+| Folder | Organize rules; folder is the RBAC boundary |
+| Evaluation group | Group name (all rules share this group's interval) |
+| Evaluation interval | How often the rule is evaluated (e.g., 1m) |
+| Pending period | How long condition must hold before firing (e.g., 5m) |
+| Keep firing for | How long alert stays firing after recovery |
+
+### Step 3: Configure labels and notifications
+
+**Labels** - Key-value pairs attached to alert instances. Used for routing and grouping:
+```
+severity=critical
+team=infrastructure
+service=database
+environment=production
+```
+
+**Annotations** - Context included in notification messages:
+```
+Summary: CPU usage above 80% on {{ $labels.instance }}
+Description: CPU usage is {{ $values.A.Value | humanize }}% on {{ $labels.instance }}
+Runbook URL: https://runbooks.company.com/cpu-high
+```
+
+**Notification settings**:
+- Set a **Contact point** to send directly, bypassing the routing policy tree
+- Or leave empty to use the notification policy routing tree
+
+### Step 4: No data and error handling
+
+| Situation | Options |
+|-----------|---------|
+| No data | Alerting, OK, No Data state, Keep last state |
+| Query error | Alerting, OK, Error state, Keep last state |
+
+---
+
+## Contact Points
+
+Contact points are notification destinations.
+
+Navigate to: **Alerting > Contact points**
+
+### Supported integrations
+
+| Integration | Notes |
+|-------------|-------|
+| Email | Requires SMTP config in grafana.ini |
+| Slack | Webhook URL or API token + channel |
+| PagerDuty | Integration key |
+| OpsGenie | API key |
+| VictorOps (Splunk) | API key |
+| Microsoft Teams | Incoming webhook URL |
+| Discord | Webhook URL |
+| Telegram | Bot token + chat ID |
+| Webhook | HTTP POST to custom URL |
+| Alertmanager | Forward to external Alertmanager |
+| Pushover | User key + API token |
+| LINE | LINE Notify token |
+
+### Email configuration in grafana.ini
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@company.com
+password = app-password
+from_address = alerts@company.com
+from_name = Grafana Alerts
+```
+
+### Webhook payload format
+Grafana POSTs a JSON payload to webhook endpoints:
+```json
+{
+  "receiver": "webhook-receiver",
+  "status": "firing",
+  "alerts": [
+    {
+      "status": "firing",
+      "labels": { "alertname": "HighCPU", "instance": "server1" },
+      "annotations": { "summary": "CPU above 80%" },
+      "startsAt": "2024-01-15T10:00:00Z",
+      "generatorURL": "http://grafana/alerting/..."
+    }
+  ],
+  "groupLabels": { "alertname": "HighCPU" },
+  "externalURL": "http://grafana"
+}
+```
+
+---
+
+## Notification Policies
+
+Notification policies route alerts to contact points based on label matchers.
+
+Navigate to: **Alerting > Notification policies**
+
+### Default policy
+The root policy - all alerts reach here if no specific policy matches.
+
+### Child policies
+Add policies that match specific labels:
+```
+Match labels:
+  severity = critical    -> contact: pagerduty
+  team = infrastructure  -> contact: slack-infra
+  environment = staging  -> contact: email-dev
+```
+
+### Policy settings
+
+| Setting | Description |
+|---------|-------------|
+| Contact point | Where to send matching alerts |
+| Continue matching | If true, also evaluate subsequent sibling policies |
+| Group by | Labels used for batching alerts into single notifications |
+| Group wait | Wait before sending first notification for a new group (default: 30s) |
+| Group interval | Wait before sending updates for an existing group (default: 5m) |
+| Repeat interval | Wait before re-sending for still-firing alerts (default: 4h) |
+
+### Grouping
+When multiple alerts have the same "group by" labels, they are batched into a single notification. This prevents notification storms.
+
+Example:
+- 50 hosts alert on HighCPU simultaneously
+- Group by: `[alertname, datacenter]`
+- Result: 1 notification per datacenter containing all affected hosts
+
+---
+
+## Notification Templates
+
+Customize notification message format using Go templating.
+
+Navigate to: **Alerting > Contact points > Notification templates**
+
+### Built-in variables
+```
+{{ $labels }}          # Alert labels as map
+{{ $values }}          # Query values map
+{{ $labels.instance }} # Specific label value
+{{ $values.A.Value }}  # Specific query value
+{{ $status }}          # firing or resolved
+{{ $startsAt }}        # When alert started firing
+```
+
+### Example Slack template
+```
+{{ define "slack_message" }}
+{{ if eq .Status "firing" }}:red_circle:{{ else }}:large_green_circle:{{ end }} *{{ .Labels.alertname }}*
+
+*Status:* {{ .Status }}
+*Severity:* {{ .Labels.severity }}
+
+{{ range .Alerts }}
+*Instance:* {{ .Labels.instance }}
+*Value:* {{ .Values.A.Value | humanize }}
+*Summary:* {{ .Annotations.summary }}
+{{ end }}
+{{ end }}
+```
+
+### Humanize functions
+```
+{{ $value | humanize }}           # "12.3k"
+{{ $value | humanize1024 }}       # "12.3Ki"
+{{ $value | humanizeBytes }}      # "12.3 kB"
+{{ $value | humanizeDuration }}   # "3h 2m 1s"
+{{ $value | humanizePercentage }} # "12.3%"
+```
+
+---
+
+## Silences
+
+Silences temporarily suppress alert notifications without stopping alert evaluation.
+
+Navigate to: **Alerting > Silences**
+
+### Create a silence
+1. Click **Add silence**
+2. Set start time and end time (or duration)
+3. Add label matchers:
+   ```
+   alertname = HighCPU
+   instance =~ ".*staging.*"   # Regex match
+   severity != critical         # Negative match
+   ```
+4. Add a comment explaining why
+5. Save
+
+### Use cases
+- Planned maintenance windows
+- Known issues being investigated
+- Silencing noisy alerts during deploys
+
+From a firing alert detail view: click **Silence** to pre-populate label matchers.
+
+---
+
+## Mute Timings
+
+Recurring schedules when notifications are suppressed (unlike silences which are one-time).
+
+Navigate to: **Alerting > Mute timings**
+
+### Example mute timings
+```
+Name: no-alerts-weekends
+  Weekdays: Saturday, Sunday
+
+Name: business-hours-only
+  Weekdays: Monday-Friday
+  Times: 09:00-17:00
+```
+
+Attach to notification policies in the policy settings.
+
+---
+
+## RBAC for Alerting
+
+Default permissions by role:
+
+| Action | Viewer | Editor | Admin |
+|--------|--------|--------|-------|
+| View alert rules | Yes | Yes | Yes |
+| Create/edit alert rules | No | Yes | Yes |
+| Delete alert rules | No | No | Yes |
+| Manage contact points | No | No | Yes |
+| Manage notification policies | No | No | Yes |
+| Create silences | No | Yes | Yes |
+
+---
+
+## grafana.ini Alerting Configuration
+
+```ini
+[unified_alerting]
+# Enable unified alerting (default: true since Grafana 9)
+enabled = true
+
+# Maximum alert instances a single rule can produce
+max_annotations_to_keep = 100
+
+# Evaluation timeout
+evaluation_timeout = 30s
+
+# Minimum evaluation interval (prevent too-frequent evaluation)
+min_interval = 10s
+
+[smtp]
+# Required for email contact points
+enabled = true
+host = smtp.example.com:587
+```
+
+---
+
+## Common Alert Rule Examples
+
+### CPU usage above threshold
+```promql
+# Query A
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+# Threshold: A > 80
+# Pending: 5m
+# Labels: severity=warning
+```
+
+### Memory usage
+```promql
+# Query A
+(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
+# Threshold: A > 90
+```
+
+### HTTP error rate
+```promql
+# Query A: error requests
+sum(rate(http_requests_total{status=~"5.."}[5m]))
+# Query B: total requests
+sum(rate(http_requests_total[5m]))
+# Expression C (Math): $A / $B * 100
+# Threshold: C > 5
+```
+
+### Disk space
+```promql
+# Query A
+(1 - (node_filesystem_free_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) * 100
+# Threshold: A > 85
+```
+
+### Service down (no data = firing)
+```promql
+# Query A
+up{job="my-service"}
+# Threshold: A < 1
+# No data handling: Alerting (treat absence as firing)
+```
+
+### Loki log error rate
+```logql
+# Query A
+sum(rate({job="api"} |= "ERROR" [5m]))
+# Threshold: A > 10
+```
+
+---
+
+## Connecting Alerts to Dashboards
+
+### Create alert from panel
+1. Open panel editor
+2. Click **Alert** tab
+3. Click **Create alert rule from this panel**
+4. Pre-populates query from the panel
+
+### Link alert to dashboard panel
+In the alert rule definition, set **Dashboard** and **Panel** to link to a specific visualization. The alert state badge appears on the panel and clicking it goes to the alert rule.
+
+---
+
+## High Availability (HA) Alerting
+
+```ini
+[unified_alerting]
+ha_peers = grafana-1:9094,grafana-2:9094,grafana-3:9094
+ha_advertise_address = ${POD_IP}:9094
+ha_peer_timeout = 15s
+ha_gossip_interval = 200ms
+ha_push_pull_interval = 60s
+```
+
+Multiple Grafana instances share state to avoid duplicate notifications.