feat: add grafana dashboarding and promql agent skills
This commit is contained in:
Moritz Graf
@@ -0,0 +1,153 @@
|
||||
---
|
||||
name: grafana-oss
|
||||
license: Apache-2.0
|
||||
description: Configure Grafana OSS — provisions dashboards from YAML, sets up data sources (Prometheus / Loki / Tempo / Pyroscope), writes dashboard JSON with template variables, builds panel queries, assigns built-in roles (Viewer / Editor / Admin / GrafanaAdmin), mints service-account tokens, edits grafana.ini server config, creates annotations, installs plugins via provisioning, and validates each step with a health-check curl. Use when building dashboards, configuring data sources, setting up provisioning YAML, picking a panel type, writing template variables, managing users and roles, configuring SMTP/OAuth in grafana.ini, creating annotations via API, troubleshooting why a provisioned dashboard isn't showing up, or running Grafana OSS locally — even when the user says "set up a Prometheus data source", "provision dashboards from git", "make a service account", or "configure SSO in OSS" without saying "Grafana OSS".
|
||||
---
|
||||
|
||||
# Grafana OSS
|
||||
|
||||
> **Docs**: https://grafana.com/docs/grafana/latest.md
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Provisioning dashboards from disk
|
||||
|
||||
1. Drop dashboard JSON file(s) under `/var/lib/grafana/dashboards/`
|
||||
2. Add a provider in `provisioning/dashboards/default.yaml` (see [§ Dashboard provisioning](#dashboard-provisioning) below)
|
||||
3. Restart Grafana so the provider config is loaded
|
||||
4. **Verify the dashboard landed**:
|
||||
```bash
|
||||
curl https://grafana.example.com/api/dashboards/uid/<uid> \
|
||||
-H "Authorization: Bearer <token>" | jq '.dashboard.title'
|
||||
```
|
||||
Returns the title → success. 404 → provisioning didn't pick it up; check Grafana server logs (`journalctl -u grafana-server | grep -i provisioning`) for parse errors.
|
||||
|
||||
### Provisioning data sources
|
||||
|
||||
1. Write `provisioning/datasources/datasources.yaml` (see [§ Data source provisioning](#data-source-provisioning) below)
|
||||
2. Restart Grafana
|
||||
3. **Health-check the data source via API**:
|
||||
```bash
|
||||
curl https://grafana.example.com/api/datasources/uid/<uid>/health \
|
||||
-H "Authorization: Bearer <token>"
|
||||
# { "status": "OK", "message": "..." } → working
|
||||
# { "status": "ERROR", ... } → URL unreachable or auth misconfigured
|
||||
```
|
||||
|
||||
### Creating a service account + token
|
||||
|
||||
1. Provision via YAML or `POST /api/serviceaccounts` (full API in [references/api.md § Users + service accounts](references/api.md#users--service-accounts))
|
||||
2. Mint a token via `POST /api/serviceaccounts/{id}/tokens`
|
||||
3. **Verify the token works**:
|
||||
```bash
|
||||
curl https://grafana.example.com/api/org \
|
||||
-H "Authorization: Bearer <new-token>"
|
||||
# 200 + org JSON → token + role assignment work
|
||||
# 401 → token wrong; 403 → role wrong
|
||||
```
|
||||
|
||||
## Dashboard provisioning
|
||||
|
||||
```yaml
|
||||
# provisioning/dashboards/default.yaml
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: default
|
||||
folder: MyFolder
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 30
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
foldersFromFilesStructure: true
|
||||
```
|
||||
|
||||
For the dashboard JSON shape itself (panels, queries, template variables), see [references/dashboard-json.md](references/dashboard-json.md).
|
||||
|
||||
## Data source provisioning
|
||||
|
||||
```yaml
|
||||
# provisioning/datasources/datasources.yaml
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
jsonData:
|
||||
timeInterval: 15s
|
||||
httpMethod: POST
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki_uid
|
||||
tags: [{ key: "service.name", value: "app" }]
|
||||
serviceMap:
|
||||
datasourceUid: prometheus_uid
|
||||
nodeGraph:
|
||||
enabled: true
|
||||
|
||||
- name: Pyroscope
|
||||
type: grafana-pyroscope-datasource
|
||||
url: http://pyroscope:4040
|
||||
```
|
||||
|
||||
## RBAC (built-in roles)
|
||||
|
||||
| Role | Permissions |
|
||||
|------|-------------|
|
||||
| **Viewer** | Read dashboards, alerts |
|
||||
| **Editor** | Create/edit dashboards, alerts |
|
||||
| **Admin** | Manage data sources, users, plugins |
|
||||
| **GrafanaAdmin** | Server-wide admin (superuser) |
|
||||
|
||||
Service-account provisioning:
|
||||
|
||||
```yaml
|
||||
# provisioning/access-control/service_accounts.yaml
|
||||
apiVersion: 1
|
||||
serviceAccounts:
|
||||
- name: ci-reader
|
||||
orgId: 1
|
||||
role: Viewer
|
||||
tokens:
|
||||
- name: ci-token
|
||||
# expires: optional ISO 8601 timestamp; omit for no-expiry tokens
|
||||
```
|
||||
|
||||
(Custom RBAC roles with fine-grained permissions are Enterprise / Cloud only — see the `grafana-cloud/admin` skill if you need those.)
|
||||
|
||||
## Plugin provisioning
|
||||
|
||||
```yaml
|
||||
# provisioning/plugins/plugins.yaml
|
||||
apiVersion: 1
|
||||
apps:
|
||||
- type: grafana-pyroscope-app
|
||||
disabled: false
|
||||
jsonData:
|
||||
backendUrl: http://pyroscope:4040
|
||||
```
|
||||
|
||||
After restart, verify via `GET /api/plugins/<plugin-id>/health`.
|
||||
|
||||
## References
|
||||
|
||||
- [`references/dashboard-json.md`](references/dashboard-json.md) — full dashboard JSON model + template variables + common problems (uid uniqueness, gridPos arithmetic, datasource uid matching)
|
||||
- [`references/dashboards.md`](references/dashboards.md) — dashboard workflows, settings, variables, annotations, sharing, versions, playlists, and provisioning-as-code
|
||||
- [`references/datasources.md`](references/datasources.md) — data source setup and query examples for Prometheus, Loki, Tempo, SQL, CloudWatch, and plugins
|
||||
- [`references/panel-types.md`](references/panel-types.md) — panel-type table + decision guide for picking the right one
|
||||
- [`references/panels.md`](references/panels.md) — panel editor, visualization options, field config, transformations, query options, inspection, and performance tips
|
||||
- [`references/alerting.md`](references/alerting.md) — alerting concepts, contact points, notification policies, templates, silences, and common rule examples
|
||||
- [`references/api.md`](references/api.md) — full Grafana OSS API reference (dashboards, data sources, users, service accounts, annotations) with verification curls and common failure modes
|
||||
- [`references/config.md`](references/config.md) — `grafana.ini` server / database / SMTP / auth / security / feature-toggle config + restart-required issues
|
||||
@@ -0,0 +1,425 @@
|
||||
# Grafana Alerting - Detailed Reference
|
||||
|
||||
## Overview
|
||||
|
||||
Grafana Alerting is a unified alerting system for monitoring metrics and logs across multiple data sources. It fires notifications when conditions are breached.
|
||||
|
||||
Key capabilities:
|
||||
- Query multiple data sources in a single alert rule
|
||||
- Multi-dimensional alerts (one rule creates many alert instances)
|
||||
- Flexible notification routing via policies
|
||||
- Silences and mute timings for planned maintenance
|
||||
- Alert history and state tracking
|
||||
|
||||
---
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Alert Rule
|
||||
Defines what to monitor and when to fire. Contains: queries, a condition (threshold), an evaluation interval, and a pending period.
|
||||
|
||||
### Alert Instance
|
||||
When a rule produces multi-dimensional data, it creates one alert instance per unique label combination. Example: an alert on `cpu_usage{host=~".*"}` creates one instance per host.
|
||||
|
||||
### Alert States
|
||||
|
||||
| State | Description |
|
||||
|-------|-------------|
|
||||
| **Normal** | Query running; condition not met |
|
||||
| **Pending** | Condition met but pending period not yet elapsed |
|
||||
| **Firing** | Condition met + pending period elapsed; notifications sent |
|
||||
| **Resolved** | Previously firing alert returned to normal |
|
||||
| **No Data** | Query returned no data (configurable behavior) |
|
||||
| **Error** | Query failed with an error (configurable behavior) |
|
||||
|
||||
### Evaluation Group
|
||||
Alert rules are organized into evaluation groups. All rules in a group share the same evaluation interval and are evaluated sequentially.
|
||||
|
||||
### Pending Period
|
||||
How long the condition must be continuously met before firing.
|
||||
- `0s` = fire immediately
|
||||
- `5m` = must be in breach for 5 continuous minutes
|
||||
|
||||
### Keep Firing For
|
||||
How long an alert continues firing after the condition resolves (prevents brief recovery from clearing the alert).
|
||||
|
||||
---
|
||||
|
||||
## Alert Rule Types
|
||||
|
||||
### Grafana-Managed Rules (Recommended)
|
||||
- Stored in Grafana's database
|
||||
- Can query any data source
|
||||
- Support multi-dimensional alerting
|
||||
- Support expressions (math, reduce, threshold)
|
||||
|
||||
### Data Source-Managed Rules (Prometheus/Mimir/Loki)
|
||||
- Rules stored in the external system
|
||||
- Evaluated by the external system
|
||||
- Grafana provides UI to manage them
|
||||
- Useful when migrating from Prometheus alerting
|
||||
|
||||
---
|
||||
|
||||
## Creating Grafana-Managed Alert Rules
|
||||
|
||||
Navigate to: **Alerting > Alert rules > New alert rule**
|
||||
|
||||
### Step 1: Define query and condition
|
||||
|
||||
Write one or more queries (labeled A, B, C...).
|
||||
|
||||
Example with Prometheus:
|
||||
```promql
|
||||
# Query A: CPU usage per host
|
||||
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
|
||||
```
|
||||
|
||||
Expression types you can add after queries:
|
||||
- **Math**: `$A > 80` or `($A + $B) / 2`
|
||||
- **Reduce**: Collapse time series to single value (Last, Mean, Sum, Max, Min)
|
||||
- **Resample**: Change time resolution
|
||||
- **Classic conditions**: Multiple threshold conditions with AND/OR
|
||||
- **Threshold**: Set the firing threshold
|
||||
|
||||
### Step 2: Set evaluation behavior
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Folder | Organize rules; folder is the RBAC boundary |
|
||||
| Evaluation group | Group name (all rules share this group's interval) |
|
||||
| Evaluation interval | How often the rule is evaluated (e.g., 1m) |
|
||||
| Pending period | How long condition must hold before firing (e.g., 5m) |
|
||||
| Keep firing for | How long alert stays firing after recovery |
|
||||
|
||||
### Step 3: Configure labels and notifications
|
||||
|
||||
**Labels** - Key-value pairs attached to alert instances. Used for routing and grouping:
|
||||
```
|
||||
severity=critical
|
||||
team=infrastructure
|
||||
service=database
|
||||
environment=production
|
||||
```
|
||||
|
||||
**Annotations** - Context included in notification messages:
|
||||
```
|
||||
Summary: CPU usage above 80% on {{ $labels.instance }}
|
||||
Description: CPU usage is {{ $values.A.Value | humanize }}% on {{ $labels.instance }}
|
||||
Runbook URL: https://runbooks.company.com/cpu-high
|
||||
```
|
||||
|
||||
**Notification settings**:
|
||||
- Set a **Contact point** to send directly, bypassing the routing policy tree
|
||||
- Or leave empty to use the notification policy routing tree
|
||||
|
||||
### Step 4: No data and error handling
|
||||
|
||||
| Situation | Options |
|
||||
|-----------|---------|
|
||||
| No data | Alerting, OK, No Data state, Keep last state |
|
||||
| Query error | Alerting, OK, Error state, Keep last state |
|
||||
|
||||
---
|
||||
|
||||
## Contact Points
|
||||
|
||||
Contact points are notification destinations.
|
||||
|
||||
Navigate to: **Alerting > Contact points**
|
||||
|
||||
### Supported integrations
|
||||
|
||||
| Integration | Notes |
|
||||
|-------------|-------|
|
||||
| Email | Requires SMTP config in grafana.ini |
|
||||
| Slack | Webhook URL or API token + channel |
|
||||
| PagerDuty | Integration key |
|
||||
| OpsGenie | API key |
|
||||
| VictorOps (Splunk) | API key |
|
||||
| Microsoft Teams | Incoming webhook URL |
|
||||
| Discord | Webhook URL |
|
||||
| Telegram | Bot token + chat ID |
|
||||
| Webhook | HTTP POST to custom URL |
|
||||
| Alertmanager | Forward to external Alertmanager |
|
||||
| Pushover | User key + API token |
|
||||
| LINE | LINE Notify token |
|
||||
|
||||
### Email configuration in grafana.ini
|
||||
```ini
|
||||
[smtp]
|
||||
enabled = true
|
||||
host = smtp.gmail.com:587
|
||||
user = alerts@company.com
|
||||
password = app-password
|
||||
from_address = alerts@company.com
|
||||
from_name = Grafana Alerts
|
||||
```
|
||||
|
||||
### Webhook payload format
|
||||
Grafana POSTs a JSON payload to webhook endpoints:
|
||||
```json
|
||||
{
|
||||
"receiver": "webhook-receiver",
|
||||
"status": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"status": "firing",
|
||||
"labels": { "alertname": "HighCPU", "instance": "server1" },
|
||||
"annotations": { "summary": "CPU above 80%" },
|
||||
"startsAt": "2024-01-15T10:00:00Z",
|
||||
"generatorURL": "http://grafana/alerting/..."
|
||||
}
|
||||
],
|
||||
"groupLabels": { "alertname": "HighCPU" },
|
||||
"externalURL": "http://grafana"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Notification Policies
|
||||
|
||||
Notification policies route alerts to contact points based on label matchers.
|
||||
|
||||
Navigate to: **Alerting > Notification policies**
|
||||
|
||||
### Default policy
|
||||
The root policy - all alerts reach here if no specific policy matches.
|
||||
|
||||
### Child policies
|
||||
Add policies that match specific labels:
|
||||
```
|
||||
Match labels:
|
||||
severity = critical -> contact: pagerduty
|
||||
team = infrastructure -> contact: slack-infra
|
||||
environment = staging -> contact: email-dev
|
||||
```
|
||||
|
||||
### Policy settings
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Contact point | Where to send matching alerts |
|
||||
| Continue matching | If true, also evaluate subsequent sibling policies |
|
||||
| Group by | Labels used for batching alerts into single notifications |
|
||||
| Group wait | Wait before sending first notification for a new group (default: 30s) |
|
||||
| Group interval | Wait before sending updates for an existing group (default: 5m) |
|
||||
| Repeat interval | Wait before re-sending for still-firing alerts (default: 4h) |
|
||||
|
||||
### Grouping
|
||||
When multiple alerts have the same "group by" labels, they are batched into a single notification. This prevents notification storms.
|
||||
|
||||
Example:
|
||||
- 50 hosts alert on HighCPU simultaneously
|
||||
- Group by: `[alertname, datacenter]`
|
||||
- Result: 1 notification per datacenter containing all affected hosts
|
||||
|
||||
---
|
||||
|
||||
## Notification Templates
|
||||
|
||||
Customize notification message format using Go templating.
|
||||
|
||||
Navigate to: **Alerting > Contact points > Notification templates**
|
||||
|
||||
### Built-in variables
|
||||
```
|
||||
{{ $labels }} # Alert labels as map
|
||||
{{ $values }} # Query values map
|
||||
{{ $labels.instance }} # Specific label value
|
||||
{{ $values.A.Value }} # Specific query value
|
||||
{{ $status }} # firing or resolved
|
||||
{{ $startsAt }} # When alert started firing
|
||||
```
|
||||
|
||||
### Example Slack template
|
||||
```
|
||||
{{ define "slack_message" }}
|
||||
{{ if eq .Status "firing" }}:red_circle:{{ else }}:large_green_circle:{{ end }} *{{ .Labels.alertname }}*
|
||||
|
||||
*Status:* {{ .Status }}
|
||||
*Severity:* {{ .Labels.severity }}
|
||||
|
||||
{{ range .Alerts }}
|
||||
*Instance:* {{ .Labels.instance }}
|
||||
*Value:* {{ .Values.A.Value | humanize }}
|
||||
*Summary:* {{ .Annotations.summary }}
|
||||
{{ end }}
|
||||
{{ end }}
|
||||
```
|
||||
|
||||
### Humanize functions
|
||||
```
|
||||
{{ $value | humanize }} # "12.3k"
|
||||
{{ $value | humanize1024 }} # "12.3Ki"
|
||||
{{ $value | humanizeBytes }} # "12.3 kB"
|
||||
{{ $value | humanizeDuration }} # "3h 2m 1s"
|
||||
{{ $value | humanizePercentage }} # "12.3%"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Silences
|
||||
|
||||
Silences temporarily suppress alert notifications without stopping alert evaluation.
|
||||
|
||||
Navigate to: **Alerting > Silences**
|
||||
|
||||
### Create a silence
|
||||
1. Click **Add silence**
|
||||
2. Set start time and end time (or duration)
|
||||
3. Add label matchers:
|
||||
```
|
||||
alertname = HighCPU
|
||||
instance =~ ".*staging.*" # Regex match
|
||||
severity != critical # Negative match
|
||||
```
|
||||
4. Add a comment explaining why
|
||||
5. Save
|
||||
|
||||
### Use cases
|
||||
- Planned maintenance windows
|
||||
- Known issues being investigated
|
||||
- Silencing noisy alerts during deploys
|
||||
|
||||
From a firing alert detail view: click **Silence** to pre-populate label matchers.
|
||||
|
||||
---
|
||||
|
||||
## Mute Timings
|
||||
|
||||
Recurring schedules when notifications are suppressed (unlike silences which are one-time).
|
||||
|
||||
Navigate to: **Alerting > Mute timings**
|
||||
|
||||
### Example mute timings
|
||||
```
|
||||
Name: no-alerts-weekends
|
||||
Weekdays: Saturday, Sunday
|
||||
|
||||
Name: business-hours-only
|
||||
Weekdays: Monday-Friday
|
||||
Times: 09:00-17:00
|
||||
```
|
||||
|
||||
Attach to notification policies in the policy settings.
|
||||
|
||||
---
|
||||
|
||||
## RBAC for Alerting
|
||||
|
||||
Default permissions by role:
|
||||
|
||||
| Action | Viewer | Editor | Admin |
|
||||
|--------|--------|--------|-------|
|
||||
| View alert rules | Yes | Yes | Yes |
|
||||
| Create/edit alert rules | No | Yes | Yes |
|
||||
| Delete alert rules | No | No | Yes |
|
||||
| Manage contact points | No | No | Yes |
|
||||
| Manage notification policies | No | No | Yes |
|
||||
| Create silences | No | Yes | Yes |
|
||||
|
||||
---
|
||||
|
||||
## grafana.ini Alerting Configuration
|
||||
|
||||
```ini
|
||||
[unified_alerting]
|
||||
# Enable unified alerting (default: true since Grafana 9)
|
||||
enabled = true
|
||||
|
||||
# Maximum alert instances a single rule can produce
|
||||
max_annotations_to_keep = 100
|
||||
|
||||
# Evaluation timeout
|
||||
evaluation_timeout = 30s
|
||||
|
||||
# Minimum evaluation interval (prevent too-frequent evaluation)
|
||||
min_interval = 10s
|
||||
|
||||
[smtp]
|
||||
# Required for email contact points
|
||||
enabled = true
|
||||
host = smtp.example.com:587
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Alert Rule Examples
|
||||
|
||||
### CPU usage above threshold
|
||||
```promql
|
||||
# Query A
|
||||
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
|
||||
# Threshold: A > 80
|
||||
# Pending: 5m
|
||||
# Labels: severity=warning
|
||||
```
|
||||
|
||||
### Memory usage
|
||||
```promql
|
||||
# Query A
|
||||
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
|
||||
# Threshold: A > 90
|
||||
```
|
||||
|
||||
### HTTP error rate
|
||||
```promql
|
||||
# Query A: error requests
|
||||
sum(rate(http_requests_total{status=~"5.."}[5m]))
|
||||
# Query B: total requests
|
||||
sum(rate(http_requests_total[5m]))
|
||||
# Expression C (Math): $A / $B * 100
|
||||
# Threshold: C > 5
|
||||
```
|
||||
|
||||
### Disk space
|
||||
```promql
|
||||
# Query A
|
||||
(1 - (node_filesystem_free_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) * 100
|
||||
# Threshold: A > 85
|
||||
```
|
||||
|
||||
### Service down (no data = firing)
|
||||
```promql
|
||||
# Query A
|
||||
up{job="my-service"}
|
||||
# Threshold: A < 1
|
||||
# No data handling: Alerting (treat absence as firing)
|
||||
```
|
||||
|
||||
### Loki log error rate
|
||||
```logql
|
||||
# Query A
|
||||
sum(rate({job="api"} |= "ERROR" [5m]))
|
||||
# Threshold: A > 10
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Connecting Alerts to Dashboards
|
||||
|
||||
### Create alert from panel
|
||||
1. Open panel editor
|
||||
2. Click **Alert** tab
|
||||
3. Click **Create alert rule from this panel**
|
||||
4. Pre-populates query from the panel
|
||||
|
||||
### Link alert to dashboard panel
|
||||
In the alert rule definition, set **Dashboard** and **Panel** to link to a specific visualization. The alert state badge appears on the panel and clicking it goes to the alert rule.
|
||||
|
||||
---
|
||||
|
||||
## High Availability (HA) Alerting
|
||||
|
||||
```ini
|
||||
[unified_alerting]
|
||||
ha_peers = grafana-1:9094,grafana-2:9094,grafana-3:9094
|
||||
ha_advertise_address = ${POD_IP}:9094
|
||||
ha_peer_timeout = 15s
|
||||
ha_gossip_interval = 200ms
|
||||
ha_push_pull_interval = 60s
|
||||
```
|
||||
|
||||
Multiple Grafana instances share state to avoid duplicate notifications.
|
||||
@@ -0,0 +1,107 @@
|
||||
# Grafana OSS API
|
||||
|
||||
Base URL: `https://your-grafana.example.com/api/`. Auth: service account token (`Authorization: Bearer <token>`).
|
||||
|
||||
## Contents
|
||||
|
||||
- [Dashboards](#dashboards)
|
||||
- [Data sources](#data-sources)
|
||||
- [Users + service accounts](#users--service-accounts)
|
||||
- [Annotations](#annotations)
|
||||
|
||||
## Dashboards
|
||||
|
||||
```bash
|
||||
# Search
|
||||
GET /api/search?query=service&type=dash-db&folderIds=1
|
||||
|
||||
# Get by UID
|
||||
GET /api/dashboards/uid/{uid}
|
||||
|
||||
# Create / update (overwrite: true replaces existing)
|
||||
POST /api/dashboards/db
|
||||
Body: { "dashboard": {...}, "folderUID": "...", "overwrite": true }
|
||||
|
||||
# Delete
|
||||
DELETE /api/dashboards/uid/{uid}
|
||||
```
|
||||
|
||||
After provisioning a dashboard via YAML, verify it landed:
|
||||
```bash
|
||||
curl https://grafana.example.com/api/dashboards/uid/<uid> \
|
||||
-H "Authorization: Bearer <token>" | jq '.dashboard.title'
|
||||
# Should print the dashboard's title. 404 = not provisioned correctly.
|
||||
```
|
||||
|
||||
## Data sources
|
||||
|
||||
```bash
|
||||
# List
|
||||
GET /api/datasources
|
||||
|
||||
# Get by UID
|
||||
GET /api/datasources/uid/{uid}
|
||||
|
||||
# Create
|
||||
POST /api/datasources
|
||||
Body: { "name": "...", "type": "...", "url": "...", "access": "proxy" }
|
||||
|
||||
# Health-check (good post-provision validation)
|
||||
GET /api/datasources/uid/{uid}/health
|
||||
# Returns { "status": "OK" | "ERROR", "message": "..." }
|
||||
```
|
||||
|
||||
## Users + service accounts
|
||||
|
||||
```bash
|
||||
# List org users
|
||||
GET /api/org/users
|
||||
|
||||
# List service accounts
|
||||
GET /api/serviceaccounts/search?perpage=100&page=1
|
||||
|
||||
# Create service account
|
||||
POST /api/serviceaccounts
|
||||
Body: { "name": "ci-reader", "role": "Viewer", "isDisabled": false }
|
||||
|
||||
# Mint a token
|
||||
POST /api/serviceaccounts/{id}/tokens
|
||||
Body: { "name": "ci-token", "secondsToLive": 0 } # 0 = no expiry
|
||||
|
||||
# Verify a token works
|
||||
curl https://grafana.example.com/api/org \
|
||||
-H "Authorization: Bearer <new-token>"
|
||||
# 200 + org JSON = token + role assignment work.
|
||||
```
|
||||
|
||||
## Annotations
|
||||
|
||||
```bash
|
||||
# Create
|
||||
curl -X POST https://grafana.example.com/api/annotations \
|
||||
-H 'Authorization: Bearer <token>' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"dashboardUID": "service-overview",
|
||||
"panelId": 1,
|
||||
"time": 1706745600000,
|
||||
"timeEnd": 1706749200000,
|
||||
"tags": ["deploy", "v2.0"],
|
||||
"text": "Deployed v2.0"
|
||||
}'
|
||||
|
||||
# Find (by tag)
|
||||
GET /api/annotations?tags=deploy&from=1706745600000&to=1706832000000
|
||||
|
||||
# Delete
|
||||
DELETE /api/annotations/{id}
|
||||
```
|
||||
|
||||
## Common failure modes
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|---|---|
|
||||
| `401 Unauthorized` | Token expired or wrong stack URL — check the Authorization header |
|
||||
| `403 Forbidden` on dashboard create | Service account lacks Editor role on the target folder |
|
||||
| `412 Precondition Failed` on POST `/dashboards/db` | UID exists and you didn't set `overwrite: true` |
|
||||
| Data source health check `ERROR` | Network unreachable (check `url` field) or credentials wrong |
|
||||
@@ -0,0 +1,100 @@
|
||||
# grafana.ini configuration
|
||||
|
||||
Reference for `grafana.ini` server-side settings.
|
||||
|
||||
## Contents
|
||||
|
||||
- [Server + database](#server--database)
|
||||
- [Alerting](#alerting)
|
||||
- [SMTP](#smtp)
|
||||
- [Auth (OAuth example)](#auth-oauth-example)
|
||||
- [Security](#security)
|
||||
- [Feature toggles](#feature-toggles)
|
||||
|
||||
## Server + database
|
||||
|
||||
```ini
|
||||
[server]
|
||||
http_port = 3000
|
||||
domain = grafana.example.com
|
||||
root_url = https://grafana.example.com/
|
||||
|
||||
[database]
|
||||
type = postgres
|
||||
host = postgres:5432
|
||||
name = grafana
|
||||
user = grafana
|
||||
password = secret
|
||||
```
|
||||
|
||||
After changing these, restart Grafana and verify:
|
||||
- `curl http://localhost:3000/api/health` returns `{"database": "ok", ...}`
|
||||
- `curl http://localhost:3000/api/admin/settings | jq '.database'` shows the new config
|
||||
|
||||
## Alerting
|
||||
|
||||
```ini
|
||||
[alerting]
|
||||
enabled = true
|
||||
|
||||
[unified_alerting]
|
||||
enabled = true
|
||||
```
|
||||
|
||||
## SMTP
|
||||
|
||||
```ini
|
||||
[smtp]
|
||||
enabled = true
|
||||
host = smtp.gmail.com:587
|
||||
user = alerts@example.com
|
||||
password = yourpassword
|
||||
from_address = alerts@example.com
|
||||
```
|
||||
|
||||
Verify by sending a test email via the Alerting → Contact points UI.
|
||||
|
||||
## Auth (OAuth example)
|
||||
|
||||
```ini
|
||||
[auth.generic_oauth]
|
||||
enabled = true
|
||||
name = Okta
|
||||
client_id = your_client_id
|
||||
client_secret = your_secret
|
||||
auth_url = https://your-org.okta.com/oauth2/v1/authorize
|
||||
token_url = https://your-org.okta.com/oauth2/v1/token
|
||||
api_url = https://your-org.okta.com/oauth2/v1/userinfo
|
||||
scopes = openid profile email groups
|
||||
```
|
||||
|
||||
For SAML and GitHub OAuth, see the [grafana-cloud/admin skill](../../../grafana-cloud/admin/references/sso.md). The configs are the same in OSS.
|
||||
|
||||
## Security
|
||||
|
||||
```ini
|
||||
[security]
|
||||
admin_user = admin
|
||||
admin_password = secret
|
||||
allow_embedding = true # required for embedding dashboards in iframes
|
||||
```
|
||||
|
||||
`admin_password` is only consulted on first startup. To change later, use `grafana-cli admin reset-admin-password <new-password>`.
|
||||
|
||||
## Feature toggles
|
||||
|
||||
```ini
|
||||
[feature_toggles]
|
||||
enable = publicDashboards
|
||||
```
|
||||
|
||||
Multiple toggles are space-separated:
|
||||
```ini
|
||||
enable = publicDashboards correlations grafanaApiServer
|
||||
```
|
||||
|
||||
## Common problems
|
||||
|
||||
- **`grafana.ini` changes need a restart** — config is read at startup, not live-reloaded
|
||||
- **`root_url` matters for OAuth callbacks** — if you set `domain` but not `root_url`, OAuth redirect URIs may not match
|
||||
- **Sections are global** — `[server]`, `[database]`, etc. apply at the process level, not per-org
|
||||
@@ -0,0 +1,81 @@
|
||||
# Dashboard JSON model + template variables
|
||||
|
||||
## Minimal dashboard JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Service Overview",
|
||||
"uid": "service-overview",
|
||||
"time": { "from": "now-1h", "to": "now" },
|
||||
"refresh": "30s",
|
||||
"panels": [
|
||||
{
|
||||
"type": "timeseries",
|
||||
"title": "Request Rate",
|
||||
"gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
|
||||
"targets": [
|
||||
{
|
||||
"datasource": { "type": "prometheus" },
|
||||
"expr": "rate(http_requests_total{job=\"$job\"}[5m])",
|
||||
"legendFormat": "{{method}} {{status}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "red", "value": 1000 }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Template variables
|
||||
|
||||
```json
|
||||
{
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
"name": "namespace",
|
||||
"type": "query",
|
||||
"datasource": { "type": "prometheus", "uid": "prom" },
|
||||
"definition": "label_values(kube_pod_info, namespace)",
|
||||
"includeAll": true,
|
||||
"multi": true
|
||||
},
|
||||
{
|
||||
"name": "env",
|
||||
"type": "custom",
|
||||
"query": "production,staging,dev",
|
||||
"current": { "value": "production" }
|
||||
},
|
||||
{
|
||||
"name": "interval",
|
||||
"type": "interval",
|
||||
"query": "1m,5m,15m,1h",
|
||||
"auto": true
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Reference variables in queries with `$variable`:
|
||||
```promql
|
||||
rate(http_requests_total{namespace="$namespace"}[$interval])
|
||||
```
|
||||
|
||||
## Common problems
|
||||
|
||||
- **`uid` must be unique across the org** — if you POST a dashboard with an existing UID and `overwrite: false`, Grafana returns 412 Precondition Failed
|
||||
- **`gridPos`** uses a 24-column grid; `w + x` must be ≤ 24
|
||||
- **`datasource.uid` in `targets`** must match an existing data source UID; misspell it and panels render as "Datasource not found"
|
||||
- **Template variable `query` field** is data-source-specific syntax (PromQL `label_values(...)`, LogQL `{label="…"}`, SQL, etc.)
|
||||
@@ -0,0 +1,215 @@
|
||||
# Grafana Dashboards - Detailed Reference
|
||||
|
||||
## What is a Dashboard?
|
||||
|
||||
A Grafana dashboard is a set of one or more panels organized into rows, providing an at-a-glance view of related information. Dashboards connect to data sources, display data through panels, and support interactive filtering via variables.
|
||||
|
||||
## Core Components
|
||||
|
||||
- **Panels**: Containers that display visualizations; each combines a query + a visualization type
|
||||
- **Rows**: Horizontal groupings of panels; can be collapsed for organization
|
||||
- **Data Sources**: Connections to databases, APIs, or services that panels query
|
||||
- **Variables**: Dropdown selectors at the top of a dashboard for dynamic filtering
|
||||
- **Annotations**: Markers overlaid on graphs to indicate events
|
||||
|
||||
---
|
||||
|
||||
## Creating Dashboards
|
||||
|
||||
### From scratch
|
||||
1. Click **Dashboards** in the left sidebar
|
||||
2. Click **New > New dashboard**
|
||||
3. Click **Add visualization** to add your first panel
|
||||
4. Select a data source, write a query, choose visualization type
|
||||
5. Click **Apply** to save the panel to the dashboard
|
||||
6. Click the save icon (or Ctrl+S) to save the dashboard
|
||||
|
||||
### From a template / Import
|
||||
- Click **Dashboards > New > Import**
|
||||
- Paste a dashboard JSON, enter a Grafana.com dashboard ID, or upload a JSON file
|
||||
- Grafana.com hosts thousands of community dashboards (grafana.com/grafana/dashboards)
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Settings
|
||||
|
||||
Access via the gear icon at the top-right of any dashboard.
|
||||
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| General | Name, description, tags, folder, editable flag |
|
||||
| Annotations | Configure annotation queries that overlay events on panels |
|
||||
| Variables | Add/edit template variables (dropdowns at top) |
|
||||
| Links | Add links to other dashboards or external URLs |
|
||||
| JSON Model | View/edit raw JSON for the entire dashboard |
|
||||
| Versions | Browse and restore prior versions |
|
||||
| Time options | Default time range, auto-refresh intervals, timezone |
|
||||
|
||||
---
|
||||
|
||||
## Time Range Controls
|
||||
|
||||
- **Time picker** (top-right): Select absolute or relative ranges (Last 6 hours, Last 7 days, etc.)
|
||||
- **Auto-refresh**: Set to Off, 5s, 10s, 30s, 1m, 5m, etc.
|
||||
- **Zoom**: Click-drag on any time series to zoom in
|
||||
|
||||
### Common relative time shortcuts
|
||||
```
|
||||
now-5m last 5 minutes
|
||||
now-1h last 1 hour
|
||||
now-24h last 24 hours
|
||||
now-7d last 7 days
|
||||
now/d today so far
|
||||
now-1d/d yesterday
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Annotations
|
||||
|
||||
Annotations are event markers overlaid on time series panels.
|
||||
|
||||
### Types
|
||||
- **Native annotations**: Manually add a note to a specific time directly in the dashboard
|
||||
- **Query annotations**: Pull events from a data source and display as markers
|
||||
|
||||
### Adding a manual annotation
|
||||
- Hold Ctrl (or Cmd on Mac) and click a time series panel
|
||||
- Type a description; optionally set a time range
|
||||
|
||||
---
|
||||
|
||||
## Library Panels
|
||||
|
||||
Library panels are reusable panel definitions shared across multiple dashboards.
|
||||
|
||||
- **Create**: Panel menu (3-dot) > "Create library panel"
|
||||
- **Use**: Add a panel > "Add from panel library"
|
||||
- **Update**: Edit the source panel; all dashboards using it reflect the change
|
||||
- **Unlink**: Detach to make it independent in a specific dashboard
|
||||
|
||||
---
|
||||
|
||||
## Panel Layout
|
||||
|
||||
- Drag panel corners to resize
|
||||
- Drag panel header to move
|
||||
- **Add > Row** to insert collapsible rows
|
||||
- Panel menu > **Duplicate** to clone within the same dashboard
|
||||
|
||||
---
|
||||
|
||||
## Dashboard Versions
|
||||
|
||||
- Access via Dashboard Settings > Versions
|
||||
- See who saved what and when
|
||||
- Compare two versions (diff view)
|
||||
- Restore any previous version
|
||||
|
||||
---
|
||||
|
||||
## JSON Model
|
||||
|
||||
Every dashboard is stored as a JSON document.
|
||||
|
||||
Key top-level JSON fields:
|
||||
```json
|
||||
{
|
||||
"title": "My Dashboard",
|
||||
"uid": "abc123",
|
||||
"tags": ["production", "infrastructure"],
|
||||
"time": { "from": "now-6h", "to": "now" },
|
||||
"refresh": "30s",
|
||||
"panels": [],
|
||||
"templating": { "list": [] },
|
||||
"annotations": { "list": [] }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sharing Dashboards
|
||||
|
||||
### Share link
|
||||
- Click **Share** icon > **Link** tab
|
||||
- Toggle "Lock time range" to embed the current time window
|
||||
- Toggle "Include template variable values"
|
||||
|
||||
### Snapshot
|
||||
- **Share > Snapshot**: Creates a read-only, public snapshot
|
||||
- Contains rendered data at share time - no live data source access needed
|
||||
- Can expire (1 hour, 1 day, 7 days, or never)
|
||||
|
||||
### Export / Import JSON
|
||||
- **Share > Export**: Download the dashboard JSON
|
||||
- Import at **Dashboards > New > Import**
|
||||
- Useful for version control, migration, sharing with the community
|
||||
|
||||
### Embed
|
||||
- **Share > Embed**: Generates an iframe HTML snippet
|
||||
- Requires anonymous access in Grafana config (OSS/Enterprise only)
|
||||
|
||||
### Public dashboards (Grafana 10+)
|
||||
- Make a dashboard publicly accessible with no login
|
||||
- Enable per-dashboard in Share > Public dashboard
|
||||
|
||||
---
|
||||
|
||||
## Playlists
|
||||
|
||||
Playlists cycle through dashboards automatically at a configurable interval.
|
||||
|
||||
- Create at **Dashboards > Playlists > New playlist**
|
||||
- Add dashboards by name or tag
|
||||
- Set interval (e.g., 5 minutes)
|
||||
- Append `?kiosk=1` to URL for kiosk/TV mode
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Organize with folders**: Use folders as RBAC permission boundaries
|
||||
2. **Use variables**: Make dashboards reusable across environments/services
|
||||
3. **Limit panels per dashboard**: Aim for <20 panels for performance
|
||||
4. **Use library panels**: Standardize common panels across teams
|
||||
5. **Tag dashboards**: Consistent tags for searchability
|
||||
6. **Version control**: Export JSON and store in Git
|
||||
7. **Use rows**: Collapse related panels to organize complex dashboards
|
||||
8. **Template your queries**: Use variables so one dashboard covers many targets
|
||||
|
||||
---
|
||||
|
||||
## Keyboard Shortcuts
|
||||
|
||||
| Key | Action |
|
||||
|-----|--------|
|
||||
| `Ctrl+S` | Save dashboard |
|
||||
| `e` | Open panel editor (when panel focused) |
|
||||
| `v` | Toggle panel fullscreen |
|
||||
| `d r` | Refresh all panels |
|
||||
| `d s` | Dashboard settings |
|
||||
| `d k` | Toggle kiosk mode |
|
||||
| `?` | Show all shortcuts |
|
||||
|
||||
---
|
||||
|
||||
## Provisioning Dashboards (as Code)
|
||||
|
||||
```yaml
|
||||
# /etc/grafana/provisioning/dashboards/default.yaml
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: default
|
||||
orgId: 1
|
||||
folder: Infrastructure
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 10
|
||||
allowUiUpdates: false
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
foldersFromFilesStructure: true
|
||||
```
|
||||
|
||||
Place dashboard JSON files in `/var/lib/grafana/dashboards/`. Subdirectories become folders when `foldersFromFilesStructure: true`.
|
||||
@@ -0,0 +1,462 @@
|
||||
# Grafana Data Sources - Detailed Reference
|
||||
|
||||
## Overview
|
||||
|
||||
Data sources are connections between Grafana and the systems storing your data. Grafana ships with built-in support for many popular data sources and supports additional sources via plugins.
|
||||
|
||||
- Only **Organization Admins** can add or remove data sources
|
||||
- Each data source has its own query editor
|
||||
- Data sources can be used in: Dashboard panels, Explore, Alert rules, Annotations
|
||||
|
||||
---
|
||||
|
||||
## Managing Data Sources
|
||||
|
||||
### Add a data source
|
||||
1. Go to **Connections > Data sources** (or **Configuration > Data sources** in older versions)
|
||||
2. Click **Add new data source**
|
||||
3. Search for and select the data source type
|
||||
4. Fill in connection details (URL, credentials, etc.)
|
||||
5. Click **Save & Test** to verify the connection
|
||||
|
||||
### Data source settings (common to all types)
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Name | Display name used in dashboards |
|
||||
| Default | If checked, pre-selected in new panels |
|
||||
| HTTP URL | The endpoint for the data source server |
|
||||
| Auth | Basic auth, TLS, bearer token, API key options |
|
||||
|
||||
---
|
||||
|
||||
## Prometheus
|
||||
|
||||
Native support - no plugin required.
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://prometheus:9090
|
||||
HTTP method: POST (recommended, supports longer queries)
|
||||
```
|
||||
|
||||
### Key options
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| Scrape interval | Default: 15s - should match your Prometheus scrape interval |
|
||||
| Query timeout | Default: 60s |
|
||||
| Exemplars | Enable to link metrics to traces |
|
||||
| Ruler URL | For Prometheus-managed alert rules |
|
||||
|
||||
### Query editor
|
||||
- **Metrics browser**: Browse available metrics with autocomplete
|
||||
- **Query type**: Range query (time series) or Instant query (single value)
|
||||
- **Legend**: Customize series labels using `{{label_name}}` syntax
|
||||
|
||||
### Template variables with Prometheus
|
||||
```
|
||||
# Variable type: Query
|
||||
# Query examples:
|
||||
label_values(metric_name, label_name) # All values of a label for a metric
|
||||
label_values(label_name) # All values across all metrics
|
||||
metrics(prefix) # All metric names matching prefix
|
||||
query_result(promql_expression) # PromQL result as variable values
|
||||
```
|
||||
|
||||
### Exemplars
|
||||
When enabled, Prometheus exemplars link high-cardinality trace IDs to metric data points. Requires a Tempo data source for trace drill-through.
|
||||
|
||||
---
|
||||
|
||||
## Loki (Log Aggregation)
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://loki:3100
|
||||
Maximum lines: 1000
|
||||
```
|
||||
|
||||
### Derived fields
|
||||
Extract values from log lines and link to other systems.
|
||||
|
||||
Example - extract trace ID and link to Tempo:
|
||||
```
|
||||
Name: TraceID
|
||||
Regex: traceID=(\w+)
|
||||
URL: (internal link to Tempo data source)
|
||||
```
|
||||
|
||||
### Query editor (LogQL)
|
||||
```logql
|
||||
# Basic log stream selector
|
||||
{job="nginx", namespace="production"}
|
||||
|
||||
# Filter by text
|
||||
{job="nginx"} |= "error"
|
||||
{job="nginx"} != "debug"
|
||||
|
||||
# Regex filter
|
||||
{job="nginx"} |~ "status=5\d\d"
|
||||
|
||||
# Parse and filter structured logs
|
||||
{job="api"} | json | level="error"
|
||||
{job="api"} | logfmt | duration > 1s
|
||||
|
||||
# Metrics from logs
|
||||
rate({job="nginx"} |= "error" [5m])
|
||||
sum(rate({job="nginx"}[5m])) by (status_code)
|
||||
```
|
||||
|
||||
### Template variables with Loki
|
||||
```
|
||||
label_names() # All label names
|
||||
label_values(label_name) # All values for a label
|
||||
label_values({job="nginx"}, pod) # Label values filtered by stream selector
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tempo (Distributed Tracing)
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://tempo:3200
|
||||
```
|
||||
|
||||
### Key options
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| Trace to logs | Link traces to Loki logs via trace ID |
|
||||
| Trace to metrics | Link trace spans to Prometheus metrics |
|
||||
| Service graph | Enable service dependency graph |
|
||||
| Node graph | Show trace as node graph |
|
||||
|
||||
### TraceQL (query language)
|
||||
```traceql
|
||||
# Find traces with error spans
|
||||
{status=error}
|
||||
|
||||
# Filter by service and duration
|
||||
{.service.name="frontend" && duration > 1s}
|
||||
|
||||
# Structural queries
|
||||
{.http.url=~"/api/.*"} >> {status=error}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alertmanager
|
||||
|
||||
For connecting to an external Alertmanager.
|
||||
|
||||
```ini
|
||||
URL: http://alertmanager:9093
|
||||
# Implementation: Prometheus, Mimir, or Cortex
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Elasticsearch / OpenSearch
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://elasticsearch:9200
|
||||
Index name: logs-*
|
||||
Time field name: @timestamp
|
||||
Elasticsearch version: 8.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MySQL
|
||||
|
||||
Built-in support for MySQL 5.7+ and compatible databases (MariaDB, Percona, Amazon Aurora MySQL, Azure Database for MySQL, Google Cloud SQL MySQL).
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: mysql:3306
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
Max open connections: 100
|
||||
Max idle connections: 100
|
||||
Connection max lifetime: 14400
|
||||
```
|
||||
|
||||
### Time series queries
|
||||
```sql
|
||||
SELECT
|
||||
UNIX_TIMESTAMP(time_col) as time_sec,
|
||||
value_col as value,
|
||||
name_col as metric
|
||||
FROM my_table
|
||||
WHERE $__timeFilter(time_col)
|
||||
ORDER BY time_col ASC
|
||||
```
|
||||
|
||||
### Macros
|
||||
| Macro | Description |
|
||||
|-------|-------------|
|
||||
| `$__time(column)` | Converts column to Unix timestamp |
|
||||
| `$__timeFilter(column)` | Adds WHERE clause for dashboard time range |
|
||||
| `$__timeFrom()` | Start of dashboard time range as Unix timestamp |
|
||||
| `$__timeTo()` | End of dashboard time range as Unix timestamp |
|
||||
| `$__timeGroup(column, interval)` | Groups by time interval |
|
||||
| `$__timeGroupAlias(column, interval)` | As above, aliases as "time" |
|
||||
| `$__unixEpochFilter(column)` | Time filter for Unix epoch columns |
|
||||
| `$__interval` | Auto-calculated interval for current time range |
|
||||
|
||||
### Example time series query
|
||||
```sql
|
||||
SELECT
|
||||
$__timeGroup(created_at, $__interval) AS time,
|
||||
status,
|
||||
count(*) AS cnt
|
||||
FROM orders
|
||||
WHERE $__timeFilter(created_at)
|
||||
GROUP BY 1, 2
|
||||
ORDER BY 1
|
||||
```
|
||||
|
||||
### Annotations
|
||||
```sql
|
||||
SELECT
|
||||
UNIX_TIMESTAMP(time) AS time,
|
||||
title AS text,
|
||||
tags
|
||||
FROM events
|
||||
WHERE $__timeFilter(time)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: postgres:5432
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
SSL mode: disable / require / verify-ca / verify-full
|
||||
```
|
||||
|
||||
### PostgreSQL time series
|
||||
```sql
|
||||
SELECT
|
||||
time_bucket('$__interval', time) AS time,
|
||||
avg(value)
|
||||
FROM metrics
|
||||
WHERE time BETWEEN $__timeFrom() AND $__timeTo()
|
||||
GROUP BY 1
|
||||
ORDER BY 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Microsoft SQL Server
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: sqlserver:1433
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
Encrypt: false / true / disable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## InfluxDB
|
||||
|
||||
Supports InfluxDB 1.x (InfluxQL) and InfluxDB 2.x / 3.x (Flux).
|
||||
|
||||
### InfluxDB 1.x (InfluxQL)
|
||||
```ini
|
||||
URL: http://influxdb:8086
|
||||
Database: telegraf
|
||||
```
|
||||
|
||||
Query example:
|
||||
```sql
|
||||
SELECT mean("value") FROM "measurement"
|
||||
WHERE $timeFilter
|
||||
GROUP BY time($interval), "host"
|
||||
```
|
||||
|
||||
### InfluxDB 2.x (Flux)
|
||||
```ini
|
||||
URL: http://influxdb:8086
|
||||
Organization: myorg
|
||||
Token: <api-token>
|
||||
Default bucket: mydata
|
||||
```
|
||||
|
||||
Flux query example:
|
||||
```flux
|
||||
from(bucket: "mydata")
|
||||
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
||||
|> filter(fn: (r) => r._measurement == "cpu")
|
||||
|> filter(fn: (r) => r._field == "usage_idle")
|
||||
|> aggregateWindow(every: v.windowPeriod, fn: mean)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AWS CloudWatch
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
# Auth options:
|
||||
# - AWS SDK Default (instance role, ~/.aws/credentials, env vars)
|
||||
# - Access & secret key (explicit credentials)
|
||||
# - Assume role ARN
|
||||
|
||||
Default region: us-east-1
|
||||
```
|
||||
|
||||
Required IAM permissions:
|
||||
- `cloudwatch:GetMetricData`
|
||||
- `cloudwatch:ListMetrics`
|
||||
- `logs:*` (for CloudWatch Logs)
|
||||
|
||||
### CloudWatch Logs Insights
|
||||
```
|
||||
fields @timestamp, @message
|
||||
| filter level = "ERROR"
|
||||
| sort @timestamp desc
|
||||
| limit 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Azure Monitor
|
||||
|
||||
Requires Azure App Registration with:
|
||||
- Tenant ID, Client ID, Client Secret
|
||||
- `Monitoring Reader` role on target subscriptions/resource groups
|
||||
|
||||
---
|
||||
|
||||
## Google Cloud Monitoring
|
||||
|
||||
Uses Google Cloud service account credentials (JSON key file or workload identity).
|
||||
|
||||
---
|
||||
|
||||
## TestData (built-in)
|
||||
|
||||
A built-in data source for generating test/demo data without a real backend.
|
||||
|
||||
Use cases: Testing visualizations, demo dashboards, development without a real data source.
|
||||
Scenarios: Random walk, CSV metric values, streaming data, etc.
|
||||
|
||||
---
|
||||
|
||||
## Graphite
|
||||
|
||||
```ini
|
||||
URL: http://graphite:8080
|
||||
```
|
||||
|
||||
Query examples:
|
||||
```
|
||||
target(servers.*.cpu)
|
||||
averageSeries(servers.*.cpu)
|
||||
groupByNode(servers.*.cpu, 1, 'averageSeries')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Jaeger (Tracing)
|
||||
|
||||
```ini
|
||||
URL: http://jaeger:16686
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zipkin (Tracing)
|
||||
|
||||
```ini
|
||||
URL: http://zipkin:9411
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pyroscope (Continuous Profiling)
|
||||
|
||||
```ini
|
||||
URL: http://pyroscope:4040
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Source Permissions (Enterprise)
|
||||
|
||||
By default, all org users can query any data source. With RBAC:
|
||||
- Assign specific users/teams to specific data sources
|
||||
- Configure at **Data source settings > Permissions**
|
||||
|
||||
---
|
||||
|
||||
## Provisioning Data Sources (as Code)
|
||||
|
||||
```yaml
|
||||
# /etc/grafana/provisioning/datasources/prometheus.yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
version: 1
|
||||
editable: false
|
||||
jsonData:
|
||||
timeInterval: "15s"
|
||||
queryTimeout: "60s"
|
||||
httpMethod: POST
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
maxLines: 1000
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
url: "${__value.raw}"
|
||||
|
||||
- name: MySQL Production
|
||||
type: mysql
|
||||
url: mysql:3306
|
||||
database: mydb
|
||||
user: grafana
|
||||
secureJsonData:
|
||||
password: "$GRAFANA_MYSQL_PASSWORD"
|
||||
jsonData:
|
||||
maxOpenConns: 100
|
||||
maxIdleConns: 100
|
||||
connMaxLifetime: 14400
|
||||
```
|
||||
|
||||
Place YAML files in the provisioning directory; Grafana loads on startup and watches for changes.
|
||||
|
||||
---
|
||||
|
||||
## Plugin Data Sources
|
||||
|
||||
Install additional data sources via:
|
||||
- **Connections > Add new connection** (search the plugin catalog)
|
||||
- `grafana-cli plugins install <plugin-id>`
|
||||
- Docker: set `GF_INSTALL_PLUGINS` environment variable
|
||||
|
||||
Popular plugin data sources:
|
||||
- `grafana-opensearch-datasource` - OpenSearch
|
||||
- `grafana-bigquery-datasource` - Google BigQuery
|
||||
- `grafana-mongodb-datasource` - MongoDB (Enterprise)
|
||||
- `grafana-splunk-datasource` - Splunk
|
||||
- `grafana-datadog-datasource` - Datadog
|
||||
@@ -0,0 +1,29 @@
|
||||
# Panel types
|
||||
|
||||
| Panel | Use Case |
|
||||
|-------|----------|
|
||||
| **Time series** | Line/bar charts over time (default for metrics) |
|
||||
| **Stat** | Single value with color thresholds |
|
||||
| **Gauge** | Radial gauge for current value |
|
||||
| **Bar gauge** | Horizontal bars for comparisons |
|
||||
| **Table** | Tabular data, sortable columns |
|
||||
| **Logs** | Log stream viewer (Loki) |
|
||||
| **Traces** | Trace visualization (Tempo) |
|
||||
| **Heatmap** | Distribution over time |
|
||||
| **Histogram** | Value distribution |
|
||||
| **Pie chart** | Part-to-whole ratios |
|
||||
| **Geomap** | Geographic data |
|
||||
| **Canvas** | Custom SVG-based layouts |
|
||||
| **Node graph** | Service/topology graphs |
|
||||
| **Flame graph** | CPU/memory profiling |
|
||||
| **Text** | Markdown/HTML content |
|
||||
| **Alert list** | Show firing alerts |
|
||||
|
||||
## Picking a panel type
|
||||
|
||||
Default is **Time series** for almost any metric-over-time visualization. Switch from the default only when:
|
||||
|
||||
- The user wants **one number, not a series** → Stat / Gauge / Bar gauge
|
||||
- The data is **categorical, not temporal** → Pie chart / Bar gauge / Table
|
||||
- The data is **logs/traces/profiles** → Logs / Traces / Flame graph (per data type)
|
||||
- The data is **2D distribution** (e.g. histogram-over-time) → Heatmap
|
||||
@@ -0,0 +1,325 @@
|
||||
# Grafana Panels and Visualizations - Detailed Reference
|
||||
|
||||
## What is a Panel?
|
||||
|
||||
A panel is the basic building block of a Grafana dashboard. Each panel combines:
|
||||
- A **query** (or multiple queries) to a data source
|
||||
- A **visualization type** to display the data
|
||||
- **Field configuration** (units, thresholds, color mappings)
|
||||
- Optional **transformations** to reshape the data before display
|
||||
|
||||
---
|
||||
|
||||
## Panel Editor
|
||||
|
||||
Open the panel editor by clicking a panel's title > Edit, or clicking **Add visualization** for a new panel.
|
||||
|
||||
### Panel Editor Layout
|
||||
|
||||
**Top bar**: Back to dashboard, Discard changes, Save dashboard
|
||||
|
||||
**Center**: Visualization preview (live update as you configure)
|
||||
|
||||
**Toggle**: "Table view" - shows raw query results as a table for debugging
|
||||
|
||||
**Right sidebar tabs**:
|
||||
1. **Query** - configure data sources and write queries
|
||||
2. **Transform** - apply data transformations
|
||||
3. **Alert** - create alert rules from this panel
|
||||
4. Below tabs: visualization-specific options and field config
|
||||
|
||||
### Query Tab
|
||||
|
||||
- **Data source selector**: Choose which data source to query
|
||||
- **Query editor**: Data-source-specific interface (PromQL for Prometheus, LogQL for Loki, SQL for databases)
|
||||
- **Query options**:
|
||||
- Max data points: Limit data points fetched
|
||||
- Min interval: Minimum auto-calculated interval
|
||||
- Interval: Override the auto-calculated interval
|
||||
- Relative time: Override dashboard time range for this panel only
|
||||
- Time shift: Shift the time range (e.g., to compare to last week)
|
||||
- **+ Query**: Add multiple queries (labeled A, B, C...)
|
||||
- **Expression**: Add server-side math expressions combining query results
|
||||
|
||||
---
|
||||
|
||||
## Visualization Types
|
||||
|
||||
### Time Series (default)
|
||||
Best for: Metrics over time, continuous data
|
||||
- Renders as lines, points, or bars
|
||||
- Supports multiple series
|
||||
- Configurable line width, fill, point size
|
||||
- Supports thresholds as colored background regions
|
||||
- Supports annotations overlay
|
||||
- **Graph styles**: Lines, Bars, Points; can mix per series
|
||||
- **Stacking**: None, Normal, 100%
|
||||
- **Axis**: Left Y, Right Y, hidden
|
||||
- Default visualization - supports alerting
|
||||
|
||||
### Stat
|
||||
Best for: Single important metric (KPI display)
|
||||
- Shows one or more values as large text
|
||||
- Supports sparkline background
|
||||
- Color modes: value, background, none
|
||||
- Can show last value, mean, sum, etc.
|
||||
- Great for dashboards that need quick status overviews
|
||||
|
||||
### Bar Chart
|
||||
Best for: Comparing categorical data
|
||||
- Horizontal or vertical orientation
|
||||
- Grouped or stacked
|
||||
- Supports labels on bars
|
||||
|
||||
### Gauge
|
||||
Best for: Showing a value relative to min/max range
|
||||
- Circular gauge with arc
|
||||
- Configurable thresholds set colors
|
||||
- Shows current value prominently
|
||||
|
||||
### Bar Gauge
|
||||
Best for: Multiple metrics as horizontal/vertical bars
|
||||
- Useful for comparing many items
|
||||
- Supports thresholds for color coding
|
||||
- Modes: gradient, retro LCD, basic
|
||||
|
||||
### Table
|
||||
Best for: Tabular data, multi-column metrics
|
||||
- Supports sorting by column
|
||||
- Column width customization
|
||||
- Cell display modes: color text, color background, gradient gauge
|
||||
- Pagination for large datasets
|
||||
- Can embed sparklines in cells
|
||||
|
||||
### Heatmap
|
||||
Best for: Distribution over time, histogram-over-time
|
||||
- X axis: time; Y axis: buckets; Color: density/value
|
||||
- Supports pre-bucketed data (Prometheus histogram) and raw values
|
||||
- Tooltip shows exact bucket counts
|
||||
|
||||
### Histogram
|
||||
Best for: Value distribution analysis
|
||||
- Groups values into buckets
|
||||
- Can combine multiple series
|
||||
- Configurable bucket size
|
||||
|
||||
### Pie Chart
|
||||
Best for: Proportional data, parts-of-whole
|
||||
- Pie or donut style
|
||||
- Labels: name, value, percentage
|
||||
|
||||
### Logs
|
||||
Best for: Log data from Loki, Elasticsearch, etc.
|
||||
- Displays raw log lines with timestamp
|
||||
- Log level coloring (info/warn/error)
|
||||
- Search/filter within results
|
||||
- Deduplication and time wrapping
|
||||
- Prettify JSON option
|
||||
|
||||
### Traces
|
||||
Best for: Distributed tracing visualization (Tempo)
|
||||
- Renders trace spans as a waterfall/Gantt chart
|
||||
- Shows service name, operation, duration
|
||||
- Click to drill into trace details
|
||||
|
||||
### Flame Graph
|
||||
Best for: CPU profiling data (Pyroscope)
|
||||
- Visualizes call stacks by CPU time
|
||||
- Click to zoom into subtrees
|
||||
|
||||
### Node Graph
|
||||
Best for: Service dependency maps, network topology
|
||||
- Renders nodes and edges
|
||||
- Node color/size configurable by metric
|
||||
|
||||
### Geomap
|
||||
Best for: Geographic data visualization
|
||||
- Layers: markers, heatmap, route
|
||||
- Multiple base map options (OpenStreetMap, CARTO, etc.)
|
||||
- Supports GeoJSON data
|
||||
|
||||
### Canvas
|
||||
Best for: Custom layouts, process diagrams, status boards
|
||||
- Drag-and-drop element placement
|
||||
- Elements: text, metric value, rectangle, ellipse, icon, image, connections
|
||||
- Dynamic data binding per element
|
||||
|
||||
### State Timeline
|
||||
Best for: State changes over time (on/off, OK/warn/crit)
|
||||
- Horizontal bands showing state duration
|
||||
- Each series = one row
|
||||
- Color per state value
|
||||
|
||||
### Status History
|
||||
Best for: Periodic state checks over time
|
||||
- Grid: Y=services, X=time buckets
|
||||
- Color per state
|
||||
|
||||
### XY Chart
|
||||
Best for: Correlation between two metrics
|
||||
- Scatter plot
|
||||
- X and Y axis from different fields
|
||||
- Bubble size from a third field
|
||||
|
||||
### Candlestick
|
||||
Best for: Financial OHLC data
|
||||
- Open/High/Low/Close representation
|
||||
- Volume bars
|
||||
|
||||
### Text
|
||||
Best for: Documentation panels, headers
|
||||
- Renders Markdown or HTML
|
||||
|
||||
### Alert List
|
||||
Best for: Dashboard overview of current alert states
|
||||
|
||||
### Dashboard List
|
||||
Best for: Navigation panels linking to other dashboards
|
||||
|
||||
---
|
||||
|
||||
## Field Configuration (Standard Options)
|
||||
|
||||
Available for most visualizations under the visualization options panel.
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| Unit | Display unit (bytes, seconds, %, requests/sec, etc.) |
|
||||
| Min / Max | Override auto-detected min/max for scales |
|
||||
| Decimals | Number of decimal places |
|
||||
| Display name | Override the series/field name |
|
||||
| Color scheme | Fixed, thresholds-based, palette, etc. |
|
||||
| No value | Text to display when value is null |
|
||||
|
||||
### Thresholds
|
||||
Define color-coded boundaries:
|
||||
- **Absolute**: Fixed numeric values (e.g., >90 = red, >70 = yellow, else green)
|
||||
- **Percentage**: Relative to min/max
|
||||
|
||||
Example threshold config:
|
||||
```
|
||||
Base (default): Green
|
||||
70: Yellow (warn)
|
||||
90: Red (crit)
|
||||
```
|
||||
|
||||
### Value Mappings
|
||||
Transform raw values into human-readable labels or colors:
|
||||
- **Value to text**: e.g., `1` -> "OK", `0` -> "Down"
|
||||
- **Range to text**: e.g., `0-50` -> "Low"
|
||||
- **Regex to text**: Match patterns
|
||||
|
||||
### Data Links
|
||||
Create clickable links from panel values:
|
||||
- Link to other dashboards with variable values interpolated
|
||||
- Link to external systems (e.g., Kibana, PagerDuty)
|
||||
- Use `${__value.raw}` and `${__field.name}` in URLs
|
||||
|
||||
---
|
||||
|
||||
## Field Overrides
|
||||
|
||||
Apply specific field options to individual series/columns rather than all data:
|
||||
|
||||
1. Click **+ Add field override** in the Overrides section
|
||||
2. Choose override target:
|
||||
- Fields with name (exact match)
|
||||
- Fields with name matching regex
|
||||
- Fields with type (number, string, time)
|
||||
- Fields returned by query (A, B, C...)
|
||||
3. Add properties to override (unit, color, alias, thresholds, etc.)
|
||||
|
||||
Example: In a table with columns `cpu_idle` and `cpu_used`, set `cpu_used` to show as percentage and color by threshold while leaving `cpu_idle` with default styling.
|
||||
|
||||
---
|
||||
|
||||
## Transformations
|
||||
|
||||
Transformations reshape query data before rendering. Apply multiple in sequence.
|
||||
|
||||
Access: Panel editor > Transform tab > Add transformation
|
||||
|
||||
### Most-Used Transformations
|
||||
|
||||
| Transformation | Description |
|
||||
|----------------|-------------|
|
||||
| **Reduce** | Collapse a time series to a single value (last, mean, sum, max, min) |
|
||||
| **Filter by name** | Keep only specific fields/columns |
|
||||
| **Filter by value** | Filter rows where a field matches a condition |
|
||||
| **Organize fields** | Rename, reorder, or hide fields |
|
||||
| **Merge** | Combine multiple query results into one table |
|
||||
| **Join by field** | SQL-style join on a common field (e.g., time) |
|
||||
| **Group by** | Group rows and aggregate (count, sum, mean, etc.) |
|
||||
| **Sort by** | Sort rows by a field |
|
||||
| **Limit** | Keep only first N rows |
|
||||
| **Add field from calculation** | Add a new column computed from existing columns |
|
||||
| **Convert field type** | Change a field's data type |
|
||||
| **Rename by regex** | Batch rename fields using regex |
|
||||
| **Extract fields** | Parse JSON or regex from a string field |
|
||||
| **Labels to fields** | Convert label key/values into separate columns |
|
||||
| **Rows to fields** | Pivot: turn row values into column headers |
|
||||
| **Prepare time series** | Normalize time series format |
|
||||
| **Time series to table** | Convert time series format to table format |
|
||||
|
||||
Enable "Debug" toggle on any transformation to see input/output for troubleshooting.
|
||||
|
||||
---
|
||||
|
||||
## Query Options
|
||||
|
||||
### Multiple queries
|
||||
Add multiple queries (A, B, C...) to one panel. They are overlaid in the visualization. Use this to compare metrics or show multiple services on one graph.
|
||||
|
||||
### Expressions (server-side)
|
||||
Server-side expressions allow math across query results:
|
||||
- **Math**: `$A + $B`, `$A / $B * 100`
|
||||
- **Reduce**: Collapse a series to scalar
|
||||
- **Resample**: Change time resolution of a series
|
||||
- **Classic conditions**: Threshold logic (used in alerting)
|
||||
|
||||
Example - calculate error rate as percentage:
|
||||
```
|
||||
Query A: total_requests{job="api"}
|
||||
Query B: error_requests{job="api"}
|
||||
Expression C (Math): $B / $A * 100
|
||||
Display C as percentage
|
||||
```
|
||||
|
||||
### Important query variables
|
||||
These variables are available in queries:
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `$__interval` | Auto-calculated interval based on time range and resolution |
|
||||
| `$__rate_interval` | Interval suitable for rate() functions (at least 4x scrape interval) |
|
||||
| `$__from` | Start of the current time range (ms epoch) |
|
||||
| `$__to` | End of the current time range (ms epoch) |
|
||||
| `$__range` | Duration of current time range (e.g., "6h") |
|
||||
| `$__range_s` | Duration in seconds |
|
||||
| `$__range_ms` | Duration in milliseconds |
|
||||
|
||||
Example PromQL using interval variable:
|
||||
```promql
|
||||
rate(http_requests_total[${__rate_interval}])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Panel Inspect
|
||||
|
||||
Click panel menu (3-dot) > **Inspect** to access:
|
||||
- **Data**: Raw table view of the data powering the panel
|
||||
- **Stats**: Query performance (time, row count)
|
||||
- **JSON**: Panel JSON model
|
||||
- **Query**: Equivalent of query inspector showing raw query and response
|
||||
|
||||
---
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. Limit max data points to avoid over-fetching
|
||||
2. Use recording rules in Prometheus for expensive queries
|
||||
3. Set longer min intervals for historical dashboards
|
||||
4. Use `$__interval` variable in queries to align with time resolution
|
||||
5. Use `$__rate_interval` instead of hardcoded intervals for rate() queries
|
||||
6. Avoid too many panels on one dashboard (aim for <20)
|
||||
Reference in New Issue
Block a user