feat: add grafana dashboarding and promql agent skills
This commit is contained in:
Moritz Graf
@@ -0,0 +1,462 @@
|
||||
# Grafana Data Sources - Detailed Reference
|
||||
|
||||
## Overview
|
||||
|
||||
Data sources are connections between Grafana and the systems storing your data. Grafana ships with built-in support for many popular data sources and supports additional sources via plugins.
|
||||
|
||||
- Only **Organization Admins** can add or remove data sources
|
||||
- Each data source has its own query editor
|
||||
- Data sources can be used in: Dashboard panels, Explore, Alert rules, Annotations
|
||||
|
||||
---
|
||||
|
||||
## Managing Data Sources
|
||||
|
||||
### Add a data source
|
||||
1. Go to **Connections > Data sources** (or **Configuration > Data sources** in older versions)
|
||||
2. Click **Add new data source**
|
||||
3. Search for and select the data source type
|
||||
4. Fill in connection details (URL, credentials, etc.)
|
||||
5. Click **Save & Test** to verify the connection
|
||||
|
||||
### Data source settings (common to all types)
|
||||
| Setting | Description |
|
||||
|---------|-------------|
|
||||
| Name | Display name used in dashboards |
|
||||
| Default | If checked, pre-selected in new panels |
|
||||
| HTTP URL | The endpoint for the data source server |
|
||||
| Auth | Basic auth, TLS, bearer token, API key options |
|
||||
|
||||
---
|
||||
|
||||
## Prometheus
|
||||
|
||||
Native support - no plugin required.
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://prometheus:9090
|
||||
HTTP method: POST (recommended, supports longer queries)
|
||||
```
|
||||
|
||||
### Key options
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| Scrape interval | Default: 15s - should match your Prometheus scrape interval |
|
||||
| Query timeout | Default: 60s |
|
||||
| Exemplars | Enable to link metrics to traces |
|
||||
| Ruler URL | For Prometheus-managed alert rules |
|
||||
|
||||
### Query editor
|
||||
- **Metrics browser**: Browse available metrics with autocomplete
|
||||
- **Query type**: Range query (time series) or Instant query (single value)
|
||||
- **Legend**: Customize series labels using `{{label_name}}` syntax
|
||||
|
||||
### Template variables with Prometheus
|
||||
```
|
||||
# Variable type: Query
|
||||
# Query examples:
|
||||
label_values(metric_name, label_name) # All values of a label for a metric
|
||||
label_values(label_name) # All values across all metrics
|
||||
metrics(prefix) # All metric names matching prefix
|
||||
query_result(promql_expression) # PromQL result as variable values
|
||||
```
|
||||
|
||||
### Exemplars
|
||||
When enabled, Prometheus exemplars link high-cardinality trace IDs to metric data points. Requires a Tempo data source for trace drill-through.
|
||||
|
||||
---
|
||||
|
||||
## Loki (Log Aggregation)
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://loki:3100
|
||||
Maximum lines: 1000
|
||||
```
|
||||
|
||||
### Derived fields
|
||||
Extract values from log lines and link to other systems.
|
||||
|
||||
Example - extract trace ID and link to Tempo:
|
||||
```
|
||||
Name: TraceID
|
||||
Regex: traceID=(\w+)
|
||||
URL: (internal link to Tempo data source)
|
||||
```
|
||||
|
||||
### Query editor (LogQL)
|
||||
```logql
|
||||
# Basic log stream selector
|
||||
{job="nginx", namespace="production"}
|
||||
|
||||
# Filter by text
|
||||
{job="nginx"} |= "error"
|
||||
{job="nginx"} != "debug"
|
||||
|
||||
# Regex filter
|
||||
{job="nginx"} |~ "status=5\d\d"
|
||||
|
||||
# Parse and filter structured logs
|
||||
{job="api"} | json | level="error"
|
||||
{job="api"} | logfmt | duration > 1s
|
||||
|
||||
# Metrics from logs
|
||||
rate({job="nginx"} |= "error" [5m])
|
||||
sum(rate({job="nginx"}[5m])) by (status_code)
|
||||
```
|
||||
|
||||
### Template variables with Loki
|
||||
```
|
||||
label_names() # All label names
|
||||
label_values(label_name) # All values for a label
|
||||
label_values({job="nginx"}, pod) # Label values filtered by stream selector
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Tempo (Distributed Tracing)
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://tempo:3200
|
||||
```
|
||||
|
||||
### Key options
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| Trace to logs | Link traces to Loki logs via trace ID |
|
||||
| Trace to metrics | Link trace spans to Prometheus metrics |
|
||||
| Service graph | Enable service dependency graph |
|
||||
| Node graph | Show trace as node graph |
|
||||
|
||||
### TraceQL (query language)
|
||||
```traceql
|
||||
# Find traces with error spans
|
||||
{status=error}
|
||||
|
||||
# Filter by service and duration
|
||||
{.service.name="frontend" && duration > 1s}
|
||||
|
||||
# Structural queries
|
||||
{.http.url=~"/api/.*"} >> {status=error}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Alertmanager
|
||||
|
||||
For connecting to an external Alertmanager.
|
||||
|
||||
```ini
|
||||
URL: http://alertmanager:9093
|
||||
# Implementation: Prometheus, Mimir, or Cortex
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Elasticsearch / OpenSearch
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
URL: http://elasticsearch:9200
|
||||
Index name: logs-*
|
||||
Time field name: @timestamp
|
||||
Elasticsearch version: 8.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MySQL
|
||||
|
||||
Built-in support for MySQL 5.7+ and compatible databases (MariaDB, Percona, Amazon Aurora MySQL, Azure Database for MySQL, Google Cloud SQL MySQL).
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: mysql:3306
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
Max open connections: 100
|
||||
Max idle connections: 100
|
||||
Connection max lifetime: 14400
|
||||
```
|
||||
|
||||
### Time series queries
|
||||
```sql
|
||||
SELECT
|
||||
UNIX_TIMESTAMP(time_col) as time_sec,
|
||||
value_col as value,
|
||||
name_col as metric
|
||||
FROM my_table
|
||||
WHERE $__timeFilter(time_col)
|
||||
ORDER BY time_col ASC
|
||||
```
|
||||
|
||||
### Macros
|
||||
| Macro | Description |
|
||||
|-------|-------------|
|
||||
| `$__time(column)` | Converts column to Unix timestamp |
|
||||
| `$__timeFilter(column)` | Adds WHERE clause for dashboard time range |
|
||||
| `$__timeFrom()` | Start of dashboard time range as Unix timestamp |
|
||||
| `$__timeTo()` | End of dashboard time range as Unix timestamp |
|
||||
| `$__timeGroup(column, interval)` | Groups by time interval |
|
||||
| `$__timeGroupAlias(column, interval)` | As above, aliases as "time" |
|
||||
| `$__unixEpochFilter(column)` | Time filter for Unix epoch columns |
|
||||
| `$__interval` | Auto-calculated interval for current time range |
|
||||
|
||||
### Example time series query
|
||||
```sql
|
||||
SELECT
|
||||
$__timeGroup(created_at, $__interval) AS time,
|
||||
status,
|
||||
count(*) AS cnt
|
||||
FROM orders
|
||||
WHERE $__timeFilter(created_at)
|
||||
GROUP BY 1, 2
|
||||
ORDER BY 1
|
||||
```
|
||||
|
||||
### Annotations
|
||||
```sql
|
||||
SELECT
|
||||
UNIX_TIMESTAMP(time) AS time,
|
||||
title AS text,
|
||||
tags
|
||||
FROM events
|
||||
WHERE $__timeFilter(time)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PostgreSQL
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: postgres:5432
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
SSL mode: disable / require / verify-ca / verify-full
|
||||
```
|
||||
|
||||
### PostgreSQL time series
|
||||
```sql
|
||||
SELECT
|
||||
time_bucket('$__interval', time) AS time,
|
||||
avg(value)
|
||||
FROM metrics
|
||||
WHERE time BETWEEN $__timeFrom() AND $__timeTo()
|
||||
GROUP BY 1
|
||||
ORDER BY 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Microsoft SQL Server
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
Host: sqlserver:1433
|
||||
Database: mydb
|
||||
User: grafana
|
||||
Password: secret
|
||||
Encrypt: false / true / disable
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## InfluxDB
|
||||
|
||||
Supports InfluxDB 1.x (InfluxQL) and InfluxDB 2.x / 3.x (Flux).
|
||||
|
||||
### InfluxDB 1.x (InfluxQL)
|
||||
```ini
|
||||
URL: http://influxdb:8086
|
||||
Database: telegraf
|
||||
```
|
||||
|
||||
Query example:
|
||||
```sql
|
||||
SELECT mean("value") FROM "measurement"
|
||||
WHERE $timeFilter
|
||||
GROUP BY time($interval), "host"
|
||||
```
|
||||
|
||||
### InfluxDB 2.x (Flux)
|
||||
```ini
|
||||
URL: http://influxdb:8086
|
||||
Organization: myorg
|
||||
Token: <api-token>
|
||||
Default bucket: mydata
|
||||
```
|
||||
|
||||
Flux query example:
|
||||
```flux
|
||||
from(bucket: "mydata")
|
||||
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
||||
|> filter(fn: (r) => r._measurement == "cpu")
|
||||
|> filter(fn: (r) => r._field == "usage_idle")
|
||||
|> aggregateWindow(every: v.windowPeriod, fn: mean)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## AWS CloudWatch
|
||||
|
||||
### Configuration
|
||||
```ini
|
||||
# Auth options:
|
||||
# - AWS SDK Default (instance role, ~/.aws/credentials, env vars)
|
||||
# - Access & secret key (explicit credentials)
|
||||
# - Assume role ARN
|
||||
|
||||
Default region: us-east-1
|
||||
```
|
||||
|
||||
Required IAM permissions:
|
||||
- `cloudwatch:GetMetricData`
|
||||
- `cloudwatch:ListMetrics`
|
||||
- `logs:*` (for CloudWatch Logs)
|
||||
|
||||
### CloudWatch Logs Insights
|
||||
```
|
||||
fields @timestamp, @message
|
||||
| filter level = "ERROR"
|
||||
| sort @timestamp desc
|
||||
| limit 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Azure Monitor
|
||||
|
||||
Requires Azure App Registration with:
|
||||
- Tenant ID, Client ID, Client Secret
|
||||
- `Monitoring Reader` role on target subscriptions/resource groups
|
||||
|
||||
---
|
||||
|
||||
## Google Cloud Monitoring
|
||||
|
||||
Uses Google Cloud service account credentials (JSON key file or workload identity).
|
||||
|
||||
---
|
||||
|
||||
## TestData (built-in)
|
||||
|
||||
A built-in data source for generating test/demo data without a real backend.
|
||||
|
||||
Use cases: Testing visualizations, demo dashboards, development without a real data source.
|
||||
Scenarios: Random walk, CSV metric values, streaming data, etc.
|
||||
|
||||
---
|
||||
|
||||
## Graphite
|
||||
|
||||
```ini
|
||||
URL: http://graphite:8080
|
||||
```
|
||||
|
||||
Query examples:
|
||||
```
|
||||
target(servers.*.cpu)
|
||||
averageSeries(servers.*.cpu)
|
||||
groupByNode(servers.*.cpu, 1, 'averageSeries')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Jaeger (Tracing)
|
||||
|
||||
```ini
|
||||
URL: http://jaeger:16686
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Zipkin (Tracing)
|
||||
|
||||
```ini
|
||||
URL: http://zipkin:9411
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pyroscope (Continuous Profiling)
|
||||
|
||||
```ini
|
||||
URL: http://pyroscope:4040
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Source Permissions (Enterprise)
|
||||
|
||||
By default, all org users can query any data source. With RBAC:
|
||||
- Assign specific users/teams to specific data sources
|
||||
- Configure at **Data source settings > Permissions**
|
||||
|
||||
---
|
||||
|
||||
## Provisioning Data Sources (as Code)
|
||||
|
||||
```yaml
|
||||
# /etc/grafana/provisioning/datasources/prometheus.yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
version: 1
|
||||
editable: false
|
||||
jsonData:
|
||||
timeInterval: "15s"
|
||||
queryTimeout: "60s"
|
||||
httpMethod: POST
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
maxLines: 1000
|
||||
derivedFields:
|
||||
- datasourceUid: tempo
|
||||
matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
url: "${__value.raw}"
|
||||
|
||||
- name: MySQL Production
|
||||
type: mysql
|
||||
url: mysql:3306
|
||||
database: mydb
|
||||
user: grafana
|
||||
secureJsonData:
|
||||
password: "$GRAFANA_MYSQL_PASSWORD"
|
||||
jsonData:
|
||||
maxOpenConns: 100
|
||||
maxIdleConns: 100
|
||||
connMaxLifetime: 14400
|
||||
```
|
||||
|
||||
Place YAML files in the provisioning directory; Grafana loads on startup and watches for changes.
|
||||
|
||||
---
|
||||
|
||||
## Plugin Data Sources
|
||||
|
||||
Install additional data sources via:
|
||||
- **Connections > Add new connection** (search the plugin catalog)
|
||||
- `grafana-cli plugins install <plugin-id>`
|
||||
- Docker: set `GF_INSTALL_PLUGINS` environment variable
|
||||
|
||||
Popular plugin data sources:
|
||||
- `grafana-opensearch-datasource` - OpenSearch
|
||||
- `grafana-bigquery-datasource` - Google BigQuery
|
||||
- `grafana-mongodb-datasource` - MongoDB (Enterprise)
|
||||
- `grafana-splunk-datasource` - Splunk
|
||||
- `grafana-datadog-datasource` - Datadog
|
||||
Reference in New Issue
Block a user