moritz
/
infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
infrapuzzle/k8s/monitoring/DASHBOARD_CREATION.md

12 KiB
Raw Blame History

DASHBOARD_CREATION.md — How to Create & Update Grafana Dashboards

Linked from: AGENTS.md

This document captures the exact process used to create the Grafana dashboards in this folder, including all pitfalls encountered and lessons learned. Use this as a reference when creating new dashboards or updating existing ones.

Step-by-Step Dashboard Creation Process

Step 1: Discovery — Find Available Metrics

Before creating any dashboard, discover what data actually exists:

1a. Query Home Assistant for Entity Metadata

# Get all entities with their friendly names, units, and device classes
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
  https://hass.moritzgraf.de/api/states | \
  jq '[.[] | select(.attributes.unit_of_measurement != null) | {
    entity_id,
    unit: .attributes.unit_of_measurement,
    friendly_name: (.attributes.friendly_name // ""),
    device_class: (.attributes.device_class // "")
  }] | group_by(.unit)'

Or use the helper script: ./scripts/discover-home-assistant.sh

1b. Query InfluxDB for Measurements and Schema

# Port-forward to InfluxDB (no ingress configured)
kubectl -n influxdb port-forward svc/influxdb-influxdb2 8086:80

# List measurements with data in last 6 months
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "default")
      |> range(start: -6mo)
      |> keep(columns: ["_measurement"])
      |> distinct()
      |> sort()'

# For a specific measurement, get fields and tags
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'import "influxdata/influxdb/schema"
      schema.measurementFieldKeys(bucket: "default", measurement: "°C")'

Or use: ./scripts/discover-influxdb.sh

1c. Verify Data Exists

Always test a sample query before building dashboards:

curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "default")
      |> range(start: -1h)
      |> filter(fn: (r) => r._measurement == "°C" and r._field == "value")
      |> limit(n: 5)'

Step 2: Discover Grafana Datasource UIDs

Datasource UIDs change when Grafana is redeployed. Always discover the current UIDs:

kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- \
  curl -s -u "admin:<GRAFANA_PASSWORD>" \
  http://localhost:3000/api/datasources | \
  jq '.[] | {name, uid, type}'

The Grafana admin password is in prometheus-operator.secret.yml under grafana.adminPassword.

Step 3: Select the Right Entities

Important: Not all sensors of the same type are equal. For example:

  • Use Wandthermostat (wall thermostat) entities for temperature/humidity, not raw sensor entities. Wandthermostats provide more reliable, calibrated readings.
  • Query Home Assistant to filter by entity name pattern or device type.
# Find all Wandthermostat temperature sensors
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
  https://hass.moritzgraf.de/api/states | \
  jq '[.[] | select(.attributes.unit_of_measurement == "°C" and
    (.entity_id | test("wandthermostat|thermostat"; "i")))]'

Step 4: Create the Dashboard ConfigMap YAML

Create a new file in dashboards/ following this template:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-<name>
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_dashboard_folder: "🏠 Home"
data:
  <name>.json: |
    {
      "uid": "<unique-uid>",
      "title": "<Dashboard Title>",
      "tags": ["home", "<category>", "auto-generated"],
      "timezone": "browser",
      "schemaVersion": 39,
      "version": 1,
      "refresh": "5m",
      "editable": false,
      "graphTooltip": 1,
      "time": { "from": "now-24h", "to": "now" },
      "panels": [ ... ]
    }

Critical JSON Structure Rules

✅ CORRECT (top-level properties, no wrapper):

{
  "uid": "my-dashboard",
  "title": "My Dashboard",
  "panels": [...]
}

❌ WRONG (wrapped in "dashboard" key — causes "Dashboard title cannot be empty"):

{
  "dashboard": {
    "uid": "my-dashboard",
    "title": "My Dashboard",
    "panels": [...]
  }
}

The "dashboard" wrapper is only used in the Grafana HTTP API (/api/dashboards/db). For file-based provisioning via the sidecar, properties must be at the top level.

Datasource Reference

✅ CORRECT (by actual UID):

"datasource": { "type": "influxdb", "uid": "P2AB959DC95E5519F" }

❌ WRONG (by name — won't resolve):

"datasource": { "type": "influxdb", "uid": "InfluxDB Home Assistant" }

Flux Query Template

Exact match on entity_id:

from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Regex match on entity_id (for panels showing multiple entities matching a pattern):

from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id =~ /^home_/)
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Current value (for stat panels showing last reading):

from(bucket: "default")
  |> range(start: -15m)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
  |> last()
  |> yield(name: "mean")

Key points:

  • Use bracket notation r["_measurement"], r["_field"] for fields starting with underscore.
  • entity_id in InfluxDB has NO sensor. prefix — HA stores domain in a separate domain tag. Use home_wohnzimmer_temperature, not sensor.home_wohnzimmer_temperature.
  • Always use v.timeRangeStart, v.timeRangeStop, v.windowPeriod — never hardcode time values.
  • Always end with |> yield(name: "mean") — required for Grafana to parse Flux results.
  • Use regex =~ /pattern/ for multi-entity panels (e.g., =~ /^home_/ matches all home entities).
  • Always filter r["_field"] == "value" — InfluxDB stores metadata strings in other fields.

Step 5: Deploy

# Apply all dashboards
kubectl apply -f dashboards/

# Apply a single dashboard
kubectl apply -f dashboards/grafana-dashboard-<name>.yaml

Step 6: Verify

# Check ConfigMaps are labeled correctly
kubectl get configmaps -n monitoring -l grafana_dashboard=1

# Check sidecar logs (container name is grafana-sc-dashboard, singular!)
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
  -c grafana-sc-dashboard --tail=20

# Check Grafana main logs for errors
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
  -c grafana --tail=30 | grep -i "error\|failed"

Update Process (Modifying an Existing Dashboard)

  1. Edit the YAML file in dashboards/
  2. Increment the version field in the dashboard JSON
  3. Run kubectl apply -f dashboards/grafana-dashboard-<name>.yaml
  4. The sidecar will detect the change and reload within ~60 seconds
  5. Verify in Grafana UI

Never edit dashboards in the Grafana UI — they are editable: false. All changes must go through the YAML files.

Removing a Dashboard

kubectl delete configmap -n monitoring grafana-dashboard-<name>

The sidecar will remove it from Grafana within ~60 seconds.

Pitfalls Encountered in This Conversation

Pitfall 1: Entity IDs lack sensor. prefix in InfluxDB

Problem: Used sensor.home_wohnzimmer_temperature in queries — no data appeared. Home Assistant stores the domain (sensor) in a separate domain tag, not in entity_id. Fix: Use home_wohnzimmer_temperature (without prefix). Always query InfluxDB directly to verify entity_id format before building queries.

Pitfall 2: Hardcoded time ranges

Problem: Used range(start: -24h) and aggregateWindow(every: 5m) — dashboard time picker had no effect, data not loaded properly. Fix: Always use v.timeRangeStart, v.timeRangeStop, v.windowPeriod. These Grafana variables adapt to the dashboard time picker.

Pitfall 3: Missing yield()

Problem: Flux queries missing yield(name: "mean") — Grafana couldn't parse results. Fix: Always end Flux queries with |> yield(name: "mean").

Pitfall 4: Dot notation for underscore fields

Problem: Used r._measurement and r._field — Flux may not resolve fields starting with _ via dot notation. Fix: Use bracket notation: r["_measurement"], r["_field"].

Pitfall 5: Datasource UID vs Name

Problem: Used datasource name as UID ("uid": "InfluxDB Home Assistant"). Dashboards loaded but showed "no data". Fix: Discover actual UID via kubectl exec + Grafana API. Current: P2AB959DC95E5519F.

Pitfall 6: "dashboard" JSON Wrapper

Problem: Dashboard JSON wrapped in {"dashboard": {...}}. Grafana showed "Dashboard title cannot be empty". Fix: Top-level properties only. The "dashboard" wrapper is for HTTP API, not file provisioning.

Pitfall 7: Adding uid to Helm Datasource Config

Problem: Added uid: influxdb-ha-flux to helm values — Grafana pod crashed. Fix: Let Grafana auto-generate UIDs. Never add uid to existing datasources.

Pitfall 8: Multi-entity panels using or in filter

Problem: Used filter(fn: (r) => r.entity_id == "A" or r.entity_id == "B") — Flux or inside a single filter call doesn't reliably work. Fix: Use regex: filter(fn: (r) => r.entity_id =~ /^home_/) for matching multiple entities.

Pitfall 9: Wrong Sensor Selection

Problem: Used raw temperature sensors instead of Wandthermostat sensors. Fix: Query HA to filter entities with wandthermostat or thermostat in entity_id.

Pitfall 10: Sidecar Container Name

Problem: kubectl logs -c grafana-sc-dashboards (plural). Fix: Container name is grafana-sc-dashboard (singular).

Dashboard Design Guidelines

  1. Use semantic thresholds with colors: Blue=cold, green=comfort, orange=warm, red=hot for temperature; reverse for grid import (green=low, red=high).
  2. Use Stat panels with sparklines for current values — they show the 24h mini-trend.
  3. Use BarGauge for day comparisons (today vs yesterday vs avg).
  4. Use Gauge for ranged values (battery SoC 0-100%, cloud cover 0-100%).
  5. Color-code panel backgrounds with "colorMode": "background" for at-a-glance status.
  6. Set "editable": false for strict GitOps.
  7. Use emoji in dashboard and panel titles for visual appeal (🏠 ⚡ 🔥 🌡️ 💧).
  8. Add "auto-generated" tag to distinguish from manually created dashboards.
  9. Include calcs in legend (mean, max, min, lastNotNull) for quick statistics.
  10. Use "spanNulls": true in timeseries to handle gaps in sensor data.

InfluxDB Measurement Reference

Measurement Unit Typical Entities
°C celsius Wandthermostat temperatures, outdoor temp, battery temp
% percent Humidity, battery levels, heating valve position, cloud cover
W watt Solar production, self-consumption, grid import/export, house consumption
kWh kilowatt-hour Daily energy totals (production, consumption, import, export)
m³ cubic meter Gas volume counter
m³/h cubic meter/hour Current gas flow rate
hPa hectopascal Barometric pressure
km/h kilometer/hour Wind speed
mm/h millimeter/hour Precipitation rate
EUR euro Gas cost, electricity cost