infrapuzzle/k8s/monitoring/DASHBOARD_CREATION.md

# DASHBOARD_CREATION.md — How to Create & Update Grafana Dashboards

> **Linked from**: [AGENTS.md](AGENTS.md)

This document captures the exact process used to create the Grafana dashboards in this folder, including all pitfalls encountered and lessons learned. Use this as a reference when creating new dashboards or updating existing ones.

---

## Step-by-Step Dashboard Creation Process

### Step 1: Discovery — Find Available Metrics

Before creating any dashboard, discover what data actually exists:

#### 1a. Query Home Assistant for Entity Metadata

```bash
# Get all entities with their friendly names, units, and device classes
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
  https://hass.moritzgraf.de/api/states | \
  jq '[.[] | select(.attributes.unit_of_measurement != null) | {
    entity_id,
    unit: .attributes.unit_of_measurement,
    friendly_name: (.attributes.friendly_name // ""),
    device_class: (.attributes.device_class // "")
  }] | group_by(.unit)'
```

Or use the helper script: `./scripts/discover-home-assistant.sh`

#### 1b. Query InfluxDB for Measurements and Schema

```bash
# Port-forward to InfluxDB (no ingress configured)
kubectl -n influxdb port-forward svc/influxdb-influxdb2 8086:80

# List measurements with data in last 6 months
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "default")
      |> range(start: -6mo)
      |> keep(columns: ["_measurement"])
      |> distinct()
      |> sort()'

# For a specific measurement, get fields and tags
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'import "influxdata/influxdb/schema"
      schema.measurementFieldKeys(bucket: "default", measurement: "°C")'
```

Or use: `./scripts/discover-influxdb.sh`

#### 1c. Verify Data Exists

Always test a sample query before building dashboards:

```bash
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
  -H "Authorization: Token <INFLUXDB_TOKEN>" \
  -H "Content-Type: application/vnd.flux" \
  -d 'from(bucket: "default")
      |> range(start: -1h)
      |> filter(fn: (r) => r._measurement == "°C" and r._field == "value")
      |> limit(n: 5)'
```

### Step 2: Discover Grafana Datasource UIDs

Datasource UIDs change when Grafana is redeployed. Always discover the current UIDs:

```bash
kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- \
  curl -s -u "admin:<GRAFANA_PASSWORD>" \
  http://localhost:3000/api/datasources | \
  jq '.[] | {name, uid, type}'
```

The Grafana admin password is in [`prometheus-operator.secret.yml`](prometheus-operator.secret.yml) under `grafana.adminPassword`.

### Step 3: Select the Right Entities

**Important**: Not all sensors of the same type are equal. For example:
- Use **Wandthermostat** (wall thermostat) entities for temperature/humidity, not raw sensor entities. Wandthermostats provide more reliable, calibrated readings.
- Query Home Assistant to filter by entity name pattern or device type.

```bash
# Find all Wandthermostat temperature sensors
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
  https://hass.moritzgraf.de/api/states | \
  jq '[.[] | select(.attributes.unit_of_measurement == "°C" and
    (.entity_id | test("wandthermostat|thermostat"; "i")))]'
```

### Step 4: Create the Dashboard ConfigMap YAML

Create a new file in `dashboards/` following this template:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-<name>
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_dashboard_folder: "🏠 Home"
data:
  <name>.json: |
    {
      "uid": "<unique-uid>",
      "title": "<Dashboard Title>",
      "tags": ["home", "<category>", "auto-generated"],
      "timezone": "browser",
      "schemaVersion": 39,
      "version": 1,
      "refresh": "5m",
      "editable": false,
      "graphTooltip": 1,
      "time": { "from": "now-24h", "to": "now" },
      "panels": [ ... ]
    }    
```

#### Critical JSON Structure Rules

**✅ CORRECT** (top-level properties, no wrapper):
```json
{
  "uid": "my-dashboard",
  "title": "My Dashboard",
  "panels": [...]
}
```

**❌ WRONG** (wrapped in "dashboard" key — causes "Dashboard title cannot be empty"):
```json
{
  "dashboard": {
    "uid": "my-dashboard",
    "title": "My Dashboard",
    "panels": [...]
  }
}
```

The `"dashboard"` wrapper is **only** used in the Grafana HTTP API (`/api/dashboards/db`). For file-based provisioning via the sidecar, properties must be at the top level.

#### Datasource Reference

**✅ CORRECT** (by actual UID):
```json
"datasource": { "type": "influxdb", "uid": "P2AB959DC95E5519F" }
```

**❌ WRONG** (by name — won't resolve):
```json
"datasource": { "type": "influxdb", "uid": "InfluxDB Home Assistant" }
```

#### Flux Query Template

**Exact match on entity_id:**
```flux
from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")
```

**Regex match on entity_id** (for panels showing multiple entities matching a pattern):
```flux
from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id =~ /^home_/)
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")
```

**Current value** (for stat panels showing last reading):
```flux
from(bucket: "default")
  |> range(start: -15m)
  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
  |> last()
  |> yield(name: "mean")
```

**Key points**:
- Use bracket notation `r["_measurement"]`, `r["_field"]` for fields starting with underscore.
- **entity_id in InfluxDB has NO `sensor.` prefix** — HA stores domain in a separate `domain` tag. Use `home_wohnzimmer_temperature`, not `sensor.home_wohnzimmer_temperature`.
- Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod` — never hardcode time values.
- Always end with `|> yield(name: "mean")` — required for Grafana to parse Flux results.
- Use regex `=~ /pattern/` for multi-entity panels (e.g., `=~ /^home_/` matches all home entities).
- Always filter `r["_field"] == "value"` — InfluxDB stores metadata strings in other fields.

### Step 5: Deploy

```bash
# Apply all dashboards
kubectl apply -f dashboards/

# Apply a single dashboard
kubectl apply -f dashboards/grafana-dashboard-<name>.yaml
```

### Step 6: Verify

```bash
# Check ConfigMaps are labeled correctly
kubectl get configmaps -n monitoring -l grafana_dashboard=1

# Check sidecar logs (container name is grafana-sc-dashboard, singular!)
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
  -c grafana-sc-dashboard --tail=20

# Check Grafana main logs for errors
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
  -c grafana --tail=30 | grep -i "error\|failed"
```

---

## Update Process (Modifying an Existing Dashboard)

1. Edit the YAML file in `dashboards/`
2. Increment the `version` field in the dashboard JSON
3. Run `kubectl apply -f dashboards/grafana-dashboard-<name>.yaml`
4. The sidecar will detect the change and reload within ~60 seconds
5. Verify in Grafana UI

**Never edit dashboards in the Grafana UI** — they are `editable: false`. All changes must go through the YAML files.

---

## Removing a Dashboard

```bash
kubectl delete configmap -n monitoring grafana-dashboard-<name>
```

The sidecar will remove it from Grafana within ~60 seconds.

---

## Pitfalls Encountered in This Conversation

### Pitfall 1: Entity IDs lack `sensor.` prefix in InfluxDB
**Problem**: Used `sensor.home_wohnzimmer_temperature` in queries — no data appeared. Home Assistant stores the domain (`sensor`) in a separate `domain` tag, not in `entity_id`.
**Fix**: Use `home_wohnzimmer_temperature` (without prefix). Always query InfluxDB directly to verify entity_id format before building queries.

### Pitfall 2: Hardcoded time ranges
**Problem**: Used `range(start: -24h)` and `aggregateWindow(every: 5m)` — dashboard time picker had no effect, data not loaded properly.
**Fix**: Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`. These Grafana variables adapt to the dashboard time picker.

### Pitfall 3: Missing `yield()`
**Problem**: Flux queries missing `yield(name: "mean")` — Grafana couldn't parse results.
**Fix**: Always end Flux queries with `|> yield(name: "mean")`.

### Pitfall 4: Dot notation for underscore fields
**Problem**: Used `r._measurement` and `r._field` — Flux may not resolve fields starting with `_` via dot notation.
**Fix**: Use bracket notation: `r["_measurement"]`, `r["_field"]`.

### Pitfall 5: Datasource UID vs Name
**Problem**: Used datasource name as UID (`"uid": "InfluxDB Home Assistant"`). Dashboards loaded but showed "no data".
**Fix**: Discover actual UID via `kubectl exec` + Grafana API. Current: `P2AB959DC95E5519F`.

### Pitfall 6: `"dashboard"` JSON Wrapper
**Problem**: Dashboard JSON wrapped in `{"dashboard": {...}}`. Grafana showed "Dashboard title cannot be empty".
**Fix**: Top-level properties only. The `"dashboard"` wrapper is for HTTP API, not file provisioning.

### Pitfall 7: Adding `uid` to Helm Datasource Config
**Problem**: Added `uid: influxdb-ha-flux` to helm values — Grafana pod crashed.
**Fix**: Let Grafana auto-generate UIDs. Never add `uid` to existing datasources.

### Pitfall 8: Multi-entity panels using `or` in filter
**Problem**: Used `filter(fn: (r) => r.entity_id == "A" or r.entity_id == "B")` — Flux `or` inside a single filter call doesn't reliably work.
**Fix**: Use regex: `filter(fn: (r) => r.entity_id =~ /^home_/)` for matching multiple entities.

### Pitfall 9: Wrong Sensor Selection
**Problem**: Used raw temperature sensors instead of Wandthermostat sensors.
**Fix**: Query HA to filter entities with `wandthermostat` or `thermostat` in entity_id.

### Pitfall 10: Sidecar Container Name
**Problem**: `kubectl logs -c grafana-sc-dashboards` (plural).
**Fix**: Container name is `grafana-sc-dashboard` (singular).

---

## Dashboard Design Guidelines

1. **Use semantic thresholds with colors**: Blue=cold, green=comfort, orange=warm, red=hot for temperature; reverse for grid import (green=low, red=high).
2. **Use Stat panels with sparklines** for current values — they show the 24h mini-trend.
3. **Use BarGauge for day comparisons** (today vs yesterday vs avg).
4. **Use Gauge for ranged values** (battery SoC 0-100%, cloud cover 0-100%).
5. **Color-code panel backgrounds** with `"colorMode": "background"` for at-a-glance status.
6. **Set `"editable": false`** for strict GitOps.
7. **Use emoji in dashboard and panel titles** for visual appeal (🏠 ⚡ 🔥 🌡️ 💧).
8. **Add `"auto-generated"` tag** to distinguish from manually created dashboards.
9. **Include calcs in legend** (`mean`, `max`, `min`, `lastNotNull`) for quick statistics.
10. **Use `"spanNulls": true`** in timeseries to handle gaps in sensor data.

---

## InfluxDB Measurement Reference

| Measurement | Unit | Typical Entities |
|-------------|------|------------------|
| `°C` | celsius | Wandthermostat temperatures, outdoor temp, battery temp |
| `%` | percent | Humidity, battery levels, heating valve position, cloud cover |
| `W` | watt | Solar production, self-consumption, grid import/export, house consumption |
| `kWh` | kilowatt-hour | Daily energy totals (production, consumption, import, export) |
| `m³` | cubic meter | Gas volume counter |
| `m³/h` | cubic meter/hour | Current gas flow rate |
| `hPa` | hectopascal | Barometric pressure |
| `km/h` | kilometer/hour | Wind speed |
| `mm/h` | millimeter/hour | Precipitation rate |
| `EUR` | euro | Gas cost, electricity cost |