moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

refactor(monitoring): rearchitect taupi-fan dashboard & clean up docs

Browse Source 
Dashboard changes:
- Merge 3 separate correlation graphs into one full-width panel
- Left Y axis: temperatures + dewpoints (°C)
- Right Y axis: humidity + mold threshold (%)
- Boolean signals (fan/mold/activation) as colored fill bands
- graphTooltip: 2 for shared crosshair + tooltip
- Add collapsed Deep Dive row with delta graphs
- Collapse BLE diagnostics row

Documentation changes:
- DASHBOARD_CREATION.md: remove content now covered by Grafana skills
  (dashboarding, grafana-oss, promql), eliminate duplicated pitfalls
  section, add Prometheus dashboard guidance, add Grafana unit codes
- AGENTS.md: add skills prerequisite table, update folder structure
  (add taupi-fan.yaml), add Prometheus data flow, remove duplicated
  InfluxDB/Flux/pitfalls sections (canonical in DASHBOARD_CREATION.md)
This commit is contained in:
Moritz Graf
2026-07-03 13:06:25 +02:00
parent 304ca52cca
commit a623ec7b85
3 changed files with 524 additions and 438 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/monitoring/DASHBOARD_CREATION.md
+140 -240
View File
@@ -2,102 +2,79 @@
> **Linked from**: [AGENTS.md](AGENTS.md)
This document captures the exact process used to create the Grafana dashboards in this folder, including all pitfalls encountered and lessons learned. Use this as a reference when creating new dashboards or updating existing ones.
This document captures the project-specific process for creating Grafana dashboards in this folder. For general Grafana dashboard JSON schema, panel types, units, transformations, and API workflows, use the **Grafana skills** (`dashboarding`, `grafana-oss`, `promql`) — they are the authoritative reference for Grafana-native topics.
> [!IMPORTANT]
> **Always read the skills first** before creating a dashboard:
> - **`dashboarding`** — JSON schema, panel types, units, template variables, transformations, links, annotations
> - **`grafana-oss`** — Dashboard provisioning, datasource config, RBAC, plugin provisioning
> - **`promql`** — PromQL query patterns, rate/irate, histograms, recording rules
---
## Step-by-Step Dashboard Creation Process
## Step 1: Identify the Data Source
### Step 1: Discovery — Find Available Metrics
This project has **two** types of dashboard data sources:
Before creating any dashboard, discover what data actually exists:
| Data Source | Type | When to Use | Query Language |
|---|---|---|---|
| Prometheus | `prometheus` | Kubernetes metrics, custom exporters (taupi-fan, tankerkoenig) | PromQL |
| InfluxDB Home Assistant | `influxdb` (Flux) | Home Assistant sensor data (temperature, humidity, energy, etc.) | Flux |
#### 1a. Query Home Assistant for Entity Metadata
**Prometheus dashboards** are simpler — just write PromQL expressions in `targets[].expr`. See the `promql` skill and existing examples like [grafana-dashboard-taupi-fan.yaml](dashboards/grafana-dashboard-taupi-fan.yaml).
```bash
# Get all entities with their friendly names, units, and device classes
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
https://hass.moritzgraf.de/api/states | \
jq '[.[] | select(.attributes.unit_of_measurement != null) | {
entity_id,
unit: .attributes.unit_of_measurement,
friendly_name: (.attributes.friendly_name // ""),
device_class: (.attributes.device_class // "")
}] | group_by(.unit)'
```
**InfluxDB dashboards** require Flux queries with project-specific conventions (see [§ InfluxDB Flux Queries](#influxdb-flux-queries) below).
Or use the helper script: `./scripts/discover-home-assistant.sh`
### Datasource UIDs
#### 1b. Query InfluxDB for Measurements and Schema
```bash
# Port-forward to InfluxDB (no ingress configured)
kubectl -n influxdb port-forward svc/influxdb-influxdb2 8086:80
# List measurements with data in last 6 months
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
-H "Authorization: Token <INFLUXDB_TOKEN>" \
-H "Content-Type: application/vnd.flux" \
-d 'from(bucket: "default")
|> range(start: -6mo)
|> keep(columns: ["_measurement"])
|> distinct()
|> sort()'
# For a specific measurement, get fields and tags
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
-H "Authorization: Token <INFLUXDB_TOKEN>" \
-H "Content-Type: application/vnd.flux" \
-d 'import "influxdata/influxdb/schema"
schema.measurementFieldKeys(bucket: "default", measurement: "°C")'
```
Or use: `./scripts/discover-influxdb.sh`
#### 1c. Verify Data Exists
Always test a sample query before building dashboards:
```bash
curl -s "http://localhost:8086/api/v2/query?org=influxdata" \
-H "Authorization: Token <INFLUXDB_TOKEN>" \
-H "Content-Type: application/vnd.flux" \
-d 'from(bucket: "default")
|> range(start: -1h)
|> filter(fn: (r) => r._measurement == "°C" and r._field == "value")
|> limit(n: 5)'
```
### Step 2: Discover Grafana Datasource UIDs
Datasource UIDs change when Grafana is redeployed. Always discover the current UIDs:
Datasource references in panels require the **actual UID**, not the display name. UIDs can change when Grafana is redeployed. Current known UIDs are documented in [AGENTS.md](AGENTS.md) — always verify before creating a new dashboard:
```bash
kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- \
curl -s -u "admin:<GRAFANA_PASSWORD>" \
http://localhost:3000/api/datasources | \
jq '.[] | {name, uid, type}'
http://localhost:3000/api/datasources | jq '.[] | {name, uid, type}'
```
The Grafana admin password is in [`prometheus-operator.secret.yml`](prometheus-operator.secret.yml) under `grafana.adminPassword`.
### Step 3: Select the Right Entities
---
**Important**: Not all sensors of the same type are equal. For example:
- Use **Wandthermostat** (wall thermostat) entities for temperature/humidity, not raw sensor entities. Wandthermostats provide more reliable, calibrated readings.
- Query Home Assistant to filter by entity name pattern or device type.
## Step 2: Discover Available Metrics
### For Prometheus Dashboards
Query Prometheus directly or use the Grafana Explore UI:
```bash
# Find all Wandthermostat temperature sensors
curl -s -H "Authorization: Bearer <HA_TOKEN>" \
https://hass.moritzgraf.de/api/states | \
jq '[.[] | select(.attributes.unit_of_measurement == "°C" and
(.entity_id | test("wandthermostat|thermostat"; "i")))]'
# Port-forward to Prometheus
kubectl -n monitoring port-forward svc/prometheus-operated 9090
# Browse available metrics
curl -s http://localhost:9090/api/v1/label/__name__/values | jq '.data[]' | grep '<prefix>'
```
### Step 4: Create the Dashboard ConfigMap YAML
### For InfluxDB Dashboards (Home Assistant Data)
Create a new file in `dashboards/` following this template:
Use the helper scripts to discover what data exists:
```bash
# Query Home Assistant for entity metadata
./scripts/discover-home-assistant.sh
# Query InfluxDB for measurements and schema
# (requires port-forward first: kubectl -n influxdb port-forward svc/influxdb-influxdb2 8086:80)
./scripts/discover-influxdb.sh
```
**Sensor selection tip**: Use **Wandthermostat** entities for temperature/humidity (they provide calibrated readings). Filter by `wandthermostat` or `thermostat` in the entity_id.
---
## Step 3: Create the Dashboard ConfigMap
Dashboards are deployed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The Grafana sidecar (`grafana-sc-dashboard` container) watches for these and auto-loads them.
### ConfigMap Template
```yaml
apiVersion: v1
@@ -114,57 +91,94 @@ data:
{
"uid": "<unique-uid>",
"title": "<Dashboard Title>",
"tags": ["home", "<category>", "auto-generated"],
"tags": ["<category>"],
"timezone": "browser",
"schemaVersion": 39,
"version": 1,
"refresh": "5m",
"refresh": "30s",
"editable": false,
"graphTooltip": 1,
"graphTooltip": 2,
"time": { "from": "now-24h", "to": "now" },
"panels": [ ... ]
}
```
#### Critical JSON Structure Rules
> [!CAUTION]
> **No `"dashboard"` wrapper.** The JSON must have `uid`, `title`, `panels` at the top level. The `{"dashboard": {...}}` wrapper is only for the Grafana HTTP API — file provisioning via the sidecar will fail with *"Dashboard title cannot be empty"* if you use it.
**✅ CORRECT** (top-level properties, no wrapper):
```json
{
"uid": "my-dashboard",
"title": "My Dashboard",
"panels": [...]
}
For the full panel JSON schema, units, thresholds, overrides, and transformations, see the **`dashboarding`** skill's `references/json-schema.md`.
### Project-Specific Dashboard Conventions
- `"editable": false` — strict GitOps; never edit in the Grafana UI
- `"graphTooltip": 2` — shared crosshair + tooltip across all panels
- Use emoji in titles for visual appeal (🏠 ⚡ 🔥 🌡️ 💧 💨)
- Include `calcs` in legend: `["lastNotNull", "min", "max", "mean"]`
- Use `"spanNulls": true` to handle gaps in sensor data
- Use semantic threshold colors: blue=cold, green=comfort, orange=warm, red=hot
---
## Step 4: Deploy & Verify
```bash
# Apply
kubectl apply -f dashboards/grafana-dashboard-<name>.yaml
# Verify ConfigMap labeling
kubectl get configmaps -n monitoring -l grafana_dashboard=1
# Check sidecar pickup (container name is singular: grafana-sc-dashboard)
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
-c grafana-sc-dashboard --tail=20
# Check for Grafana errors
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
-c grafana --tail=30 | grep -i "error\|failed"
```
**❌ WRONG** (wrapped in "dashboard" key — causes "Dashboard title cannot be empty"):
```json
{
"dashboard": {
"uid": "my-dashboard",
"title": "My Dashboard",
"panels": [...]
}
}
### Updating an Existing Dashboard
1. Edit the YAML file in `dashboards/`
2. Increment the `version` field in the dashboard JSON
3. `kubectl apply -f dashboards/grafana-dashboard-<name>.yaml`
4. Sidecar detects the change within ~60 seconds
### Removing a Dashboard
```bash
kubectl delete configmap -n monitoring grafana-dashboard-<name>
```
The `"dashboard"` wrapper is **only** used in the Grafana HTTP API (`/api/dashboards/db`). For file-based provisioning via the sidecar, properties must be at the top level.
---
#### Datasource Reference
## InfluxDB Flux Queries
**✅ CORRECT** (by actual UID):
```json
"datasource": { "type": "influxdb", "uid": "P2AB959DC95E5519F" }
```
This section covers the **project-specific Flux conventions** for querying Home Assistant data in InfluxDB. These patterns are unique to this project and not covered by the Grafana skills.
**❌ WRONG** (by name — won't resolve):
```json
"datasource": { "type": "influxdb", "uid": "InfluxDB Home Assistant" }
```
### Data Schema
#### Flux Query Template
Home Assistant writes to InfluxDB with this structure:
**Exact match on entity_id:**
- **Measurement name** = unit of measurement (e.g., `°C`, `%`, `W`, `kWh`)
- **Tags**: `entity_id` (e.g., `home_wohnzimmer_temperature`), `domain` (e.g., `sensor`)
- **Fields**: `value` (numeric), plus metadata strings (`friendly_name_str`, etc.)
### Critical Flux Query Rules
> [!WARNING]
> These rules come from hard-won debugging. Violating any of them will result in "no data" panels.
1. **`entity_id` has NO `sensor.` prefix** — HA stores the domain in a separate `domain` tag. Use `home_wohnzimmer_temperature`, NOT `sensor.home_wohnzimmer_temperature`.
2. **Always use bracket notation** for underscore-prefixed fields: `r["_measurement"]`, `r["_field"]` — dot notation (`r._measurement`) may silently fail.
3. **Always use Grafana time variables** — `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`. Never hardcode `range(start: -24h)`.
4. **Always end with `|> yield(name: "mean")`** — required for Grafana to parse Flux results.
5. **Always filter `r["_field"] == "value"`** — InfluxDB stores metadata strings in other fields.
6. **Use regex for multi-entity panels** — `r.entity_id =~ /^home_/` instead of `or` chains (Flux `or` inside a single filter is unreliable).
### Flux Query Templates
**Time series panel (aggregated):**
```flux
from(bucket: "default")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
@@ -174,17 +188,7 @@ from(bucket: "default")
|> yield(name: "mean")
```
**Regex match on entity_id** (for panels showing multiple entities matching a pattern):
```flux
from(bucket: "default")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
|> filter(fn: (r) => r.entity_id =~ /^home_/)
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```
**Current value** (for stat panels showing last reading):
**Stat panel (current value):**
```flux
from(bucket: "default")
|> range(start: -15m)
@@ -194,133 +198,29 @@ from(bucket: "default")
|> yield(name: "mean")
```
**Key points**:
- Use bracket notation `r["_measurement"]`, `r["_field"]` for fields starting with underscore.
- **entity_id in InfluxDB has NO `sensor.` prefix** — HA stores domain in a separate `domain` tag. Use `home_wohnzimmer_temperature`, not `sensor.home_wohnzimmer_temperature`.
- Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod` — never hardcode time values.
- Always end with `|> yield(name: "mean")` — required for Grafana to parse Flux results.
- Use regex `=~ /pattern/` for multi-entity panels (e.g., `=~ /^home_/` matches all home entities).
- Always filter `r["_field"] == "value"` — InfluxDB stores metadata strings in other fields.
### Step 5: Deploy
```bash
# Apply all dashboards
kubectl apply -f dashboards/
# Apply a single dashboard
kubectl apply -f dashboards/grafana-dashboard-<name>.yaml
**Multi-entity panel (regex):**
```flux
from(bucket: "default")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
|> filter(fn: (r) => r.entity_id =~ /^home_/)
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```
### Step 6: Verify
```bash
# Check ConfigMaps are labeled correctly
kubectl get configmaps -n monitoring -l grafana_dashboard=1
# Check sidecar logs (container name is grafana-sc-dashboard, singular!)
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
-c grafana-sc-dashboard --tail=20
# Check Grafana main logs for errors
kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
-c grafana --tail=30 | grep -i "error\|failed"
```
---
## Update Process (Modifying an Existing Dashboard)
1. Edit the YAML file in `dashboards/`
2. Increment the `version` field in the dashboard JSON
3. Run `kubectl apply -f dashboards/grafana-dashboard-<name>.yaml`
4. The sidecar will detect the change and reload within ~60 seconds
5. Verify in Grafana UI
**Never edit dashboards in the Grafana UI** — they are `editable: false`. All changes must go through the YAML files.
---
## Removing a Dashboard
```bash
kubectl delete configmap -n monitoring grafana-dashboard-<name>
```
The sidecar will remove it from Grafana within ~60 seconds.
---
## Pitfalls Encountered in This Conversation
### Pitfall 1: Entity IDs lack `sensor.` prefix in InfluxDB
**Problem**: Used `sensor.home_wohnzimmer_temperature` in queries — no data appeared. Home Assistant stores the domain (`sensor`) in a separate `domain` tag, not in `entity_id`.
**Fix**: Use `home_wohnzimmer_temperature` (without prefix). Always query InfluxDB directly to verify entity_id format before building queries.
### Pitfall 2: Hardcoded time ranges
**Problem**: Used `range(start: -24h)` and `aggregateWindow(every: 5m)` — dashboard time picker had no effect, data not loaded properly.
**Fix**: Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`. These Grafana variables adapt to the dashboard time picker.
### Pitfall 3: Missing `yield()`
**Problem**: Flux queries missing `yield(name: "mean")` — Grafana couldn't parse results.
**Fix**: Always end Flux queries with `|> yield(name: "mean")`.
### Pitfall 4: Dot notation for underscore fields
**Problem**: Used `r._measurement` and `r._field` — Flux may not resolve fields starting with `_` via dot notation.
**Fix**: Use bracket notation: `r["_measurement"]`, `r["_field"]`.
### Pitfall 5: Datasource UID vs Name
**Problem**: Used datasource name as UID (`"uid": "InfluxDB Home Assistant"`). Dashboards loaded but showed "no data".
**Fix**: Discover actual UID via `kubectl exec` + Grafana API. Current: `P2AB959DC95E5519F`.
### Pitfall 6: `"dashboard"` JSON Wrapper
**Problem**: Dashboard JSON wrapped in `{"dashboard": {...}}`. Grafana showed "Dashboard title cannot be empty".
**Fix**: Top-level properties only. The `"dashboard"` wrapper is for HTTP API, not file provisioning.
### Pitfall 7: Adding `uid` to Helm Datasource Config
**Problem**: Added `uid: influxdb-ha-flux` to helm values — Grafana pod crashed.
**Fix**: Let Grafana auto-generate UIDs. Never add `uid` to existing datasources.
### Pitfall 8: Multi-entity panels using `or` in filter
**Problem**: Used `filter(fn: (r) => r.entity_id == "A" or r.entity_id == "B")` — Flux `or` inside a single filter call doesn't reliably work.
**Fix**: Use regex: `filter(fn: (r) => r.entity_id =~ /^home_/)` for matching multiple entities.
### Pitfall 9: Wrong Sensor Selection
**Problem**: Used raw temperature sensors instead of Wandthermostat sensors.
**Fix**: Query HA to filter entities with `wandthermostat` or `thermostat` in entity_id.
### Pitfall 10: Sidecar Container Name
**Problem**: `kubectl logs -c grafana-sc-dashboards` (plural).
**Fix**: Container name is `grafana-sc-dashboard` (singular).
---
## Dashboard Design Guidelines
1. **Use semantic thresholds with colors**: Blue=cold, green=comfort, orange=warm, red=hot for temperature; reverse for grid import (green=low, red=high).
2. **Use Stat panels with sparklines** for current values — they show the 24h mini-trend.
3. **Use BarGauge for day comparisons** (today vs yesterday vs avg).
4. **Use Gauge for ranged values** (battery SoC 0-100%, cloud cover 0-100%).
5. **Color-code panel backgrounds** with `"colorMode": "background"` for at-a-glance status.
6. **Set `"editable": false`** for strict GitOps.
7. **Use emoji in dashboard and panel titles** for visual appeal (🏠 ⚡ 🔥 🌡️ 💧).
8. **Add `"auto-generated"` tag** to distinguish from manually created dashboards.
9. **Include calcs in legend** (`mean`, `max`, `min`, `lastNotNull`) for quick statistics.
10. **Use `"spanNulls": true`** in timeseries to handle gaps in sensor data.
---
## InfluxDB Measurement Reference
| Measurement | Unit | Typical Entities |
|-------------|------|------------------|
| `°C` | celsius | Wandthermostat temperatures, outdoor temp, battery temp |
| `%` | percent | Humidity, battery levels, heating valve position, cloud cover |
| `W` | watt | Solar production, self-consumption, grid import/export, house consumption |
| `kWh` | kilowatt-hour | Daily energy totals (production, consumption, import, export) |
| `m³` | cubic meter | Gas volume counter |
| `m³/h` | cubic meter/hour | Current gas flow rate |
| `hPa` | hectopascal | Barometric pressure |
| `km/h` | kilometer/hour | Wind speed |
| `mm/h` | millimeter/hour | Precipitation rate |
| `EUR` | euro | Gas cost, electricity cost |
| Measurement | Grafana Unit | Typical Entities |
|---|---|---|
| `°C` | `celsius` | Wandthermostat temperatures, outdoor temp, battery temp |
| `%` | `percent` | Humidity, battery levels, heating valve position, cloud cover |
| `W` | `watt` | Solar production, self-consumption, grid import/export |
| `kWh` | `kwatth` | Daily energy totals (production, consumption, import, export) |
| `m³` | `m3` | Gas volume counter |
| `m³/h` | `m3/h` | Current gas flow rate |
| `hPa` | `pressurehpa` | Barometric pressure |
| `km/h` | `velocitykmh` | Wind speed |
| `mm/h` | `lengthmm` | Precipitation rate |
| `EUR` | `currencyEUR` | Gas cost, electricity cost |