moritz
/
infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
infrapuzzle/k8s/monitoring/AGENTS.md

8.2 KiB
Raw Blame History

AGENTS.md — Monitoring Folder Guide for AI Agents

Overview

This folder manages the kube-prometheus-stack deployment and Grafana dashboards as code for the haumdaucher Kubernetes cluster. The monitoring stack runs in the monitoring namespace.

Folder Structure

monitoring/
├── AGENTS.md                          ← You are here
├── README.md                          ← Human-oriented deployment guide
├── DASHBOARD_CREATION.md              ← How to create/update dashboards (read this!)
├── prometheus-operator.secret.yml     ← Helm values for kube-prometheus-stack
├── alertmanagerconfig.secret.yaml     ← Alertmanager config (Telegram alerts)
├── servicemonitor.secret.yml          ← ServiceMonitor examples (commented out)
├── tankerkoenig.yml                   ← Tankerkoenig fuel price exporter
├── dashboards/                        ← Grafana dashboards as ConfigMaps
│   ├── grafana-dashboard-home-climate.yaml
│   ├── grafana-dashboard-energy-monitor.yaml
│   ├── grafana-dashboard-heating-gas.yaml
│   └── grafana-dashboard-outdoor-weather.yaml
└── scripts/                           ← Discovery & helper scripts
    ├── discover-influxdb.sh           ← Query InfluxDB schema
    └── discover-home-assistant.sh     ← Query Home Assistant entities

Key Infrastructure Context

Namespaces & Services

Namespace Service Purpose
monitoring kube-prometheus-stack Prometheus, Alertmanager, Grafana
influxdb InfluxDB2 Time-series storage for Home Assistant sensor data
home-assistant Home Assistant Smart home hub, writes sensor data to InfluxDB

Data Flow

Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
                                    ↓
                         Grafana (Flux queries via datasource)

Grafana Datasources

Name UID Type URL
InfluxDB Home Assistant P2AB959DC95E5519F influxdb (Flux) http://influxdb-influxdb2.influxdb.svc.cluster.local:80
InfluxDB Home Assistant InfluxQL P86455D1023ECC470 influxdb (InfluxQL) same
Prometheus prometheus prometheus internal

⚠️ IMPORTANT: The datasource UIDs above are the current values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources before creating new dashboards.

Grafana Admin Credentials

  • URL: https://grafana.haumdaucher.de
  • User: admin
  • Password: stored in prometheus-operator.secret.yml under grafana.adminPassword

Home Assistant API

  • URL: https://hass.moritzgraf.de
  • Token: stored in ../home-assistant/tmp_long_lived_token.secret.yml

InfluxDB API

  • Internal URL: http://influxdb-influxdb2.influxdb.svc.cluster.local:80
  • Org: influxdata
  • Bucket: default
  • Token: stored in prometheus-operator.secret.yml under grafana.additionalDataSources[0].secureJsonData.token

Dashboard Architecture

Dashboards are managed as Kubernetes ConfigMaps with the label grafana_dashboard: "1". The kube-prometheus-stack's Grafana Dashboard Sidecar (grafana-sc-dashboard container) watches for these ConfigMaps and auto-loads them into Grafana.

Key Design Decisions

  1. One file per dashboard — Each ConfigMap YAML contains the dashboard JSON embedded directly in data. No separate JSON files (avoids duplication).
  2. Strict GitOps — Dashboards are editable: false in Grafana UI. All changes must be made to the YAML files and re-applied.
  3. Namespace-scoped — Sidecar only watches the monitoring namespace (searchNamespace: monitoring).
  4. Folder annotation — Use grafana_dashboard_folder annotation to organize dashboards in Grafana folders.

ConfigMap Template

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-<name>
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_dashboard_folder: "🏠 Home"
data:
  <name>.json: |
    {
      "uid": "<unique-uid>",
      "title": "<Dashboard Title>",
      ...
    }

InfluxDB Data Schema

Home Assistant writes sensor data to InfluxDB with this structure:

  • Measurement name = unit of measurement (e.g., °C, %, W, kWh, m³, hPa, km/h, mm/h)
  • Fields: value (numeric), plus metadata fields (friendly_name_str, device_class_str, id_str, etc.)
  • Tags: entity_id (e.g., home_wohnzimmer_temperature — NOTE: no sensor. prefix!), domain (e.g., sensor)

Flux Query Pattern (MUST follow this exactly)

Exact match on entity_id:

from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "°C")
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "home_wandthermostat_bad_temperature")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Regex match on entity_id (use =~ operator with /pattern/):

from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "°C")
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id =~ /^home_/)
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")

Critical rules:

  • Always use bracket notation: r["_measurement"], r["_field"] (fields starting with _)
  • Always use dynamic time: range(start: v.timeRangeStart, stop: v.timeRangeStop) and aggregateWindow(every: v.windowPeriod, ...)
  • Always end with |> yield(name: "mean")
  • entity_id in InfluxDB does NOT have the sensor. prefix — use home_wohnzimmer_temperature, NOT sensor.home_wohnzimmer_temperature
  • For "current value" stat panels, use range(start: -15m) with last() instead of aggregateWindow
  • Regex operator is =~ with /pattern/ syntax (e.g., =~ /^home_/ matches all entities starting with home_)

Common Pitfalls (Lessons Learned)

  1. Entity IDs have no sensor. prefix in InfluxDB — HA stores domain separately. Use home_wohnzimmer_temperature, not sensor.home_wohnzimmer_temperature. This is the #1 cause of "no data".
  2. Dashboard JSON must NOT be wrapped in "dashboard" key — Grafana file provisioning expects top-level properties (uid, title, panels, etc.). The "dashboard" wrapper is only used in the HTTP API.
  3. Datasource reference must use actual UID — Discover via kubectl exec + Grafana API. Current UID: P2AB959DC95E5519F.
  4. Don't add uid to helm datasource config — Adding uid to existing datasources in additionalDataSources causes Grafana provisioning to fail with "data source not found". Let UIDs be auto-generated.
  5. Always use bracket notation for _measurement and _field — Use r["_measurement"] and r["_field"], not dot notation. Fields starting with underscore may not resolve with dot notation in Flux.
  6. Always use Grafana time variables — Never hardcode range(start: -24h) or aggregateWindow(every: 5m). Always use v.timeRangeStart, v.timeRangeStop, v.windowPeriod.
  7. Always add yield(name: "mean") — Required for Grafana to properly parse Flux results.
  8. InfluxDB has multiple fields per measurement — Always filter r["_field"] == "value" to get numeric data.
  9. Sidecar container name is grafana-sc-dashboard (singular), not grafana-sc-dashboards.
  10. Use Wandthermostat entities for temperature/humidity — Query HA to filter by device type. Wandthermostat entities have wandthermostat or thermostat in their entity_id.

Related Files Outside This Folder

  • ../home-assistant/home-assistant.secret.yaml — HA helm values (InfluxDB integration config)
  • ../home-assistant/tmp_long_lived_token.secret.yml — HA API token for discovery
  • ../influxdb/influxdb2.secret.yml — InfluxDB helm values
  • ../../kochbuch/plans/grafana-dashboards-as-code.md — Original architecture plan