6.0 KiB

Raw Blame History

AGENTS.md — Monitoring Folder Guide for AI Agents

Overview

This folder manages the kube-prometheus-stack deployment and Grafana dashboards as code for the haumdaucher Kubernetes cluster. The monitoring stack runs in the monitoring namespace.

Prerequisites — Grafana Skills

Before creating or modifying dashboards, read these skills first — they cover general Grafana topics (JSON schema, panel types, PromQL, provisioning) that this document does not repeat:

Skill	Covers
`dashboarding`	Dashboard JSON schema, panel types, units, template variables, transformations, annotations
`grafana-oss`	Dashboard/datasource provisioning, RBAC, plugin config, health checks
`promql`	PromQL query writing, rate/irate, histograms, recording rules, cardinality

For project-specific dashboard creation workflows (ConfigMap template, InfluxDB Flux queries, deploy/verify steps), see DASHBOARD_CREATION.md.

Folder Structure

monitoring/
├── AGENTS.md                          ← You are here
├── README.md                          ← Human-oriented deployment guide
├── DASHBOARD_CREATION.md              ← Project-specific dashboard creation guide
├── prometheus-operator.secret.yml     ← Helm values for kube-prometheus-stack
├── alertmanagerconfig.secret.yaml     ← Alertmanager config (Telegram alerts)
├── servicemonitor.secret.yml          ← ServiceMonitor examples (commented out)
├── taupi-fan.yml                      ← Cellar fan exporter (Service, Endpoints, ServiceMonitor, Alerts)
├── tankerkoenig.yml                   ← Tankerkoenig fuel price exporter
├── dashboards/                        ← Grafana dashboards as ConfigMaps
│   ├── grafana-dashboard-home-climate.yaml
│   ├── grafana-dashboard-energy-monitor.yaml
│   ├── grafana-dashboard-heating-gas.yaml
│   ├── grafana-dashboard-outdoor-weather.yaml
│   └── grafana-dashboard-taupi-fan.yaml
└── scripts/                           ← Discovery & helper scripts
    ├── discover-influxdb.sh           ← Query InfluxDB schema
    └── discover-home-assistant.sh     ← Query Home Assistant entities

Key Infrastructure Context

Namespaces & Services

Namespace	Service	Purpose
`monitoring`	kube-prometheus-stack	Prometheus, Alertmanager, Grafana
`influxdb`	InfluxDB2	Time-series storage for Home Assistant sensor data
`home-assistant`	Home Assistant	Smart home hub, writes sensor data to InfluxDB

Data Flow

Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
                                    ↓
                         Grafana (Flux queries via datasource)

Custom exporters (taupi-fan, etc.) → Prometheus (scraped via ServiceMonitor)
                                              ↓
                                   Grafana (PromQL queries)

Grafana Datasources

Name	UID	Type	URL
InfluxDB Home Assistant	`P2AB959DC95E5519F`	influxdb (Flux)	`http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
InfluxDB Home Assistant InfluxQL	`P86455D1023ECC470`	influxdb (InfluxQL)	same
Prometheus	`prometheus`	prometheus	internal

⚠️ IMPORTANT: The datasource UIDs above are the current values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources before creating new dashboards.

Grafana Admin Credentials

URL: https://grafana.haumdaucher.de
User: admin
Password: stored in prometheus-operator.secret.yml under grafana.adminPassword

Home Assistant API

URL: https://hass.moritzgraf.de
Token: stored in ../home-assistant/tmp_long_lived_token.secret.yml

InfluxDB API

Internal URL: http://influxdb-influxdb2.influxdb.svc.cluster.local:80
Org: influxdata
Bucket: default
Token: stored in prometheus-operator.secret.yml under grafana.additionalDataSources[0].secureJsonData.token

Dashboard Architecture

Dashboards are managed as Kubernetes ConfigMaps with the label grafana_dashboard: "1". The kube-prometheus-stack's Grafana Dashboard Sidecar (grafana-sc-dashboard container, singular!) watches for these ConfigMaps and auto-loads them into Grafana.

Key Design Decisions

One file per dashboard — Each ConfigMap YAML contains the dashboard JSON embedded directly in data.
Strict GitOps — Dashboards are editable: false in Grafana UI. All changes go through YAML files.
Namespace-scoped — Sidecar only watches monitoring namespace (searchNamespace: monitoring).
Folder annotation — Use grafana_dashboard_folder annotation to organize dashboards.
No "dashboard" wrapper — File provisioning requires top-level JSON properties. The {"dashboard": {...}} wrapper is only for the HTTP API.
Never add uid to helm datasource config — Let Grafana auto-generate UIDs. Adding uid to additionalDataSources crashes provisioning.

For the full creation workflow, Flux query templates, and InfluxDB conventions, see DASHBOARD_CREATION.md.

../home-assistant/home-assistant.secret.yaml — HA helm values (InfluxDB integration config)
../home-assistant/tmp_long_lived_token.secret.yml — HA API token for discovery
../influxdb/influxdb2.secret.yml — InfluxDB helm values

Alerting Policy

We prioritize a high threshold for alerts to prevent alarm fatigue in this private home lab environment:

Alert Duration Threshold: Most alerts (including container downtime, offline bridges, and missing device metrics) are configured with a for: 1h duration. They are only treated as critical and trigger active alerts if they remain unresolved for longer than 1 hour.

6.0 KiB Raw Blame History