164 lines
8.2 KiB
Markdown
164 lines
8.2 KiB
Markdown
# AGENTS.md — Monitoring Folder Guide for AI Agents
|
|
|
|
## Overview
|
|
|
|
This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.
|
|
|
|
## Folder Structure
|
|
|
|
```
|
|
monitoring/
|
|
├── AGENTS.md ← You are here
|
|
├── README.md ← Human-oriented deployment guide
|
|
├── DASHBOARD_CREATION.md ← How to create/update dashboards (read this!)
|
|
├── prometheus-operator.secret.yml ← Helm values for kube-prometheus-stack
|
|
├── alertmanagerconfig.secret.yaml ← Alertmanager config (Telegram alerts)
|
|
├── servicemonitor.secret.yml ← ServiceMonitor examples (commented out)
|
|
├── tankerkoenig.yml ← Tankerkoenig fuel price exporter
|
|
├── dashboards/ ← Grafana dashboards as ConfigMaps
|
|
│ ├── grafana-dashboard-home-climate.yaml
|
|
│ ├── grafana-dashboard-energy-monitor.yaml
|
|
│ ├── grafana-dashboard-heating-gas.yaml
|
|
│ └── grafana-dashboard-outdoor-weather.yaml
|
|
└── scripts/ ← Discovery & helper scripts
|
|
├── discover-influxdb.sh ← Query InfluxDB schema
|
|
└── discover-home-assistant.sh ← Query Home Assistant entities
|
|
```
|
|
|
|
## Key Infrastructure Context
|
|
|
|
### Namespaces & Services
|
|
|
|
| Namespace | Service | Purpose |
|
|
|-----------|---------|---------|
|
|
| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
|
|
| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
|
|
| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
|
|
↓
|
|
Grafana (Flux queries via datasource)
|
|
```
|
|
|
|
### Grafana Datasources
|
|
|
|
| Name | UID | Type | URL |
|
|
|------|-----|------|-----|
|
|
| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
|
|
| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
|
|
| Prometheus | `prometheus` | prometheus | internal |
|
|
|
|
> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.
|
|
|
|
### Grafana Admin Credentials
|
|
|
|
- URL: `https://grafana.haumdaucher.de`
|
|
- User: `admin`
|
|
- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`
|
|
|
|
### Home Assistant API
|
|
|
|
- URL: `https://hass.moritzgraf.de`
|
|
- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`
|
|
|
|
### InfluxDB API
|
|
|
|
- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
|
|
- Org: `influxdata`
|
|
- Bucket: `default`
|
|
- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`
|
|
|
|
## Dashboard Architecture
|
|
|
|
Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container) watches for these ConfigMaps and auto-loads them into Grafana.
|
|
|
|
### Key Design Decisions
|
|
|
|
1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`. No separate JSON files (avoids duplication).
|
|
2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes must be made to the YAML files and re-applied.
|
|
3. **Namespace-scoped** — Sidecar only watches the `monitoring` namespace (`searchNamespace: monitoring`).
|
|
4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards in Grafana folders.
|
|
|
|
### ConfigMap Template
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: grafana-dashboard-<name>
|
|
namespace: monitoring
|
|
labels:
|
|
grafana_dashboard: "1"
|
|
annotations:
|
|
grafana_dashboard_folder: "🏠 Home"
|
|
data:
|
|
<name>.json: |
|
|
{
|
|
"uid": "<unique-uid>",
|
|
"title": "<Dashboard Title>",
|
|
...
|
|
}
|
|
```
|
|
|
|
## InfluxDB Data Schema
|
|
|
|
Home Assistant writes sensor data to InfluxDB with this structure:
|
|
|
|
- **Measurement name** = unit of measurement (e.g., `°C`, `%`, `W`, `kWh`, `m³`, `hPa`, `km/h`, `mm/h`)
|
|
- **Fields**: `value` (numeric), plus metadata fields (`friendly_name_str`, `device_class_str`, `id_str`, etc.)
|
|
- **Tags**: `entity_id` (e.g., `home_wohnzimmer_temperature` — **NOTE: no `sensor.` prefix!**), `domain` (e.g., `sensor`)
|
|
|
|
### Flux Query Pattern (MUST follow this exactly)
|
|
|
|
**Exact match on entity_id:**
|
|
```flux
|
|
from(bucket: "default")
|
|
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
|
|> filter(fn: (r) => r["_measurement"] == "°C")
|
|
|> filter(fn: (r) => r["_field"] == "value")
|
|
|> filter(fn: (r) => r.entity_id == "home_wandthermostat_bad_temperature")
|
|
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|
|
|> yield(name: "mean")
|
|
```
|
|
|
|
**Regex match on entity_id** (use `=~` operator with `/pattern/`):
|
|
```flux
|
|
from(bucket: "default")
|
|
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
|
|> filter(fn: (r) => r["_measurement"] == "°C")
|
|
|> filter(fn: (r) => r["_field"] == "value")
|
|
|> filter(fn: (r) => r.entity_id =~ /^home_/)
|
|
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|
|
|> yield(name: "mean")
|
|
```
|
|
|
|
**Critical rules:**
|
|
- Always use bracket notation: `r["_measurement"]`, `r["_field"]` (fields starting with `_`)
|
|
- Always use dynamic time: `range(start: v.timeRangeStart, stop: v.timeRangeStop)` and `aggregateWindow(every: v.windowPeriod, ...)`
|
|
- Always end with `|> yield(name: "mean")`
|
|
- **entity_id in InfluxDB does NOT have the `sensor.` prefix** — use `home_wohnzimmer_temperature`, NOT `sensor.home_wohnzimmer_temperature`
|
|
- For "current value" stat panels, use `range(start: -15m)` with `last()` instead of aggregateWindow
|
|
- Regex operator is `=~` with `/pattern/` syntax (e.g., `=~ /^home_/` matches all entities starting with `home_`)
|
|
|
|
## Common Pitfalls (Lessons Learned)
|
|
|
|
1. **Entity IDs have no `sensor.` prefix in InfluxDB** — HA stores domain separately. Use `home_wohnzimmer_temperature`, not `sensor.home_wohnzimmer_temperature`. This is the #1 cause of "no data".
|
|
2. **Dashboard JSON must NOT be wrapped in `"dashboard"` key** — Grafana file provisioning expects top-level properties (`uid`, `title`, `panels`, etc.). The `"dashboard"` wrapper is only used in the HTTP API.
|
|
3. **Datasource reference must use actual UID** — Discover via `kubectl exec` + Grafana API. Current UID: `P2AB959DC95E5519F`.
|
|
4. **Don't add `uid` to helm datasource config** — Adding `uid` to existing datasources in `additionalDataSources` causes Grafana provisioning to fail with "data source not found". Let UIDs be auto-generated.
|
|
5. **Always use bracket notation for `_measurement` and `_field`** — Use `r["_measurement"]` and `r["_field"]`, not dot notation. Fields starting with underscore may not resolve with dot notation in Flux.
|
|
6. **Always use Grafana time variables** — Never hardcode `range(start: -24h)` or `aggregateWindow(every: 5m)`. Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`.
|
|
7. **Always add `yield(name: "mean")`** — Required for Grafana to properly parse Flux results.
|
|
8. **InfluxDB has multiple fields per measurement** — Always filter `r["_field"] == "value"` to get numeric data.
|
|
9. **Sidecar container name is `grafana-sc-dashboard`** (singular), not `grafana-sc-dashboards`.
|
|
10. **Use Wandthermostat entities for temperature/humidity** — Query HA to filter by device type. Wandthermostat entities have `wandthermostat` or `thermostat` in their entity_id.
|
|
|
|
## Related Files Outside This Folder
|
|
|
|
- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
|
|
- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
|
|
- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values
|
|
- `../../kochbuch/plans/grafana-dashboards-as-code.md` — Original architecture plan |