I have added a lot of changes from various ai sessions
This commit is contained in:
Moritz Graf
@@ -0,0 +1,164 @@
|
||||
# AGENTS.md — Monitoring Folder Guide for AI Agents
|
||||
|
||||
## Overview
|
||||
|
||||
This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.
|
||||
|
||||
## Folder Structure
|
||||
|
||||
```
|
||||
monitoring/
|
||||
├── AGENTS.md ← You are here
|
||||
├── README.md ← Human-oriented deployment guide
|
||||
├── DASHBOARD_CREATION.md ← How to create/update dashboards (read this!)
|
||||
├── prometheus-operator.secret.yml ← Helm values for kube-prometheus-stack
|
||||
├── alertmanagerconfig.secret.yaml ← Alertmanager config (Telegram alerts)
|
||||
├── servicemonitor.secret.yml ← ServiceMonitor examples (commented out)
|
||||
├── tankerkoenig.yml ← Tankerkoenig fuel price exporter
|
||||
├── dashboards/ ← Grafana dashboards as ConfigMaps
|
||||
│ ├── grafana-dashboard-home-climate.yaml
|
||||
│ ├── grafana-dashboard-energy-monitor.yaml
|
||||
│ ├── grafana-dashboard-heating-gas.yaml
|
||||
│ └── grafana-dashboard-outdoor-weather.yaml
|
||||
└── scripts/ ← Discovery & helper scripts
|
||||
├── discover-influxdb.sh ← Query InfluxDB schema
|
||||
└── discover-home-assistant.sh ← Query Home Assistant entities
|
||||
```
|
||||
|
||||
## Key Infrastructure Context
|
||||
|
||||
### Namespaces & Services
|
||||
|
||||
| Namespace | Service | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
|
||||
| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
|
||||
| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
|
||||
↓
|
||||
Grafana (Flux queries via datasource)
|
||||
```
|
||||
|
||||
### Grafana Datasources
|
||||
|
||||
| Name | UID | Type | URL |
|
||||
|------|-----|------|-----|
|
||||
| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
|
||||
| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
|
||||
| Prometheus | `prometheus` | prometheus | internal |
|
||||
|
||||
> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.
|
||||
|
||||
### Grafana Admin Credentials
|
||||
|
||||
- URL: `https://grafana.haumdaucher.de`
|
||||
- User: `admin`
|
||||
- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`
|
||||
|
||||
### Home Assistant API
|
||||
|
||||
- URL: `https://hass.moritzgraf.de`
|
||||
- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`
|
||||
|
||||
### InfluxDB API
|
||||
|
||||
- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
|
||||
- Org: `influxdata`
|
||||
- Bucket: `default`
|
||||
- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`
|
||||
|
||||
## Dashboard Architecture
|
||||
|
||||
Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container) watches for these ConfigMaps and auto-loads them into Grafana.
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`. No separate JSON files (avoids duplication).
|
||||
2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes must be made to the YAML files and re-applied.
|
||||
3. **Namespace-scoped** — Sidecar only watches the `monitoring` namespace (`searchNamespace: monitoring`).
|
||||
4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards in Grafana folders.
|
||||
|
||||
### ConfigMap Template
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-dashboard-<name>
|
||||
namespace: monitoring
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
annotations:
|
||||
grafana_dashboard_folder: "🏠 Home"
|
||||
data:
|
||||
<name>.json: |
|
||||
{
|
||||
"uid": "<unique-uid>",
|
||||
"title": "<Dashboard Title>",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
## InfluxDB Data Schema
|
||||
|
||||
Home Assistant writes sensor data to InfluxDB with this structure:
|
||||
|
||||
- **Measurement name** = unit of measurement (e.g., `°C`, `%`, `W`, `kWh`, `m³`, `hPa`, `km/h`, `mm/h`)
|
||||
- **Fields**: `value` (numeric), plus metadata fields (`friendly_name_str`, `device_class_str`, `id_str`, etc.)
|
||||
- **Tags**: `entity_id` (e.g., `home_wohnzimmer_temperature` — **NOTE: no `sensor.` prefix!**), `domain` (e.g., `sensor`)
|
||||
|
||||
### Flux Query Pattern (MUST follow this exactly)
|
||||
|
||||
**Exact match on entity_id:**
|
||||
```flux
|
||||
from(bucket: "default")
|
||||
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
||||
|> filter(fn: (r) => r["_measurement"] == "°C")
|
||||
|> filter(fn: (r) => r["_field"] == "value")
|
||||
|> filter(fn: (r) => r.entity_id == "home_wandthermostat_bad_temperature")
|
||||
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|
||||
|> yield(name: "mean")
|
||||
```
|
||||
|
||||
**Regex match on entity_id** (use `=~` operator with `/pattern/`):
|
||||
```flux
|
||||
from(bucket: "default")
|
||||
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|
||||
|> filter(fn: (r) => r["_measurement"] == "°C")
|
||||
|> filter(fn: (r) => r["_field"] == "value")
|
||||
|> filter(fn: (r) => r.entity_id =~ /^home_/)
|
||||
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|
||||
|> yield(name: "mean")
|
||||
```
|
||||
|
||||
**Critical rules:**
|
||||
- Always use bracket notation: `r["_measurement"]`, `r["_field"]` (fields starting with `_`)
|
||||
- Always use dynamic time: `range(start: v.timeRangeStart, stop: v.timeRangeStop)` and `aggregateWindow(every: v.windowPeriod, ...)`
|
||||
- Always end with `|> yield(name: "mean")`
|
||||
- **entity_id in InfluxDB does NOT have the `sensor.` prefix** — use `home_wohnzimmer_temperature`, NOT `sensor.home_wohnzimmer_temperature`
|
||||
- For "current value" stat panels, use `range(start: -15m)` with `last()` instead of aggregateWindow
|
||||
- Regex operator is `=~` with `/pattern/` syntax (e.g., `=~ /^home_/` matches all entities starting with `home_`)
|
||||
|
||||
## Common Pitfalls (Lessons Learned)
|
||||
|
||||
1. **Entity IDs have no `sensor.` prefix in InfluxDB** — HA stores domain separately. Use `home_wohnzimmer_temperature`, not `sensor.home_wohnzimmer_temperature`. This is the #1 cause of "no data".
|
||||
2. **Dashboard JSON must NOT be wrapped in `"dashboard"` key** — Grafana file provisioning expects top-level properties (`uid`, `title`, `panels`, etc.). The `"dashboard"` wrapper is only used in the HTTP API.
|
||||
3. **Datasource reference must use actual UID** — Discover via `kubectl exec` + Grafana API. Current UID: `P2AB959DC95E5519F`.
|
||||
4. **Don't add `uid` to helm datasource config** — Adding `uid` to existing datasources in `additionalDataSources` causes Grafana provisioning to fail with "data source not found". Let UIDs be auto-generated.
|
||||
5. **Always use bracket notation for `_measurement` and `_field`** — Use `r["_measurement"]` and `r["_field"]`, not dot notation. Fields starting with underscore may not resolve with dot notation in Flux.
|
||||
6. **Always use Grafana time variables** — Never hardcode `range(start: -24h)` or `aggregateWindow(every: 5m)`. Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`.
|
||||
7. **Always add `yield(name: "mean")`** — Required for Grafana to properly parse Flux results.
|
||||
8. **InfluxDB has multiple fields per measurement** — Always filter `r["_field"] == "value"` to get numeric data.
|
||||
9. **Sidecar container name is `grafana-sc-dashboard`** (singular), not `grafana-sc-dashboards`.
|
||||
10. **Use Wandthermostat entities for temperature/humidity** — Query HA to filter by device type. Wandthermostat entities have `wandthermostat` or `thermostat` in their entity_id.
|
||||
|
||||
## Related Files Outside This Folder
|
||||
|
||||
- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
|
||||
- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
|
||||
- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values
|
||||
- `../../kochbuch/plans/grafana-dashboards-as-code.md` — Original architecture plan
|
||||
Reference in New Issue
Block a user