# AGENTS.md — Monitoring Folder Guide for AI Agents

## Overview

This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.

## Folder Structure

```
monitoring/
├── AGENTS.md                          ← You are here
├── README.md                          ← Human-oriented deployment guide
├── DASHBOARD_CREATION.md              ← How to create/update dashboards (read this!)
├── prometheus-operator.secret.yml     ← Helm values for kube-prometheus-stack
├── alertmanagerconfig.secret.yaml     ← Alertmanager config (Telegram alerts)
├── servicemonitor.secret.yml          ← ServiceMonitor examples (commented out)
├── tankerkoenig.yml                   ← Tankerkoenig fuel price exporter
├── dashboards/                        ← Grafana dashboards as ConfigMaps
│   ├── grafana-dashboard-home-climate.yaml
│   ├── grafana-dashboard-energy-monitor.yaml
│   ├── grafana-dashboard-heating-gas.yaml
│   └── grafana-dashboard-outdoor-weather.yaml
└── scripts/                           ← Discovery & helper scripts
    ├── discover-influxdb.sh           ← Query InfluxDB schema
    └── discover-home-assistant.sh     ← Query Home Assistant entities
```

## Key Infrastructure Context

### Namespaces & Services

| Namespace | Service | Purpose |
|-----------|---------|---------|
| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |

### Data Flow

```
Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
                                    ↓
                         Grafana (Flux queries via datasource)
```

### Grafana Datasources

| Name | UID | Type | URL |
|------|-----|------|-----|
| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
| Prometheus | `prometheus` | prometheus | internal |

> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.

### Grafana Admin Credentials

- URL: `https://grafana.haumdaucher.de`
- User: `admin`
- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`

### Home Assistant API

- URL: `https://hass.moritzgraf.de`
- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`

### InfluxDB API

- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
- Org: `influxdata`
- Bucket: `default`
- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`

## Dashboard Architecture

Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container) watches for these ConfigMaps and auto-loads them into Grafana.

### Key Design Decisions

1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`. No separate JSON files (avoids duplication).
2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes must be made to the YAML files and re-applied.
3. **Namespace-scoped** — Sidecar only watches the `monitoring` namespace (`searchNamespace: monitoring`).
4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards in Grafana folders.

### ConfigMap Template

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-<name>
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_dashboard_folder: "🏠 Home"
data:
  <name>.json: |
    {
      "uid": "<unique-uid>",
      "title": "<Dashboard Title>",
      ...
    }
```

## InfluxDB Data Schema

Home Assistant writes sensor data to InfluxDB with this structure:

- **Measurement name** = unit of measurement (e.g., `°C`, `%`, `W`, `kWh`, `m³`, `hPa`, `km/h`, `mm/h`)
- **Fields**: `value` (numeric), plus metadata fields (`friendly_name_str`, `device_class_str`, `id_str`, etc.)
- **Tags**: `entity_id` (e.g., `home_wohnzimmer_temperature` — **NOTE: no `sensor.` prefix!**), `domain` (e.g., `sensor`)

### Flux Query Pattern (MUST follow this exactly)

**Exact match on entity_id:**
```flux
from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "°C")
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id == "home_wandthermostat_bad_temperature")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")
```

**Regex match on entity_id** (use `=~` operator with `/pattern/`):
```flux
from(bucket: "default")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "°C")
  |> filter(fn: (r) => r["_field"] == "value")
  |> filter(fn: (r) => r.entity_id =~ /^home_/)
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "mean")
```

**Critical rules:**
- Always use bracket notation: `r["_measurement"]`, `r["_field"]` (fields starting with `_`)
- Always use dynamic time: `range(start: v.timeRangeStart, stop: v.timeRangeStop)` and `aggregateWindow(every: v.windowPeriod, ...)`
- Always end with `|> yield(name: "mean")`
- **entity_id in InfluxDB does NOT have the `sensor.` prefix** — use `home_wohnzimmer_temperature`, NOT `sensor.home_wohnzimmer_temperature`
- For "current value" stat panels, use `range(start: -15m)` with `last()` instead of aggregateWindow
- Regex operator is `=~` with `/pattern/` syntax (e.g., `=~ /^home_/` matches all entities starting with `home_`)

## Common Pitfalls (Lessons Learned)

1. **Entity IDs have no `sensor.` prefix in InfluxDB** — HA stores domain separately. Use `home_wohnzimmer_temperature`, not `sensor.home_wohnzimmer_temperature`. This is the #1 cause of "no data".
2. **Dashboard JSON must NOT be wrapped in `"dashboard"` key** — Grafana file provisioning expects top-level properties (`uid`, `title`, `panels`, etc.). The `"dashboard"` wrapper is only used in the HTTP API.
3. **Datasource reference must use actual UID** — Discover via `kubectl exec` + Grafana API. Current UID: `P2AB959DC95E5519F`.
4. **Don't add `uid` to helm datasource config** — Adding `uid` to existing datasources in `additionalDataSources` causes Grafana provisioning to fail with "data source not found". Let UIDs be auto-generated.
5. **Always use bracket notation for `_measurement` and `_field`** — Use `r["_measurement"]` and `r["_field"]`, not dot notation. Fields starting with underscore may not resolve with dot notation in Flux.
6. **Always use Grafana time variables** — Never hardcode `range(start: -24h)` or `aggregateWindow(every: 5m)`. Always use `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`.
7. **Always add `yield(name: "mean")`** — Required for Grafana to properly parse Flux results.
8. **InfluxDB has multiple fields per measurement** — Always filter `r["_field"] == "value"` to get numeric data.
9. **Sidecar container name is `grafana-sc-dashboard`** (singular), not `grafana-sc-dashboards`.
10. **Use Wandthermostat entities for temperature/humidity** — Query HA to filter by device type. Wandthermostat entities have `wandthermostat` or `thermostat` in their entity_id.

## Related Files Outside This Folder

- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values
- `../../kochbuch/plans/grafana-dashboards-as-code.md` — Original architecture plan