infrapuzzle/k8s/monitoring/AGENTS.md

# AGENTS.md — Monitoring Folder Guide for AI Agents

## Overview

This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.

## Prerequisites — Grafana Skills

Before creating or modifying dashboards, **read these skills first** — they cover general Grafana topics (JSON schema, panel types, PromQL, provisioning) that this document does not repeat:

| Skill | Covers |
|---|---|
| **`dashboarding`** | Dashboard JSON schema, panel types, units, template variables, transformations, annotations |
| **`grafana-oss`** | Dashboard/datasource provisioning, RBAC, plugin config, health checks |
| **`promql`** | PromQL query writing, rate/irate, histograms, recording rules, cardinality |

For project-specific dashboard creation workflows (ConfigMap template, InfluxDB Flux queries, deploy/verify steps), see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.

## Folder Structure

```
monitoring/
├── AGENTS.md                          ← You are here
├── README.md                          ← Human-oriented deployment guide
├── DASHBOARD_CREATION.md              ← Project-specific dashboard creation guide
├── prometheus-operator.secret.yml     ← Helm values for kube-prometheus-stack
├── alertmanagerconfig.secret.yaml     ← Alertmanager config (Telegram alerts)
├── servicemonitor.secret.yml          ← ServiceMonitor examples (commented out)
├── taupi-fan.yml                      ← Cellar fan exporter (Service, Endpoints, ServiceMonitor, Alerts)
├── tankerkoenig.yml                   ← Tankerkoenig fuel price exporter
├── dashboards/                        ← Grafana dashboards as ConfigMaps
│   ├── grafana-dashboard-home-climate.yaml
│   ├── grafana-dashboard-energy-monitor.yaml
│   ├── grafana-dashboard-heating-gas.yaml
│   ├── grafana-dashboard-outdoor-weather.yaml
│   └── grafana-dashboard-taupi-fan.yaml
└── scripts/                           ← Discovery & helper scripts
    ├── discover-influxdb.sh           ← Query InfluxDB schema
    └── discover-home-assistant.sh     ← Query Home Assistant entities
```

## Key Infrastructure Context

### Namespaces & Services

| Namespace | Service | Purpose |
|-----------|---------|---------|
| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |

### Data Flow

```
Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
                                    ↓
                         Grafana (Flux queries via datasource)

Custom exporters (taupi-fan, etc.) → Prometheus (scraped via ServiceMonitor)
                                              ↓
                                   Grafana (PromQL queries)
```

### Grafana Datasources

| Name | UID | Type | URL |
|------|-----|------|-----|
| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
| Prometheus | `prometheus` | prometheus | internal |

> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.

### Grafana Admin Credentials

- URL: `https://grafana.haumdaucher.de`
- User: `admin`
- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`

### Home Assistant API

- URL: `https://hass.moritzgraf.de`
- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`

### InfluxDB API

- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
- Org: `influxdata`
- Bucket: `default`
- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`

## Dashboard Architecture

Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container, singular!) watches for these ConfigMaps and auto-loads them into Grafana.

### Key Design Decisions

1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`.
2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes go through YAML files.
3. **Namespace-scoped** — Sidecar only watches `monitoring` namespace (`searchNamespace: monitoring`).
4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards.
5. **No `"dashboard"` wrapper** — File provisioning requires top-level JSON properties. The `{"dashboard": {...}}` wrapper is only for the HTTP API.
6. **Never add `uid` to helm datasource config** — Let Grafana auto-generate UIDs. Adding `uid` to `additionalDataSources` crashes provisioning.

For the full creation workflow, Flux query templates, and InfluxDB conventions, see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.

## Related Files Outside This Folder

- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values

## Alerting Policy

We prioritize a high threshold for alerts to prevent alarm fatigue in this private home lab environment:
* **Alert Duration Threshold:** Most alerts (including container downtime, offline bridges, and missing device metrics) are configured with a `for: 1h` duration. They are only treated as critical and trigger active alerts if they remain unresolved for longer than 1 hour.