111 lines
5.6 KiB
Markdown
111 lines
5.6 KiB
Markdown
# AGENTS.md — Monitoring Folder Guide for AI Agents
|
|
|
|
## Overview
|
|
|
|
This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.
|
|
|
|
## Prerequisites — Grafana Skills
|
|
|
|
Before creating or modifying dashboards, **read these skills first** — they cover general Grafana topics (JSON schema, panel types, PromQL, provisioning) that this document does not repeat:
|
|
|
|
| Skill | Covers |
|
|
|---|---|
|
|
| **`dashboarding`** | Dashboard JSON schema, panel types, units, template variables, transformations, annotations |
|
|
| **`grafana-oss`** | Dashboard/datasource provisioning, RBAC, plugin config, health checks |
|
|
| **`promql`** | PromQL query writing, rate/irate, histograms, recording rules, cardinality |
|
|
|
|
For project-specific dashboard creation workflows (ConfigMap template, InfluxDB Flux queries, deploy/verify steps), see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.
|
|
|
|
## Folder Structure
|
|
|
|
```
|
|
monitoring/
|
|
├── AGENTS.md ← You are here
|
|
├── README.md ← Human-oriented deployment guide
|
|
├── DASHBOARD_CREATION.md ← Project-specific dashboard creation guide
|
|
├── prometheus-operator.secret.yml ← Helm values for kube-prometheus-stack
|
|
├── alertmanagerconfig.secret.yaml ← Alertmanager config (Telegram alerts)
|
|
├── servicemonitor.secret.yml ← ServiceMonitor examples (commented out)
|
|
├── taupi-fan.yml ← Cellar fan exporter (Service, Endpoints, ServiceMonitor, Alerts)
|
|
├── tankerkoenig.yml ← Tankerkoenig fuel price exporter
|
|
├── dashboards/ ← Grafana dashboards as ConfigMaps
|
|
│ ├── grafana-dashboard-home-climate.yaml
|
|
│ ├── grafana-dashboard-energy-monitor.yaml
|
|
│ ├── grafana-dashboard-heating-gas.yaml
|
|
│ ├── grafana-dashboard-outdoor-weather.yaml
|
|
│ └── grafana-dashboard-taupi-fan.yaml
|
|
└── scripts/ ← Discovery & helper scripts
|
|
├── discover-influxdb.sh ← Query InfluxDB schema
|
|
└── discover-home-assistant.sh ← Query Home Assistant entities
|
|
```
|
|
|
|
## Key Infrastructure Context
|
|
|
|
### Namespaces & Services
|
|
|
|
| Namespace | Service | Purpose |
|
|
|-----------|---------|---------|
|
|
| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
|
|
| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
|
|
| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |
|
|
|
|
### Data Flow
|
|
|
|
```
|
|
Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
|
|
↓
|
|
Grafana (Flux queries via datasource)
|
|
|
|
Custom exporters (taupi-fan, etc.) → Prometheus (scraped via ServiceMonitor)
|
|
↓
|
|
Grafana (PromQL queries)
|
|
```
|
|
|
|
### Grafana Datasources
|
|
|
|
| Name | UID | Type | URL |
|
|
|------|-----|------|-----|
|
|
| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
|
|
| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
|
|
| Prometheus | `prometheus` | prometheus | internal |
|
|
|
|
> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.
|
|
|
|
### Grafana Admin Credentials
|
|
|
|
- URL: `https://grafana.haumdaucher.de`
|
|
- User: `admin`
|
|
- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`
|
|
|
|
### Home Assistant API
|
|
|
|
- URL: `https://hass.moritzgraf.de`
|
|
- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`
|
|
|
|
### InfluxDB API
|
|
|
|
- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
|
|
- Org: `influxdata`
|
|
- Bucket: `default`
|
|
- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`
|
|
|
|
## Dashboard Architecture
|
|
|
|
Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container, singular!) watches for these ConfigMaps and auto-loads them into Grafana.
|
|
|
|
### Key Design Decisions
|
|
|
|
1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`.
|
|
2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes go through YAML files.
|
|
3. **Namespace-scoped** — Sidecar only watches `monitoring` namespace (`searchNamespace: monitoring`).
|
|
4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards.
|
|
5. **No `"dashboard"` wrapper** — File provisioning requires top-level JSON properties. The `{"dashboard": {...}}` wrapper is only for the HTTP API.
|
|
6. **Never add `uid` to helm datasource config** — Let Grafana auto-generate UIDs. Adding `uid` to `additionalDataSources` crashes provisioning.
|
|
|
|
For the full creation workflow, Flux query templates, and InfluxDB conventions, see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.
|
|
|
|
## Related Files Outside This Folder
|
|
|
|
- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
|
|
- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
|
|
- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values |