feat(shelly): optimize dew point ventilation setpoints, duty cycle, and night-window gating

- Raise target humidity setpoints from 40% to 55% with a 2% hysteresis shut-off
- Eliminate 9999-min unlimited runtime bypass and strictly cap bursts at 35 min
- Rebalance duty cycle to 1:1 ratio (35 min run, 35 min cooldown rest)
- Implement Shelly internal RTC time-of-day check to restrict routine drying to 02:00-07:00
- Add delta-T guard requiring spread >= 3.5°C and max 20 min bursts when outside is >8°C colder
- Update AGENTS.md documentation to reflect optimized gates and thresholds

.agents/skills/dashboarding/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: dashboarding
+license: Apache-2.0
+description: Build, modify, and ship Grafana dashboards as JSON via the HTTP API — panel types (timeseries / stat / gauge / table / heatmap / logs / traces / node-graph), `gridPos` 24-column layout, units, thresholds, template + datasource + chained variables, transformations (`organize` / `calculateField` / `filterByValue`), panel + dashboard links with `${__field.labels.x}` / `${__from}`, and Loki/Prometheus annotations. Use when scripting dashboard creation, writing the dashboard JSON for a new service, adding a `$job` dropdown variable, computing an "Error %" column with a transformation, overlaying deploys as annotations, or pushing a dashboard via `POST /api/dashboards/db` — even when the user says "create a dashboard for this metric", "add a service dropdown", "show errors as percentage", "overlay our deploys", or "export the dashboard JSON" without naming the API or schema. After every API push, verify with the returned `version` plus a GET on the dashboard UID.
+---
+
+# Grafana Dashboard Authoring
+
+> **Docs**: https://grafana.com/docs/grafana/latest/dashboards/
+
+Dashboards are JSON. Author once, push via API, share by `uid`.
+
+## Prerequisites
+
+- Grafana stack (OSS, Enterprise, or Cloud) reachable from your machine
+- API token with `dashboards:write` (`Authorization: Bearer <token>`)
+- `jq` for inspecting responses
+- The JSON-schema cheat sheet in [`references/json-schema.md`](references/json-schema.md)
+
+## Common Workflows
+
+### 1. Push a new dashboard via the API + verify
+
+```bash
+# 1. Build the payload — wrap the dashboard JSON, set folder, mark overwrite
+cat > /tmp/dash.json <<'JSON'
+{
+  "dashboard": {
+    "uid": "demo-svc-v1",
+    "title": "Demo Service",
+    "schemaVersion": 41,
+    "tags": ["demo"],
+    "time": { "from": "now-1h", "to": "now" },
+    "templating": { "list": [] },
+    "panels": [{
+      "id": 1, "type": "timeseries", "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 24, "h": 8 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{
+        "expr": "sum(rate(http_requests_total[5m])) by (status_code)",
+        "legendFormat": "{{status_code}}", "refId": "A"
+      }],
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] }
+    }]
+  },
+  "folderUid": "",
+  "overwrite": true,
+  "message": "initial push"
+}
+JSON
+
+# 2. Validate the JSON BEFORE you send it (catches trailing-comma typos)
+jq empty /tmp/dash.json && echo "json ok"
+
+# 3. POST
+RESP=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  "$GRAFANA/api/dashboards/db" -d @/tmp/dash.json)
+echo "$RESP" | jq '{status, uid, url, version}'
+# Expect: status="success", url="/d/demo-svc-v1/...", version=1 (incremented on each push)
+
+# 4. Verify the round-trip — read it back and confirm one panel + the expected title
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" \
+  | jq '{title: .dashboard.title, panels: (.dashboard.panels | length)}'
+# Expect: {"title":"Demo Service","panels":1}
+
+# 5. Open the dashboard in a browser — confirm the panel renders with data.
+```
+
+### 2. Add a `$job` template variable to an existing dashboard
+
+```bash
+# 1. Fetch existing dashboard
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" > /tmp/dash.json
+
+# 2. Edit templating.list — append:
+#   { "name":"job", "type":"query",
+#     "datasource":{"type":"prometheus","uid":"prometheus"},
+#     "query":{"query":"label_values(up, job)","refId":"A"},
+#     "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+#  (Use jq, an editor, or the Grafana UI — schema in references/json-schema.md.)
+
+# 3. Update the panel expr to use the variable: rate(http_requests_total{job=~"$job"}[5m])
+
+# 4. POST it back with overwrite: true. Verify the variable appears in the UI dropdown.
+```
+
+### 3. Compute an "Error %" column with a transformation
+
+```json
+{
+  "id": "calculateField",
+  "options": {
+    "alias": "Error %", "mode": "reduceRow",
+    "reduce": { "reducer": "last" },
+    "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }
+}
+```
+
+Add this to the panel's `transformations: []`. Verify in the UI panel inspector — the new field should appear and update with the variable selection.
+
+Full schema (panels, units, all transformations, annotations, links): [`references/json-schema.md`](references/json-schema.md).
+
+## API reference
+
+```bash
+# Get
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/<uid>" | jq '.dashboard'
+
+# Search
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/search?query=kubernetes&type=dash-db" | jq '.[] | {uid,title,folderTitle}'
+
+# Create folder
+curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" "$GRAFANA/api/folders" \
+  -d '{"uid":"platform-team","title":"Platform Team"}'
+```
+
+For dashboards embedded in app plugins, use `@grafana/scenes` (skill `grafana-o11y:grafana-scenes`).
+
+## Resources
+
+- [Dashboard JSON model](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/)
+- [HTTP API — dashboards](https://grafana.com/docs/grafana/latest/developers/http_api/dashboard/)
+- [Panel types](https://grafana.com/docs/grafana/latest/panels-visualizations/)
+- [Variables](https://grafana.com/docs/grafana/latest/dashboards/variables/)
+- [Transformations](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/)

.agents/skills/dashboarding/references/json-schema.md

@@ -0,0 +1,157 @@
+# Dashboard + panel JSON schema
+
+## Dashboard root
+
+```json
+{
+  "title": "My Dashboard",
+  "uid": "my-dashboard-v1",
+  "tags": ["service", "production"],
+  "time":     { "from": "now-1h", "to": "now" },
+  "refresh":  "30s",
+  "timezone": "browser",
+  "schemaVersion": 41,
+  "templating":  { "list": [] },
+  "annotations": { "list": [] },
+  "panels": []
+}
+```
+
+- `uid` — stable identifier; keep short
+- `schemaVersion` — `41` for Grafana 11+
+- `time.from` / `to` — relative (`now-1h`) or absolute ISO
+- `refresh` — `"30s"`, `"1m"`, `"5m"`, `""` (off)
+
+## Panel
+
+```json
+{
+  "id": 1,
+  "type": "timeseries",
+  "title": "Request Rate",
+  "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+  "datasource": { "type": "prometheus", "uid": "${datasource}" },
+  "targets": [{
+    "expr": "sum(rate(http_requests_total{job=\"$job\"}[5m])) by (status_code)",
+    "legendFormat": "{{status_code}}",
+    "refId": "A"
+  }],
+  "fieldConfig": {
+    "defaults": {
+      "unit": "reqps",
+      "thresholds": { "mode": "absolute", "steps": [
+        { "color": "green",  "value": null },
+        { "color": "yellow", "value": 1000 },
+        { "color": "red",    "value": 5000 }
+      ]}
+    },
+    "overrides": []
+  },
+  "options": {
+    "legend":  { "calcs": ["mean","max","last"], "displayMode": "table", "placement": "bottom" },
+    "tooltip": { "mode": "multi", "sort": "desc" }
+  }
+}
+```
+
+`gridPos`: 24-column grid. Widths: full=24, half=12, third=8, quarter=6. Height 1 unit ≈ 30 px.
+
+## Panel types
+
+| Panel | Use case |
+|---|---|
+| **Time series** | Any metric over time |
+| **Stat** | Single value + sparkline |
+| **Gauge** | % or value against min/max |
+| **Bar gauge** | Side-by-side comparison |
+| **Table** | Multi-column data |
+| **Heatmap** | Distribution over time |
+| **Logs** | Loki log streams |
+| **Traces** | Tempo trace search |
+| **Text** | Markdown docs |
+| **Candlestick** | OHLC / min-max-avg |
+| **Node graph** | Service dependency graph |
+
+## Useful units
+
+```
+reqps        requests/sec
+ops          ops/sec
+Bps          bytes/sec
+percentunit  0.0-1.0 as %
+bytes        bytes (auto-scales)
+decbytes     decimal bytes (1 KB = 1000 B)
+ms / s       milliseconds / seconds
+dtdurationms 1h 2m 3s
+short        compact (1.2k, 3.4M)
+none         raw number
+```
+
+## Template variables
+
+```json
+{ "name":"job", "type":"query", "datasource":{"type":"prometheus","uid":"prometheus"},
+  "query":{"query":"label_values(up, job)","refId":"A"},
+  "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+
+{ "name":"cluster", "type":"constant", "query":"production", "label":"Cluster" }
+
+{ "name":"datasource", "type":"datasource", "pluginId":"prometheus",
+  "includeAll":false, "label":"Prometheus" }
+```
+
+Multi-value variables expand to a regex OR: `$job=["api","worker"]` → `job=~"api|worker"`.
+Chained: `label_values(kube_pod_info{namespace=\"$namespace\"}, pod)`.
+
+## Transformations
+
+```json
+"transformations": [
+  { "id": "merge", "options": {} },
+  { "id": "organize", "options": {
+      "renameByName":  { "Value #A": "Request Rate", "Value #B": "Error Rate" },
+      "excludeByName": { "Time": true }
+  }},
+  { "id": "calculateField", "options": {
+      "alias": "Error %", "mode": "reduceRow",
+      "reduce": { "reducer": "last" },
+      "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }},
+  { "id": "filterByValue", "options": {
+      "filters": [{ "fieldName":"Error %", "config":{ "id":"greater", "options":{ "value":0.01 }}}],
+      "type": "include", "match": "any"
+  }}
+]
+```
+
+Common IDs: `merge`, `organize`, `rename`, `calculateField`, `filterByValue`, `groupBy`, `sortBy`, `limit`, `labelsToFields`, `seriesToRows`, `partitionByValues`.
+
+## Links & annotations
+
+```json
+"links": [
+  { "title":"Go to details", "url":"/d/details?var-service=${__field.labels.service}", "targetBlank":false },
+  { "title":"Runbook",        "url":"https://wiki.example.com/runbook/${job}", "icon":"external link",
+    "targetBlank":true, "type":"link" }
+]
+```
+
+Built-in vars: `${__value.raw}`, `${__field.labels.job}`, `${__url.params}`, `${__from}` / `${__to}` (Unix ms).
+
+Loki annotation:
+
+```json
+{ "datasource":{"type":"loki","uid":"loki"},
+  "expr":"{job=\"deployments\"} |= \"deployed\"",
+  "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"{{service}} deployed", "textFormat":"{{version}} by {{author}}" }
+```
+
+Prometheus annotation:
+
+```json
+{ "datasource":{"type":"prometheus","uid":"prometheus"},
+  "expr":"changes(kube_deployment_status_observed_generation{namespace=\"production\"}[5m]) > 0",
+  "step":"60s", "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"Deploy: {{deployment}}" }
+```

.agents/skills/grafana-oss/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: grafana-oss
+license: Apache-2.0
+description: Configure Grafana OSS — provisions dashboards from YAML, sets up data sources (Prometheus / Loki / Tempo / Pyroscope), writes dashboard JSON with template variables, builds panel queries, assigns built-in roles (Viewer / Editor / Admin / GrafanaAdmin), mints service-account tokens, edits grafana.ini server config, creates annotations, installs plugins via provisioning, and validates each step with a health-check curl. Use when building dashboards, configuring data sources, setting up provisioning YAML, picking a panel type, writing template variables, managing users and roles, configuring SMTP/OAuth in grafana.ini, creating annotations via API, troubleshooting why a provisioned dashboard isn't showing up, or running Grafana OSS locally — even when the user says "set up a Prometheus data source", "provision dashboards from git", "make a service account", or "configure SSO in OSS" without saying "Grafana OSS".
+---
+
+# Grafana OSS
+
+> **Docs**: https://grafana.com/docs/grafana/latest.md
+
+## Common Workflows
+
+### Provisioning dashboards from disk
+
+1. Drop dashboard JSON file(s) under `/var/lib/grafana/dashboards/`
+2. Add a provider in `provisioning/dashboards/default.yaml` (see [§ Dashboard provisioning](#dashboard-provisioning) below)
+3. Restart Grafana so the provider config is loaded
+4. **Verify the dashboard landed**:
+   ```bash
+   curl https://grafana.example.com/api/dashboards/uid/<uid> \
+     -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+   ```
+   Returns the title → success. 404 → provisioning didn't pick it up; check Grafana server logs (`journalctl -u grafana-server | grep -i provisioning`) for parse errors.
+
+### Provisioning data sources
+
+1. Write `provisioning/datasources/datasources.yaml` (see [§ Data source provisioning](#data-source-provisioning) below)
+2. Restart Grafana
+3. **Health-check the data source via API**:
+   ```bash
+   curl https://grafana.example.com/api/datasources/uid/<uid>/health \
+     -H "Authorization: Bearer <token>"
+   # { "status": "OK", "message": "..." } → working
+   # { "status": "ERROR", ... } → URL unreachable or auth misconfigured
+   ```
+
+### Creating a service account + token
+
+1. Provision via YAML or `POST /api/serviceaccounts` (full API in [references/api.md § Users + service accounts](references/api.md#users--service-accounts))
+2. Mint a token via `POST /api/serviceaccounts/{id}/tokens`
+3. **Verify the token works**:
+   ```bash
+   curl https://grafana.example.com/api/org \
+     -H "Authorization: Bearer <new-token>"
+   # 200 + org JSON → token + role assignment work
+   # 401 → token wrong; 403 → role wrong
+   ```
+
+## Dashboard provisioning
+
+```yaml
+# provisioning/dashboards/default.yaml
+apiVersion: 1
+providers:
+  - name: default
+    folder: MyFolder
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+For the dashboard JSON shape itself (panels, queries, template variables), see [references/dashboard-json.md](references/dashboard-json.md).
+
+## Data source provisioning
+
+```yaml
+# provisioning/datasources/datasources.yaml
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    jsonData:
+      timeInterval: 15s
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+
+  - name: Tempo
+    type: tempo
+    access: proxy
+    url: http://tempo:3200
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki_uid
+        tags: [{ key: "service.name", value: "app" }]
+      serviceMap:
+        datasourceUid: prometheus_uid
+      nodeGraph:
+        enabled: true
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+```
+
+## RBAC (built-in roles)
+
+| Role | Permissions |
+|------|-------------|
+| **Viewer** | Read dashboards, alerts |
+| **Editor** | Create/edit dashboards, alerts |
+| **Admin** | Manage data sources, users, plugins |
+| **GrafanaAdmin** | Server-wide admin (superuser) |
+
+Service-account provisioning:
+
+```yaml
+# provisioning/access-control/service_accounts.yaml
+apiVersion: 1
+serviceAccounts:
+  - name: ci-reader
+    orgId: 1
+    role: Viewer
+    tokens:
+      - name: ci-token
+        # expires: optional ISO 8601 timestamp; omit for no-expiry tokens
+```
+
+(Custom RBAC roles with fine-grained permissions are Enterprise / Cloud only — see the `grafana-cloud/admin` skill if you need those.)
+
+## Plugin provisioning
+
+```yaml
+# provisioning/plugins/plugins.yaml
+apiVersion: 1
+apps:
+  - type: grafana-pyroscope-app
+    disabled: false
+    jsonData:
+      backendUrl: http://pyroscope:4040
+```
+
+After restart, verify via `GET /api/plugins/<plugin-id>/health`.
+
+## References
+
+- [`references/dashboard-json.md`](references/dashboard-json.md) — full dashboard JSON model + template variables + common problems (uid uniqueness, gridPos arithmetic, datasource uid matching)
+- [`references/dashboards.md`](references/dashboards.md) — dashboard workflows, settings, variables, annotations, sharing, versions, playlists, and provisioning-as-code
+- [`references/datasources.md`](references/datasources.md) — data source setup and query examples for Prometheus, Loki, Tempo, SQL, CloudWatch, and plugins
+- [`references/panel-types.md`](references/panel-types.md) — panel-type table + decision guide for picking the right one
+- [`references/panels.md`](references/panels.md) — panel editor, visualization options, field config, transformations, query options, inspection, and performance tips
+- [`references/alerting.md`](references/alerting.md) — alerting concepts, contact points, notification policies, templates, silences, and common rule examples
+- [`references/api.md`](references/api.md) — full Grafana OSS API reference (dashboards, data sources, users, service accounts, annotations) with verification curls and common failure modes
+- [`references/config.md`](references/config.md) — `grafana.ini` server / database / SMTP / auth / security / feature-toggle config + restart-required issues

.agents/skills/grafana-oss/references/alerting.md

@@ -0,0 +1,425 @@
+# Grafana Alerting - Detailed Reference
+
+## Overview
+
+Grafana Alerting is a unified alerting system for monitoring metrics and logs across multiple data sources. It fires notifications when conditions are breached.
+
+Key capabilities:
+- Query multiple data sources in a single alert rule
+- Multi-dimensional alerts (one rule creates many alert instances)
+- Flexible notification routing via policies
+- Silences and mute timings for planned maintenance
+- Alert history and state tracking
+
+---
+
+## Core Concepts
+
+### Alert Rule
+Defines what to monitor and when to fire. Contains: queries, a condition (threshold), an evaluation interval, and a pending period.
+
+### Alert Instance
+When a rule produces multi-dimensional data, it creates one alert instance per unique label combination. Example: an alert on `cpu_usage{host=~".*"}` creates one instance per host.
+
+### Alert States
+
+| State | Description |
+|-------|-------------|
+| **Normal** | Query running; condition not met |
+| **Pending** | Condition met but pending period not yet elapsed |
+| **Firing** | Condition met + pending period elapsed; notifications sent |
+| **Resolved** | Previously firing alert returned to normal |
+| **No Data** | Query returned no data (configurable behavior) |
+| **Error** | Query failed with an error (configurable behavior) |
+
+### Evaluation Group
+Alert rules are organized into evaluation groups. All rules in a group share the same evaluation interval and are evaluated sequentially.
+
+### Pending Period
+How long the condition must be continuously met before firing.
+- `0s` = fire immediately
+- `5m` = must be in breach for 5 continuous minutes
+
+### Keep Firing For
+How long an alert continues firing after the condition resolves (prevents brief recovery from clearing the alert).
+
+---
+
+## Alert Rule Types
+
+### Grafana-Managed Rules (Recommended)
+- Stored in Grafana's database
+- Can query any data source
+- Support multi-dimensional alerting
+- Support expressions (math, reduce, threshold)
+
+### Data Source-Managed Rules (Prometheus/Mimir/Loki)
+- Rules stored in the external system
+- Evaluated by the external system
+- Grafana provides UI to manage them
+- Useful when migrating from Prometheus alerting
+
+---
+
+## Creating Grafana-Managed Alert Rules
+
+Navigate to: **Alerting > Alert rules > New alert rule**
+
+### Step 1: Define query and condition
+
+Write one or more queries (labeled A, B, C...).
+
+Example with Prometheus:
+```promql
+# Query A: CPU usage per host
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Expression types you can add after queries:
+- **Math**: `$A > 80` or `($A + $B) / 2`
+- **Reduce**: Collapse time series to single value (Last, Mean, Sum, Max, Min)
+- **Resample**: Change time resolution
+- **Classic conditions**: Multiple threshold conditions with AND/OR
+- **Threshold**: Set the firing threshold
+
+### Step 2: Set evaluation behavior
+
+| Setting | Description |
+|---------|-------------|
+| Folder | Organize rules; folder is the RBAC boundary |
+| Evaluation group | Group name (all rules share this group's interval) |
+| Evaluation interval | How often the rule is evaluated (e.g., 1m) |
+| Pending period | How long condition must hold before firing (e.g., 5m) |
+| Keep firing for | How long alert stays firing after recovery |
+
+### Step 3: Configure labels and notifications
+
+**Labels** - Key-value pairs attached to alert instances. Used for routing and grouping:
+```
+severity=critical
+team=infrastructure
+service=database
+environment=production
+```
+
+**Annotations** - Context included in notification messages:
+```
+Summary: CPU usage above 80% on {{ $labels.instance }}
+Description: CPU usage is {{ $values.A.Value | humanize }}% on {{ $labels.instance }}
+Runbook URL: https://runbooks.company.com/cpu-high
+```
+
+**Notification settings**:
+- Set a **Contact point** to send directly, bypassing the routing policy tree
+- Or leave empty to use the notification policy routing tree
+
+### Step 4: No data and error handling
+
+| Situation | Options |
+|-----------|---------|
+| No data | Alerting, OK, No Data state, Keep last state |
+| Query error | Alerting, OK, Error state, Keep last state |
+
+---
+
+## Contact Points
+
+Contact points are notification destinations.
+
+Navigate to: **Alerting > Contact points**
+
+### Supported integrations
+
+| Integration | Notes |
+|-------------|-------|
+| Email | Requires SMTP config in grafana.ini |
+| Slack | Webhook URL or API token + channel |
+| PagerDuty | Integration key |
+| OpsGenie | API key |
+| VictorOps (Splunk) | API key |
+| Microsoft Teams | Incoming webhook URL |
+| Discord | Webhook URL |
+| Telegram | Bot token + chat ID |
+| Webhook | HTTP POST to custom URL |
+| Alertmanager | Forward to external Alertmanager |
+| Pushover | User key + API token |
+| LINE | LINE Notify token |
+
+### Email configuration in grafana.ini
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@company.com
+password = app-password
+from_address = alerts@company.com
+from_name = Grafana Alerts
+```
+
+### Webhook payload format
+Grafana POSTs a JSON payload to webhook endpoints:
+```json
+{
+  "receiver": "webhook-receiver",
+  "status": "firing",
+  "alerts": [
+    {
+      "status": "firing",
+      "labels": { "alertname": "HighCPU", "instance": "server1" },
+      "annotations": { "summary": "CPU above 80%" },
+      "startsAt": "2024-01-15T10:00:00Z",
+      "generatorURL": "http://grafana/alerting/..."
+    }
+  ],
+  "groupLabels": { "alertname": "HighCPU" },
+  "externalURL": "http://grafana"
+}
+```
+
+---
+
+## Notification Policies
+
+Notification policies route alerts to contact points based on label matchers.
+
+Navigate to: **Alerting > Notification policies**
+
+### Default policy
+The root policy - all alerts reach here if no specific policy matches.
+
+### Child policies
+Add policies that match specific labels:
+```
+Match labels:
+  severity = critical    -> contact: pagerduty
+  team = infrastructure  -> contact: slack-infra
+  environment = staging  -> contact: email-dev
+```
+
+### Policy settings
+
+| Setting | Description |
+|---------|-------------|
+| Contact point | Where to send matching alerts |
+| Continue matching | If true, also evaluate subsequent sibling policies |
+| Group by | Labels used for batching alerts into single notifications |
+| Group wait | Wait before sending first notification for a new group (default: 30s) |
+| Group interval | Wait before sending updates for an existing group (default: 5m) |
+| Repeat interval | Wait before re-sending for still-firing alerts (default: 4h) |
+
+### Grouping
+When multiple alerts have the same "group by" labels, they are batched into a single notification. This prevents notification storms.
+
+Example:
+- 50 hosts alert on HighCPU simultaneously
+- Group by: `[alertname, datacenter]`
+- Result: 1 notification per datacenter containing all affected hosts
+
+---
+
+## Notification Templates
+
+Customize notification message format using Go templating.
+
+Navigate to: **Alerting > Contact points > Notification templates**
+
+### Built-in variables
+```
+{{ $labels }}          # Alert labels as map
+{{ $values }}          # Query values map
+{{ $labels.instance }} # Specific label value
+{{ $values.A.Value }}  # Specific query value
+{{ $status }}          # firing or resolved
+{{ $startsAt }}        # When alert started firing
+```
+
+### Example Slack template
+```
+{{ define "slack_message" }}
+{{ if eq .Status "firing" }}:red_circle:{{ else }}:large_green_circle:{{ end }} *{{ .Labels.alertname }}*
+
+*Status:* {{ .Status }}
+*Severity:* {{ .Labels.severity }}
+
+{{ range .Alerts }}
+*Instance:* {{ .Labels.instance }}
+*Value:* {{ .Values.A.Value | humanize }}
+*Summary:* {{ .Annotations.summary }}
+{{ end }}
+{{ end }}
+```
+
+### Humanize functions
+```
+{{ $value | humanize }}           # "12.3k"
+{{ $value | humanize1024 }}       # "12.3Ki"
+{{ $value | humanizeBytes }}      # "12.3 kB"
+{{ $value | humanizeDuration }}   # "3h 2m 1s"
+{{ $value | humanizePercentage }} # "12.3%"
+```
+
+---
+
+## Silences
+
+Silences temporarily suppress alert notifications without stopping alert evaluation.
+
+Navigate to: **Alerting > Silences**
+
+### Create a silence
+1. Click **Add silence**
+2. Set start time and end time (or duration)
+3. Add label matchers:
+   ```
+   alertname = HighCPU
+   instance =~ ".*staging.*"   # Regex match
+   severity != critical         # Negative match
+   ```
+4. Add a comment explaining why
+5. Save
+
+### Use cases
+- Planned maintenance windows
+- Known issues being investigated
+- Silencing noisy alerts during deploys
+
+From a firing alert detail view: click **Silence** to pre-populate label matchers.
+
+---
+
+## Mute Timings
+
+Recurring schedules when notifications are suppressed (unlike silences which are one-time).
+
+Navigate to: **Alerting > Mute timings**
+
+### Example mute timings
+```
+Name: no-alerts-weekends
+  Weekdays: Saturday, Sunday
+
+Name: business-hours-only
+  Weekdays: Monday-Friday
+  Times: 09:00-17:00
+```
+
+Attach to notification policies in the policy settings.
+
+---
+
+## RBAC for Alerting
+
+Default permissions by role:
+
+| Action | Viewer | Editor | Admin |
+|--------|--------|--------|-------|
+| View alert rules | Yes | Yes | Yes |
+| Create/edit alert rules | No | Yes | Yes |
+| Delete alert rules | No | No | Yes |
+| Manage contact points | No | No | Yes |
+| Manage notification policies | No | No | Yes |
+| Create silences | No | Yes | Yes |
+
+---
+
+## grafana.ini Alerting Configuration
+
+```ini
+[unified_alerting]
+# Enable unified alerting (default: true since Grafana 9)
+enabled = true
+
+# Maximum alert instances a single rule can produce
+max_annotations_to_keep = 100
+
+# Evaluation timeout
+evaluation_timeout = 30s
+
+# Minimum evaluation interval (prevent too-frequent evaluation)
+min_interval = 10s
+
+[smtp]
+# Required for email contact points
+enabled = true
+host = smtp.example.com:587
+```
+
+---
+
+## Common Alert Rule Examples
+
+### CPU usage above threshold
+```promql
+# Query A
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+# Threshold: A > 80
+# Pending: 5m
+# Labels: severity=warning
+```
+
+### Memory usage
+```promql
+# Query A
+(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
+# Threshold: A > 90
+```
+
+### HTTP error rate
+```promql
+# Query A: error requests
+sum(rate(http_requests_total{status=~"5.."}[5m]))
+# Query B: total requests
+sum(rate(http_requests_total[5m]))
+# Expression C (Math): $A / $B * 100
+# Threshold: C > 5
+```
+
+### Disk space
+```promql
+# Query A
+(1 - (node_filesystem_free_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) * 100
+# Threshold: A > 85
+```
+
+### Service down (no data = firing)
+```promql
+# Query A
+up{job="my-service"}
+# Threshold: A < 1
+# No data handling: Alerting (treat absence as firing)
+```
+
+### Loki log error rate
+```logql
+# Query A
+sum(rate({job="api"} |= "ERROR" [5m]))
+# Threshold: A > 10
+```
+
+---
+
+## Connecting Alerts to Dashboards
+
+### Create alert from panel
+1. Open panel editor
+2. Click **Alert** tab
+3. Click **Create alert rule from this panel**
+4. Pre-populates query from the panel
+
+### Link alert to dashboard panel
+In the alert rule definition, set **Dashboard** and **Panel** to link to a specific visualization. The alert state badge appears on the panel and clicking it goes to the alert rule.
+
+---
+
+## High Availability (HA) Alerting
+
+```ini
+[unified_alerting]
+ha_peers = grafana-1:9094,grafana-2:9094,grafana-3:9094
+ha_advertise_address = ${POD_IP}:9094
+ha_peer_timeout = 15s
+ha_gossip_interval = 200ms
+ha_push_pull_interval = 60s
+```
+
+Multiple Grafana instances share state to avoid duplicate notifications.

.agents/skills/grafana-oss/references/api.md

@@ -0,0 +1,107 @@
+# Grafana OSS API
+
+Base URL: `https://your-grafana.example.com/api/`. Auth: service account token (`Authorization: Bearer <token>`).
+
+## Contents
+
+- [Dashboards](#dashboards)
+- [Data sources](#data-sources)
+- [Users + service accounts](#users--service-accounts)
+- [Annotations](#annotations)
+
+## Dashboards
+
+```bash
+# Search
+GET /api/search?query=service&type=dash-db&folderIds=1
+
+# Get by UID
+GET /api/dashboards/uid/{uid}
+
+# Create / update (overwrite: true replaces existing)
+POST /api/dashboards/db
+Body: { "dashboard": {...}, "folderUID": "...", "overwrite": true }
+
+# Delete
+DELETE /api/dashboards/uid/{uid}
+```
+
+After provisioning a dashboard via YAML, verify it landed:
+```bash
+curl https://grafana.example.com/api/dashboards/uid/<uid> \
+  -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+# Should print the dashboard's title. 404 = not provisioned correctly.
+```
+
+## Data sources
+
+```bash
+# List
+GET /api/datasources
+
+# Get by UID
+GET /api/datasources/uid/{uid}
+
+# Create
+POST /api/datasources
+Body: { "name": "...", "type": "...", "url": "...", "access": "proxy" }
+
+# Health-check (good post-provision validation)
+GET /api/datasources/uid/{uid}/health
+# Returns { "status": "OK" | "ERROR", "message": "..." }
+```
+
+## Users + service accounts
+
+```bash
+# List org users
+GET /api/org/users
+
+# List service accounts
+GET /api/serviceaccounts/search?perpage=100&page=1
+
+# Create service account
+POST /api/serviceaccounts
+Body: { "name": "ci-reader", "role": "Viewer", "isDisabled": false }
+
+# Mint a token
+POST /api/serviceaccounts/{id}/tokens
+Body: { "name": "ci-token", "secondsToLive": 0 }   # 0 = no expiry
+
+# Verify a token works
+curl https://grafana.example.com/api/org \
+  -H "Authorization: Bearer <new-token>"
+# 200 + org JSON = token + role assignment work.
+```
+
+## Annotations
+
+```bash
+# Create
+curl -X POST https://grafana.example.com/api/annotations \
+  -H 'Authorization: Bearer <token>' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "dashboardUID": "service-overview",
+    "panelId": 1,
+    "time": 1706745600000,
+    "timeEnd": 1706749200000,
+    "tags": ["deploy", "v2.0"],
+    "text": "Deployed v2.0"
+  }'
+
+# Find (by tag)
+GET /api/annotations?tags=deploy&from=1706745600000&to=1706832000000
+
+# Delete
+DELETE /api/annotations/{id}
+```
+
+## Common failure modes
+
+| Symptom | Likely cause |
+|---|---|
+| `401 Unauthorized` | Token expired or wrong stack URL — check the Authorization header |
+| `403 Forbidden` on dashboard create | Service account lacks Editor role on the target folder |
+| `412 Precondition Failed` on POST `/dashboards/db` | UID exists and you didn't set `overwrite: true` |
+| Data source health check `ERROR` | Network unreachable (check `url` field) or credentials wrong |

.agents/skills/grafana-oss/references/config.md

@@ -0,0 +1,100 @@
+# grafana.ini configuration
+
+Reference for `grafana.ini` server-side settings.
+
+## Contents
+
+- [Server + database](#server--database)
+- [Alerting](#alerting)
+- [SMTP](#smtp)
+- [Auth (OAuth example)](#auth-oauth-example)
+- [Security](#security)
+- [Feature toggles](#feature-toggles)
+
+## Server + database
+
+```ini
+[server]
+http_port = 3000
+domain = grafana.example.com
+root_url = https://grafana.example.com/
+
+[database]
+type = postgres
+host = postgres:5432
+name = grafana
+user = grafana
+password = secret
+```
+
+After changing these, restart Grafana and verify:
+- `curl http://localhost:3000/api/health` returns `{"database": "ok", ...}`
+- `curl http://localhost:3000/api/admin/settings | jq '.database'` shows the new config
+
+## Alerting
+
+```ini
+[alerting]
+enabled = true
+
+[unified_alerting]
+enabled = true
+```
+
+## SMTP
+
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@example.com
+password = yourpassword
+from_address = alerts@example.com
+```
+
+Verify by sending a test email via the Alerting → Contact points UI.
+
+## Auth (OAuth example)
+
+```ini
+[auth.generic_oauth]
+enabled = true
+name = Okta
+client_id = your_client_id
+client_secret = your_secret
+auth_url = https://your-org.okta.com/oauth2/v1/authorize
+token_url = https://your-org.okta.com/oauth2/v1/token
+api_url = https://your-org.okta.com/oauth2/v1/userinfo
+scopes = openid profile email groups
+```
+
+For SAML and GitHub OAuth, see the [grafana-cloud/admin skill](../../../grafana-cloud/admin/references/sso.md). The configs are the same in OSS.
+
+## Security
+
+```ini
+[security]
+admin_user = admin
+admin_password = secret
+allow_embedding = true       # required for embedding dashboards in iframes
+```
+
+`admin_password` is only consulted on first startup. To change later, use `grafana-cli admin reset-admin-password <new-password>`.
+
+## Feature toggles
+
+```ini
+[feature_toggles]
+enable = publicDashboards
+```
+
+Multiple toggles are space-separated:
+```ini
+enable = publicDashboards correlations grafanaApiServer
+```
+
+## Common problems
+
+- **`grafana.ini` changes need a restart** — config is read at startup, not live-reloaded
+- **`root_url` matters for OAuth callbacks** — if you set `domain` but not `root_url`, OAuth redirect URIs may not match
+- **Sections are global** — `[server]`, `[database]`, etc. apply at the process level, not per-org

.agents/skills/grafana-oss/references/dashboard-json.md

@@ -0,0 +1,81 @@
+# Dashboard JSON model + template variables
+
+## Minimal dashboard JSON
+
+```json
+{
+  "title": "Service Overview",
+  "uid": "service-overview",
+  "time": { "from": "now-1h", "to": "now" },
+  "refresh": "30s",
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus" },
+          "expr": "rate(http_requests_total{job=\"$job\"}[5m])",
+          "legendFormat": "{{method}} {{status}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 1000 }
+            ]
+          }
+        }
+      }
+    }
+  ]
+}
+```
+
+## Template variables
+
+```json
+{
+  "templating": {
+    "list": [
+      {
+        "name": "namespace",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prom" },
+        "definition": "label_values(kube_pod_info, namespace)",
+        "includeAll": true,
+        "multi": true
+      },
+      {
+        "name": "env",
+        "type": "custom",
+        "query": "production,staging,dev",
+        "current": { "value": "production" }
+      },
+      {
+        "name": "interval",
+        "type": "interval",
+        "query": "1m,5m,15m,1h",
+        "auto": true
+      }
+    ]
+  }
+}
+```
+
+Reference variables in queries with `$variable`:
+```promql
+rate(http_requests_total{namespace="$namespace"}[$interval])
+```
+
+## Common problems
+
+- **`uid` must be unique across the org** — if you POST a dashboard with an existing UID and `overwrite: false`, Grafana returns 412 Precondition Failed
+- **`gridPos`** uses a 24-column grid; `w + x` must be ≤ 24
+- **`datasource.uid` in `targets`** must match an existing data source UID; misspell it and panels render as "Datasource not found"
+- **Template variable `query` field** is data-source-specific syntax (PromQL `label_values(...)`, LogQL `{label="…"}`, SQL, etc.)

.agents/skills/grafana-oss/references/dashboards.md

@@ -0,0 +1,215 @@
+# Grafana Dashboards - Detailed Reference
+
+## What is a Dashboard?
+
+A Grafana dashboard is a set of one or more panels organized into rows, providing an at-a-glance view of related information. Dashboards connect to data sources, display data through panels, and support interactive filtering via variables.
+
+## Core Components
+
+- **Panels**: Containers that display visualizations; each combines a query + a visualization type
+- **Rows**: Horizontal groupings of panels; can be collapsed for organization
+- **Data Sources**: Connections to databases, APIs, or services that panels query
+- **Variables**: Dropdown selectors at the top of a dashboard for dynamic filtering
+- **Annotations**: Markers overlaid on graphs to indicate events
+
+---
+
+## Creating Dashboards
+
+### From scratch
+1. Click **Dashboards** in the left sidebar
+2. Click **New > New dashboard**
+3. Click **Add visualization** to add your first panel
+4. Select a data source, write a query, choose visualization type
+5. Click **Apply** to save the panel to the dashboard
+6. Click the save icon (or Ctrl+S) to save the dashboard
+
+### From a template / Import
+- Click **Dashboards > New > Import**
+- Paste a dashboard JSON, enter a Grafana.com dashboard ID, or upload a JSON file
+- Grafana.com hosts thousands of community dashboards (grafana.com/grafana/dashboards)
+
+---
+
+## Dashboard Settings
+
+Access via the gear icon at the top-right of any dashboard.
+
+| Setting | Description |
+|---------|-------------|
+| General | Name, description, tags, folder, editable flag |
+| Annotations | Configure annotation queries that overlay events on panels |
+| Variables | Add/edit template variables (dropdowns at top) |
+| Links | Add links to other dashboards or external URLs |
+| JSON Model | View/edit raw JSON for the entire dashboard |
+| Versions | Browse and restore prior versions |
+| Time options | Default time range, auto-refresh intervals, timezone |
+
+---
+
+## Time Range Controls
+
+- **Time picker** (top-right): Select absolute or relative ranges (Last 6 hours, Last 7 days, etc.)
+- **Auto-refresh**: Set to Off, 5s, 10s, 30s, 1m, 5m, etc.
+- **Zoom**: Click-drag on any time series to zoom in
+
+### Common relative time shortcuts
+```
+now-5m    last 5 minutes
+now-1h    last 1 hour
+now-24h   last 24 hours
+now-7d    last 7 days
+now/d     today so far
+now-1d/d  yesterday
+```
+
+---
+
+## Annotations
+
+Annotations are event markers overlaid on time series panels.
+
+### Types
+- **Native annotations**: Manually add a note to a specific time directly in the dashboard
+- **Query annotations**: Pull events from a data source and display as markers
+
+### Adding a manual annotation
+- Hold Ctrl (or Cmd on Mac) and click a time series panel
+- Type a description; optionally set a time range
+
+---
+
+## Library Panels
+
+Library panels are reusable panel definitions shared across multiple dashboards.
+
+- **Create**: Panel menu (3-dot) > "Create library panel"
+- **Use**: Add a panel > "Add from panel library"
+- **Update**: Edit the source panel; all dashboards using it reflect the change
+- **Unlink**: Detach to make it independent in a specific dashboard
+
+---
+
+## Panel Layout
+
+- Drag panel corners to resize
+- Drag panel header to move
+- **Add > Row** to insert collapsible rows
+- Panel menu > **Duplicate** to clone within the same dashboard
+
+---
+
+## Dashboard Versions
+
+- Access via Dashboard Settings > Versions
+- See who saved what and when
+- Compare two versions (diff view)
+- Restore any previous version
+
+---
+
+## JSON Model
+
+Every dashboard is stored as a JSON document.
+
+Key top-level JSON fields:
+```json
+{
+  "title": "My Dashboard",
+  "uid": "abc123",
+  "tags": ["production", "infrastructure"],
+  "time": { "from": "now-6h", "to": "now" },
+  "refresh": "30s",
+  "panels": [],
+  "templating": { "list": [] },
+  "annotations": { "list": [] }
+}
+```
+
+---
+
+## Sharing Dashboards
+
+### Share link
+- Click **Share** icon > **Link** tab
+- Toggle "Lock time range" to embed the current time window
+- Toggle "Include template variable values"
+
+### Snapshot
+- **Share > Snapshot**: Creates a read-only, public snapshot
+- Contains rendered data at share time - no live data source access needed
+- Can expire (1 hour, 1 day, 7 days, or never)
+
+### Export / Import JSON
+- **Share > Export**: Download the dashboard JSON
+- Import at **Dashboards > New > Import**
+- Useful for version control, migration, sharing with the community
+
+### Embed
+- **Share > Embed**: Generates an iframe HTML snippet
+- Requires anonymous access in Grafana config (OSS/Enterprise only)
+
+### Public dashboards (Grafana 10+)
+- Make a dashboard publicly accessible with no login
+- Enable per-dashboard in Share > Public dashboard
+
+---
+
+## Playlists
+
+Playlists cycle through dashboards automatically at a configurable interval.
+
+- Create at **Dashboards > Playlists > New playlist**
+- Add dashboards by name or tag
+- Set interval (e.g., 5 minutes)
+- Append `?kiosk=1` to URL for kiosk/TV mode
+
+---
+
+## Best Practices
+
+1. **Organize with folders**: Use folders as RBAC permission boundaries
+2. **Use variables**: Make dashboards reusable across environments/services
+3. **Limit panels per dashboard**: Aim for <20 panels for performance
+4. **Use library panels**: Standardize common panels across teams
+5. **Tag dashboards**: Consistent tags for searchability
+6. **Version control**: Export JSON and store in Git
+7. **Use rows**: Collapse related panels to organize complex dashboards
+8. **Template your queries**: Use variables so one dashboard covers many targets
+
+---
+
+## Keyboard Shortcuts
+
+| Key | Action |
+|-----|--------|
+| `Ctrl+S` | Save dashboard |
+| `e` | Open panel editor (when panel focused) |
+| `v` | Toggle panel fullscreen |
+| `d r` | Refresh all panels |
+| `d s` | Dashboard settings |
+| `d k` | Toggle kiosk mode |
+| `?` | Show all shortcuts |
+
+---
+
+## Provisioning Dashboards (as Code)
+
+```yaml
+# /etc/grafana/provisioning/dashboards/default.yaml
+apiVersion: 1
+
+providers:
+  - name: default
+    orgId: 1
+    folder: Infrastructure
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: false
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+Place dashboard JSON files in `/var/lib/grafana/dashboards/`. Subdirectories become folders when `foldersFromFilesStructure: true`.

.agents/skills/grafana-oss/references/datasources.md

@@ -0,0 +1,462 @@
+# Grafana Data Sources - Detailed Reference
+
+## Overview
+
+Data sources are connections between Grafana and the systems storing your data. Grafana ships with built-in support for many popular data sources and supports additional sources via plugins.
+
+- Only **Organization Admins** can add or remove data sources
+- Each data source has its own query editor
+- Data sources can be used in: Dashboard panels, Explore, Alert rules, Annotations
+
+---
+
+## Managing Data Sources
+
+### Add a data source
+1. Go to **Connections > Data sources** (or **Configuration > Data sources** in older versions)
+2. Click **Add new data source**
+3. Search for and select the data source type
+4. Fill in connection details (URL, credentials, etc.)
+5. Click **Save & Test** to verify the connection
+
+### Data source settings (common to all types)
+| Setting | Description |
+|---------|-------------|
+| Name | Display name used in dashboards |
+| Default | If checked, pre-selected in new panels |
+| HTTP URL | The endpoint for the data source server |
+| Auth | Basic auth, TLS, bearer token, API key options |
+
+---
+
+## Prometheus
+
+Native support - no plugin required.
+
+### Configuration
+```ini
+URL: http://prometheus:9090
+HTTP method: POST (recommended, supports longer queries)
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Scrape interval | Default: 15s - should match your Prometheus scrape interval |
+| Query timeout | Default: 60s |
+| Exemplars | Enable to link metrics to traces |
+| Ruler URL | For Prometheus-managed alert rules |
+
+### Query editor
+- **Metrics browser**: Browse available metrics with autocomplete
+- **Query type**: Range query (time series) or Instant query (single value)
+- **Legend**: Customize series labels using `{{label_name}}` syntax
+
+### Template variables with Prometheus
+```
+# Variable type: Query
+# Query examples:
+label_values(metric_name, label_name)    # All values of a label for a metric
+label_values(label_name)                  # All values across all metrics
+metrics(prefix)                           # All metric names matching prefix
+query_result(promql_expression)           # PromQL result as variable values
+```
+
+### Exemplars
+When enabled, Prometheus exemplars link high-cardinality trace IDs to metric data points. Requires a Tempo data source for trace drill-through.
+
+---
+
+## Loki (Log Aggregation)
+
+### Configuration
+```ini
+URL: http://loki:3100
+Maximum lines: 1000
+```
+
+### Derived fields
+Extract values from log lines and link to other systems.
+
+Example - extract trace ID and link to Tempo:
+```
+Name: TraceID
+Regex: traceID=(\w+)
+URL: (internal link to Tempo data source)
+```
+
+### Query editor (LogQL)
+```logql
+# Basic log stream selector
+{job="nginx", namespace="production"}
+
+# Filter by text
+{job="nginx"} |= "error"
+{job="nginx"} != "debug"
+
+# Regex filter
+{job="nginx"} |~ "status=5\d\d"
+
+# Parse and filter structured logs
+{job="api"} | json | level="error"
+{job="api"} | logfmt | duration > 1s
+
+# Metrics from logs
+rate({job="nginx"} |= "error" [5m])
+sum(rate({job="nginx"}[5m])) by (status_code)
+```
+
+### Template variables with Loki
+```
+label_names()                          # All label names
+label_values(label_name)               # All values for a label
+label_values({job="nginx"}, pod)       # Label values filtered by stream selector
+```
+
+---
+
+## Tempo (Distributed Tracing)
+
+### Configuration
+```ini
+URL: http://tempo:3200
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Trace to logs | Link traces to Loki logs via trace ID |
+| Trace to metrics | Link trace spans to Prometheus metrics |
+| Service graph | Enable service dependency graph |
+| Node graph | Show trace as node graph |
+
+### TraceQL (query language)
+```traceql
+# Find traces with error spans
+{status=error}
+
+# Filter by service and duration
+{.service.name="frontend" && duration > 1s}
+
+# Structural queries
+{.http.url=~"/api/.*"} >> {status=error}
+```
+
+---
+
+## Alertmanager
+
+For connecting to an external Alertmanager.
+
+```ini
+URL: http://alertmanager:9093
+# Implementation: Prometheus, Mimir, or Cortex
+```
+
+---
+
+## Elasticsearch / OpenSearch
+
+### Configuration
+```ini
+URL: http://elasticsearch:9200
+Index name: logs-*
+Time field name: @timestamp
+Elasticsearch version: 8.x
+```
+
+---
+
+## MySQL
+
+Built-in support for MySQL 5.7+ and compatible databases (MariaDB, Percona, Amazon Aurora MySQL, Azure Database for MySQL, Google Cloud SQL MySQL).
+
+### Configuration
+```ini
+Host: mysql:3306
+Database: mydb
+User: grafana
+Password: secret
+Max open connections: 100
+Max idle connections: 100
+Connection max lifetime: 14400
+```
+
+### Time series queries
+```sql
+SELECT
+  UNIX_TIMESTAMP(time_col) as time_sec,
+  value_col as value,
+  name_col as metric
+FROM my_table
+WHERE $__timeFilter(time_col)
+ORDER BY time_col ASC
+```
+
+### Macros
+| Macro | Description |
+|-------|-------------|
+| `$__time(column)` | Converts column to Unix timestamp |
+| `$__timeFilter(column)` | Adds WHERE clause for dashboard time range |
+| `$__timeFrom()` | Start of dashboard time range as Unix timestamp |
+| `$__timeTo()` | End of dashboard time range as Unix timestamp |
+| `$__timeGroup(column, interval)` | Groups by time interval |
+| `$__timeGroupAlias(column, interval)` | As above, aliases as "time" |
+| `$__unixEpochFilter(column)` | Time filter for Unix epoch columns |
+| `$__interval` | Auto-calculated interval for current time range |
+
+### Example time series query
+```sql
+SELECT
+  $__timeGroup(created_at, $__interval) AS time,
+  status,
+  count(*) AS cnt
+FROM orders
+WHERE $__timeFilter(created_at)
+GROUP BY 1, 2
+ORDER BY 1
+```
+
+### Annotations
+```sql
+SELECT
+  UNIX_TIMESTAMP(time) AS time,
+  title AS text,
+  tags
+FROM events
+WHERE $__timeFilter(time)
+```
+
+---
+
+## PostgreSQL
+
+### Configuration
+```ini
+Host: postgres:5432
+Database: mydb
+User: grafana
+Password: secret
+SSL mode: disable / require / verify-ca / verify-full
+```
+
+### PostgreSQL time series
+```sql
+SELECT
+  time_bucket('$__interval', time) AS time,
+  avg(value)
+FROM metrics
+WHERE time BETWEEN $__timeFrom() AND $__timeTo()
+GROUP BY 1
+ORDER BY 1
+```
+
+---
+
+## Microsoft SQL Server
+
+### Configuration
+```ini
+Host: sqlserver:1433
+Database: mydb
+User: grafana
+Password: secret
+Encrypt: false / true / disable
+```
+
+---
+
+## InfluxDB
+
+Supports InfluxDB 1.x (InfluxQL) and InfluxDB 2.x / 3.x (Flux).
+
+### InfluxDB 1.x (InfluxQL)
+```ini
+URL: http://influxdb:8086
+Database: telegraf
+```
+
+Query example:
+```sql
+SELECT mean("value") FROM "measurement"
+WHERE $timeFilter
+GROUP BY time($interval), "host"
+```
+
+### InfluxDB 2.x (Flux)
+```ini
+URL: http://influxdb:8086
+Organization: myorg
+Token: <api-token>
+Default bucket: mydata
+```
+
+Flux query example:
+```flux
+from(bucket: "mydata")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r._measurement == "cpu")
+  |> filter(fn: (r) => r._field == "usage_idle")
+  |> aggregateWindow(every: v.windowPeriod, fn: mean)
+```
+
+---
+
+## AWS CloudWatch
+
+### Configuration
+```ini
+# Auth options:
+# - AWS SDK Default (instance role, ~/.aws/credentials, env vars)
+# - Access & secret key (explicit credentials)
+# - Assume role ARN
+
+Default region: us-east-1
+```
+
+Required IAM permissions:
+- `cloudwatch:GetMetricData`
+- `cloudwatch:ListMetrics`
+- `logs:*` (for CloudWatch Logs)
+
+### CloudWatch Logs Insights
+```
+fields @timestamp, @message
+| filter level = "ERROR"
+| sort @timestamp desc
+| limit 100
+```
+
+---
+
+## Azure Monitor
+
+Requires Azure App Registration with:
+- Tenant ID, Client ID, Client Secret
+- `Monitoring Reader` role on target subscriptions/resource groups
+
+---
+
+## Google Cloud Monitoring
+
+Uses Google Cloud service account credentials (JSON key file or workload identity).
+
+---
+
+## TestData (built-in)
+
+A built-in data source for generating test/demo data without a real backend.
+
+Use cases: Testing visualizations, demo dashboards, development without a real data source.
+Scenarios: Random walk, CSV metric values, streaming data, etc.
+
+---
+
+## Graphite
+
+```ini
+URL: http://graphite:8080
+```
+
+Query examples:
+```
+target(servers.*.cpu)
+averageSeries(servers.*.cpu)
+groupByNode(servers.*.cpu, 1, 'averageSeries')
+```
+
+---
+
+## Jaeger (Tracing)
+
+```ini
+URL: http://jaeger:16686
+```
+
+---
+
+## Zipkin (Tracing)
+
+```ini
+URL: http://zipkin:9411
+```
+
+---
+
+## Pyroscope (Continuous Profiling)
+
+```ini
+URL: http://pyroscope:4040
+```
+
+---
+
+## Data Source Permissions (Enterprise)
+
+By default, all org users can query any data source. With RBAC:
+- Assign specific users/teams to specific data sources
+- Configure at **Data source settings > Permissions**
+
+---
+
+## Provisioning Data Sources (as Code)
+
+```yaml
+# /etc/grafana/provisioning/datasources/prometheus.yaml
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    version: 1
+    editable: false
+    jsonData:
+      timeInterval: "15s"
+      queryTimeout: "60s"
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    jsonData:
+      maxLines: 1000
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: "traceID=(\\w+)"
+          name: TraceID
+          url: "${__value.raw}"
+
+  - name: MySQL Production
+    type: mysql
+    url: mysql:3306
+    database: mydb
+    user: grafana
+    secureJsonData:
+      password: "$GRAFANA_MYSQL_PASSWORD"
+    jsonData:
+      maxOpenConns: 100
+      maxIdleConns: 100
+      connMaxLifetime: 14400
+```
+
+Place YAML files in the provisioning directory; Grafana loads on startup and watches for changes.
+
+---
+
+## Plugin Data Sources
+
+Install additional data sources via:
+- **Connections > Add new connection** (search the plugin catalog)
+- `grafana-cli plugins install <plugin-id>`
+- Docker: set `GF_INSTALL_PLUGINS` environment variable
+
+Popular plugin data sources:
+- `grafana-opensearch-datasource` - OpenSearch
+- `grafana-bigquery-datasource` - Google BigQuery
+- `grafana-mongodb-datasource` - MongoDB (Enterprise)
+- `grafana-splunk-datasource` - Splunk
+- `grafana-datadog-datasource` - Datadog

.agents/skills/grafana-oss/references/panel-types.md

@@ -0,0 +1,29 @@
+# Panel types
+
+| Panel | Use Case |
+|-------|----------|
+| **Time series** | Line/bar charts over time (default for metrics) |
+| **Stat** | Single value with color thresholds |
+| **Gauge** | Radial gauge for current value |
+| **Bar gauge** | Horizontal bars for comparisons |
+| **Table** | Tabular data, sortable columns |
+| **Logs** | Log stream viewer (Loki) |
+| **Traces** | Trace visualization (Tempo) |
+| **Heatmap** | Distribution over time |
+| **Histogram** | Value distribution |
+| **Pie chart** | Part-to-whole ratios |
+| **Geomap** | Geographic data |
+| **Canvas** | Custom SVG-based layouts |
+| **Node graph** | Service/topology graphs |
+| **Flame graph** | CPU/memory profiling |
+| **Text** | Markdown/HTML content |
+| **Alert list** | Show firing alerts |
+
+## Picking a panel type
+
+Default is **Time series** for almost any metric-over-time visualization. Switch from the default only when:
+
+- The user wants **one number, not a series** → Stat / Gauge / Bar gauge
+- The data is **categorical, not temporal** → Pie chart / Bar gauge / Table
+- The data is **logs/traces/profiles** → Logs / Traces / Flame graph (per data type)
+- The data is **2D distribution** (e.g. histogram-over-time) → Heatmap

.agents/skills/grafana-oss/references/panels.md

@@ -0,0 +1,325 @@
+# Grafana Panels and Visualizations - Detailed Reference
+
+## What is a Panel?
+
+A panel is the basic building block of a Grafana dashboard. Each panel combines:
+- A **query** (or multiple queries) to a data source
+- A **visualization type** to display the data
+- **Field configuration** (units, thresholds, color mappings)
+- Optional **transformations** to reshape the data before display
+
+---
+
+## Panel Editor
+
+Open the panel editor by clicking a panel's title > Edit, or clicking **Add visualization** for a new panel.
+
+### Panel Editor Layout
+
+**Top bar**: Back to dashboard, Discard changes, Save dashboard
+
+**Center**: Visualization preview (live update as you configure)
+
+**Toggle**: "Table view" - shows raw query results as a table for debugging
+
+**Right sidebar tabs**:
+1. **Query** - configure data sources and write queries
+2. **Transform** - apply data transformations
+3. **Alert** - create alert rules from this panel
+4. Below tabs: visualization-specific options and field config
+
+### Query Tab
+
+- **Data source selector**: Choose which data source to query
+- **Query editor**: Data-source-specific interface (PromQL for Prometheus, LogQL for Loki, SQL for databases)
+- **Query options**:
+  - Max data points: Limit data points fetched
+  - Min interval: Minimum auto-calculated interval
+  - Interval: Override the auto-calculated interval
+  - Relative time: Override dashboard time range for this panel only
+  - Time shift: Shift the time range (e.g., to compare to last week)
+- **+ Query**: Add multiple queries (labeled A, B, C...)
+- **Expression**: Add server-side math expressions combining query results
+
+---
+
+## Visualization Types
+
+### Time Series (default)
+Best for: Metrics over time, continuous data
+- Renders as lines, points, or bars
+- Supports multiple series
+- Configurable line width, fill, point size
+- Supports thresholds as colored background regions
+- Supports annotations overlay
+- **Graph styles**: Lines, Bars, Points; can mix per series
+- **Stacking**: None, Normal, 100%
+- **Axis**: Left Y, Right Y, hidden
+- Default visualization - supports alerting
+
+### Stat
+Best for: Single important metric (KPI display)
+- Shows one or more values as large text
+- Supports sparkline background
+- Color modes: value, background, none
+- Can show last value, mean, sum, etc.
+- Great for dashboards that need quick status overviews
+
+### Bar Chart
+Best for: Comparing categorical data
+- Horizontal or vertical orientation
+- Grouped or stacked
+- Supports labels on bars
+
+### Gauge
+Best for: Showing a value relative to min/max range
+- Circular gauge with arc
+- Configurable thresholds set colors
+- Shows current value prominently
+
+### Bar Gauge
+Best for: Multiple metrics as horizontal/vertical bars
+- Useful for comparing many items
+- Supports thresholds for color coding
+- Modes: gradient, retro LCD, basic
+
+### Table
+Best for: Tabular data, multi-column metrics
+- Supports sorting by column
+- Column width customization
+- Cell display modes: color text, color background, gradient gauge
+- Pagination for large datasets
+- Can embed sparklines in cells
+
+### Heatmap
+Best for: Distribution over time, histogram-over-time
+- X axis: time; Y axis: buckets; Color: density/value
+- Supports pre-bucketed data (Prometheus histogram) and raw values
+- Tooltip shows exact bucket counts
+
+### Histogram
+Best for: Value distribution analysis
+- Groups values into buckets
+- Can combine multiple series
+- Configurable bucket size
+
+### Pie Chart
+Best for: Proportional data, parts-of-whole
+- Pie or donut style
+- Labels: name, value, percentage
+
+### Logs
+Best for: Log data from Loki, Elasticsearch, etc.
+- Displays raw log lines with timestamp
+- Log level coloring (info/warn/error)
+- Search/filter within results
+- Deduplication and time wrapping
+- Prettify JSON option
+
+### Traces
+Best for: Distributed tracing visualization (Tempo)
+- Renders trace spans as a waterfall/Gantt chart
+- Shows service name, operation, duration
+- Click to drill into trace details
+
+### Flame Graph
+Best for: CPU profiling data (Pyroscope)
+- Visualizes call stacks by CPU time
+- Click to zoom into subtrees
+
+### Node Graph
+Best for: Service dependency maps, network topology
+- Renders nodes and edges
+- Node color/size configurable by metric
+
+### Geomap
+Best for: Geographic data visualization
+- Layers: markers, heatmap, route
+- Multiple base map options (OpenStreetMap, CARTO, etc.)
+- Supports GeoJSON data
+
+### Canvas
+Best for: Custom layouts, process diagrams, status boards
+- Drag-and-drop element placement
+- Elements: text, metric value, rectangle, ellipse, icon, image, connections
+- Dynamic data binding per element
+
+### State Timeline
+Best for: State changes over time (on/off, OK/warn/crit)
+- Horizontal bands showing state duration
+- Each series = one row
+- Color per state value
+
+### Status History
+Best for: Periodic state checks over time
+- Grid: Y=services, X=time buckets
+- Color per state
+
+### XY Chart
+Best for: Correlation between two metrics
+- Scatter plot
+- X and Y axis from different fields
+- Bubble size from a third field
+
+### Candlestick
+Best for: Financial OHLC data
+- Open/High/Low/Close representation
+- Volume bars
+
+### Text
+Best for: Documentation panels, headers
+- Renders Markdown or HTML
+
+### Alert List
+Best for: Dashboard overview of current alert states
+
+### Dashboard List
+Best for: Navigation panels linking to other dashboards
+
+---
+
+## Field Configuration (Standard Options)
+
+Available for most visualizations under the visualization options panel.
+
+| Option | Description |
+|--------|-------------|
+| Unit | Display unit (bytes, seconds, %, requests/sec, etc.) |
+| Min / Max | Override auto-detected min/max for scales |
+| Decimals | Number of decimal places |
+| Display name | Override the series/field name |
+| Color scheme | Fixed, thresholds-based, palette, etc. |
+| No value | Text to display when value is null |
+
+### Thresholds
+Define color-coded boundaries:
+- **Absolute**: Fixed numeric values (e.g., >90 = red, >70 = yellow, else green)
+- **Percentage**: Relative to min/max
+
+Example threshold config:
+```
+Base (default): Green
+70: Yellow (warn)
+90: Red (crit)
+```
+
+### Value Mappings
+Transform raw values into human-readable labels or colors:
+- **Value to text**: e.g., `1` -> "OK", `0` -> "Down"
+- **Range to text**: e.g., `0-50` -> "Low"
+- **Regex to text**: Match patterns
+
+### Data Links
+Create clickable links from panel values:
+- Link to other dashboards with variable values interpolated
+- Link to external systems (e.g., Kibana, PagerDuty)
+- Use `${__value.raw}` and `${__field.name}` in URLs
+
+---
+
+## Field Overrides
+
+Apply specific field options to individual series/columns rather than all data:
+
+1. Click **+ Add field override** in the Overrides section
+2. Choose override target:
+   - Fields with name (exact match)
+   - Fields with name matching regex
+   - Fields with type (number, string, time)
+   - Fields returned by query (A, B, C...)
+3. Add properties to override (unit, color, alias, thresholds, etc.)
+
+Example: In a table with columns `cpu_idle` and `cpu_used`, set `cpu_used` to show as percentage and color by threshold while leaving `cpu_idle` with default styling.
+
+---
+
+## Transformations
+
+Transformations reshape query data before rendering. Apply multiple in sequence.
+
+Access: Panel editor > Transform tab > Add transformation
+
+### Most-Used Transformations
+
+| Transformation | Description |
+|----------------|-------------|
+| **Reduce** | Collapse a time series to a single value (last, mean, sum, max, min) |
+| **Filter by name** | Keep only specific fields/columns |
+| **Filter by value** | Filter rows where a field matches a condition |
+| **Organize fields** | Rename, reorder, or hide fields |
+| **Merge** | Combine multiple query results into one table |
+| **Join by field** | SQL-style join on a common field (e.g., time) |
+| **Group by** | Group rows and aggregate (count, sum, mean, etc.) |
+| **Sort by** | Sort rows by a field |
+| **Limit** | Keep only first N rows |
+| **Add field from calculation** | Add a new column computed from existing columns |
+| **Convert field type** | Change a field's data type |
+| **Rename by regex** | Batch rename fields using regex |
+| **Extract fields** | Parse JSON or regex from a string field |
+| **Labels to fields** | Convert label key/values into separate columns |
+| **Rows to fields** | Pivot: turn row values into column headers |
+| **Prepare time series** | Normalize time series format |
+| **Time series to table** | Convert time series format to table format |
+
+Enable "Debug" toggle on any transformation to see input/output for troubleshooting.
+
+---
+
+## Query Options
+
+### Multiple queries
+Add multiple queries (A, B, C...) to one panel. They are overlaid in the visualization. Use this to compare metrics or show multiple services on one graph.
+
+### Expressions (server-side)
+Server-side expressions allow math across query results:
+- **Math**: `$A + $B`, `$A / $B * 100`
+- **Reduce**: Collapse a series to scalar
+- **Resample**: Change time resolution of a series
+- **Classic conditions**: Threshold logic (used in alerting)
+
+Example - calculate error rate as percentage:
+```
+Query A: total_requests{job="api"}
+Query B: error_requests{job="api"}
+Expression C (Math): $B / $A * 100
+Display C as percentage
+```
+
+### Important query variables
+These variables are available in queries:
+
+| Variable | Description |
+|----------|-------------|
+| `$__interval` | Auto-calculated interval based on time range and resolution |
+| `$__rate_interval` | Interval suitable for rate() functions (at least 4x scrape interval) |
+| `$__from` | Start of the current time range (ms epoch) |
+| `$__to` | End of the current time range (ms epoch) |
+| `$__range` | Duration of current time range (e.g., "6h") |
+| `$__range_s` | Duration in seconds |
+| `$__range_ms` | Duration in milliseconds |
+
+Example PromQL using interval variable:
+```promql
+rate(http_requests_total[${__rate_interval}])
+```
+
+---
+
+## Panel Inspect
+
+Click panel menu (3-dot) > **Inspect** to access:
+- **Data**: Raw table view of the data powering the panel
+- **Stats**: Query performance (time, row count)
+- **JSON**: Panel JSON model
+- **Query**: Equivalent of query inspector showing raw query and response
+
+---
+
+## Performance Tips
+
+1. Limit max data points to avoid over-fetching
+2. Use recording rules in Prometheus for expensive queries
+3. Set longer min intervals for historical dashboards
+4. Use `$__interval` variable in queries to align with time resolution
+5. Use `$__rate_interval` instead of hardcoded intervals for rate() queries
+6. Avoid too many panels on one dashboard (aim for <20)

.agents/skills/marktguru/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: marktguru
+license: Apache-2.0
+description: Query weekly supermarket offers, flyer discounts, and price comparisons across German supermarkets (ALDI Süd, ALDI Nord, Lidl, EDEKA, REWE, Kaufland, PENNY, Netto, dm, ROSSMANN, etc.) via the Marktguru API and local MCP server. Use when the user asks to check supermarket discounts, find deals in brochures/prospekte, compare grocery prices for a postal code (PLZ), or create shopping lists from current promotions without needing fragile PDF OCR.
+---
+
+# Marktguru Supermarket Offers & MCP Server
+
+> Query live promotional flyers and circular offers from German supermarkets and drugstores without brittle PDF scraping or layout OCR.
+
+This skill is inspired by and expands upon the Home Assistant integration pattern (`thorschtn/ha-marktguru`), leveraging Marktguru's backend API (`api.marktguru.de`) to retrieve structured deal data for any German postal code.
+
+---
+
+## Supported Supermarkets & Retailers
+
+Marktguru covers **over 450 retail chains** across Germany. The primary supported chains are:
+
+*   **Discounter**:
+    *   **ALDI SÜD** (`127153`) & **ALDI NORD** (`127139`)
+    *   **Lidl** (`126679`)
+    *   **PENNY** (`126765`)
+    *   **Netto Marken-Discount** (`126735`) & **Netto (mit Hund)** (`126734`)
+    *   **NORMA** (`126745`)
+*   **Supermärkte & Vollsortimenter**:
+    *   **EDEKA** (`127093`) & **E center** (`126699`) + regionale Kaufleute (Simmel, Zurheide, etc.)
+    *   **REWE** (`126802`) & **REWE Center** (`126804`)
+    *   **nahkauf** (`137320`), **HIT** (`126598`), **tegut** (`126866`)
+*   **Großfläche / Hypermarket**:
+    *   **Kaufland** (`126654`)
+    *   **Globus** (`127177`)
+*   **Drogerien & Reform**:
+    *   **dm-drogerie markt** (`127057`)
+    *   **ROSSMANN** (`126811`)
+    *   **Müller Drogeriemarkt** (`126720`)
+
+Full details and ID mappings are documented in [`references/supermarkets.md`](references/supermarkets.md).
+
+---
+
+## Components
+
+The skill provides two primary components:
+
+1.  **CLI & Python Client (`scripts/marktguru_client.py`)**: Zero external dependencies (uses standard library `urllib` and `json`).
+2.  **MCP Server (`scripts/marktguru_mcp.py`)**: Standard JSON-RPC 2.0 stdio server providing AI tools for search, comparison, and catalog retrieval.
+
+---
+
+## 1. CLI Usage
+
+Run directly with standard `python3` (uses default postal code **`93059` Regensburg** if `--zip` is omitted):
+
+### A. Search for Deals
+```bash
+# Search for butter in Regensburg (93059 default) across Aldi, Lidl, and Edeka
+python3 scripts/marktguru_client.py search "Butter" --supermarket "Aldi,Lidl,Edeka"
+
+# Search for coffee with raw JSON output
+python3 scripts/marktguru_client.py search "Kaffee" --json
+```
+
+### B. Price Comparison (Sorted Ascending)
+```bash
+# Compare chocolate prices across all supermarkets in 93059
+python3 scripts/marktguru_client.py compare "Schokolade"
+
+# Compare beer prices between Lidl and Rewe in another postal code (e.g. 80331)
+python3 scripts/marktguru_client.py compare "Bier" --zip 80331 --supermarket "Lidl,Rewe"
+```
+
+### C. List Supported Chains
+```bash
+python3 scripts/marktguru_client.py supermarkets
+```
+
+---
+
+## 2. Model Context Protocol (MCP) Integration
+
+The MCP server exposes three tools to any MCP-compatible AI agent:
+
+*   `search_supermarket_offers(query, zip_code="93059", supermarkets=None, limit=20)`
+*   `compare_supermarket_prices(query, zip_code="93059", supermarkets=None)`
+*   `list_supported_supermarkets()`
+
+### MCP Server Configuration (`mcp_config.json`)
+
+To enable the server in Antigravity or other MCP hosts:
+
+```json
+{
+  "mcpServers": {
+    "marktguru": {
+      "command": "python3",
+      "args": [
+        "/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts/marktguru_mcp.py"
+      ]
+    }
+  }
+}
+```
+
+---
+
+## 3. Python Integration
+
+You can import the module directly in Python scripts or agent sidecars:
+
+```python
+import sys
+sys.path.append("/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts")
+from marktguru_client import search_offers, compare_prices, DEFAULT_ZIP_CODE
+
+# Search deals (defaults to 93059)
+results = search_offers("Kaffee", supermarkets=["Aldi", "Lidl"])
+for offer in results["offers"]:
+    print(f"[{offer['supermarket']}] {offer['product_name'] or offer['description']}: {offer['price']} €")
+
+# Compare prices
+comparison = compare_prices("Milch")
+cheapest = comparison["cheapest_offer"]
+print(f"Cheapest milk: {cheapest['price']} € at {cheapest['supermarket']}")
+```
+
+---
+
+## Data Extracted Per Offer
+
+Each item returned from the Marktguru API includes:
+*   `id`: Internal Marktguru offer ID.
+*   `supermarket`: Retailer name (e.g. "ALDI SÜD", "Lidl", "EDEKA").
+*   `product_name`: Generic product name (e.g. "Butter", "Filterkaffee").
+*   `brand`: Brand name (e.g. "Kerrygold", "Melitta", "Milka").
+*   `description`: Promotional description / pack size (e.g. "je 250g Packung").
+*   `price`: Promotional sale price in Euros (€).
+*   `old_price`: Regular price or crossed-out price (UVP).
+*   `discount_percent`: Calculated percentage savings (e.g. 28%).
+*   `reference_price`: Unit price (e.g. 5.16 € / kg).
+*   `loyalty_app_required`: Boolean flag indicating if a digital store app (Lidl Plus, Edeka App, Penny App) is required to receive the discount.
+*   `valid_from` & `valid_to`: Start and end dates of the promotion.
+*   `image_url`: High-resolution product image hosted on CDN.

.agents/skills/marktguru/references/supermarkets.md

@@ -0,0 +1,94 @@
+# Unterstützte Supermärkte & Händler (Marktguru)
+
+Über die Marktguru-Schnittstelle können Werbeprospekte, Wochenangebote und Rabattaktionen von über 450 Handelsketten und Einzelhändlern in Deutschland abgefragt werden.
+
+---
+
+## 1. Discounter
+
+Discounter machen einen Großteil der wöchentlich wechselnden Prospektaktionen aus. Marktguru unterscheidet sauber zwischen den Ketten:
+
+| Kette | Marktguru ID | Typischer Aktionsrhythmus | App-Vorteile erfasst? |
+| :--- | :--- | :--- | :--- |
+| **ALDI SÜD** | `127153` | Mo. & Do. / Fr. & Sa. | Ja (ALDI App Angebote) |
+| **ALDI NORD** | `127139` | Mo. & Do. / Fr. & Sa. | Ja |
+| **Lidl** | `126679` | Mo.–Sa. / Do.–Sa. / Fr.–Sa. | Ja (`loyalty_app_required` für Lidl Plus) |
+| **PENNY** | `126765` | Mo.–Sa. / Do.–Sa. | Ja (Penny App / Payback) |
+| **Netto Marken-Discount** | `126735` | Mo.–Sa. / Do.–Sa. | Ja (Netto App / DeutschlandCard) |
+| **Netto (mit Hund)** | `126734` | Mo.–Sa. (primär Ost-/Norddeutschland) | Ja |
+| **NORMA** | `126745` | Mo. & Mi. / Fr. & Sa. | Ja |
+
+---
+
+## 2. Vollsortimenter & Supermärkte
+
+Hier gibt es oft filialgenaue oder regionale Prospekte. Die API filtert automatisch anhand der übergebenen Postleitzahl (PLZ):
+
+| Kette | Marktguru ID | Besonderheiten |
+| :--- | :--- | :--- |
+| **EDEKA** | `127093` | Nationale & regionale Angebote; Marktguru aggregiert auch selbstständige Kaufleute (z.B. EDEKA Simmel, Zurheide, Niemerszein). |
+| **E center** | `126699` | Großformatige EDEKA-Märkte mit erweitertem Non-Food- und Frischeangebot. |
+| **REWE** | `126802` | Bundesweite Angebote sowie regionale Aktionen. |
+| **REWE Center** | `126804` | Großflächenmärkte von REWE. |
+| **nahkauf** | `137320` | Nachbarschaftsmärkte der REWE Group. |
+| **HIT** | `126598` | Große Frische-Supermärkte (Dohle Handelsgruppe). |
+| **tegut** | `126866` | Fokus auf Bio, Nachhaltigkeit und Regionales (Hessen, Bayern, Thüringen). |
+
+---
+
+## 3. Großfläche & SB-Warenhäuser (Hypermarkets)
+
+| Kette | Marktguru ID | Besonderheiten |
+| :--- | :--- | :--- |
+| **Kaufland** | `126654` | Riesiges wöchentliches Prospektangebot (Lebensmittel + Non-Food, Kaufland Card). |
+| **Globus** | `127177` | SB-Warenhäuser mit eigener Metzgerei und Bäckerei. |
+
+---
+
+## 4. Drogeriemärkte
+
+Drogerieprospekte enthalten neben Pflege- und Haushaltsartikeln auch viele Bio-Lebensmittel, Snacks, Kaffee und Babynahrung:
+
+| Kette | Marktguru ID | Typische Angebote |
+| :--- | :--- | :--- |
+| **dm-drogerie markt** | `127057` | dmBio, Drogeriewaren, Tiernahrung. |
+| **ROSSMANN** | `126811` | EnerBio, Kaffeeangebote, wöchentliche 10%-Coupons. |
+| **Müller Drogeriemarkt** | `126720` | Drogerie, Parfümerie, Spielwaren, Naturshop. |
+| **budni** | `126944` | Primär im Norden (Metropolregion Hamburg). |
+
+---
+
+## 5. Getränkemärkte & Bio-Fachmärkte
+
+| Kette | Marktguru ID | Sortiment |
+| :--- | :--- | :--- |
+| **Netto Getränke-Discount** | `134009` | Bier, Mineralwasser, Softdrinks. |
+| **Denns BioMarkt / Alnatura** | Diverse | Bio-Vollsortiment, Wochenaktionen. |
+
+---
+
+## 6. Unterstützte Aliase im Client & MCP Server
+
+Wenn im Tool-Aufruf der Parameter `supermarkets` übergeben wird, werden folgende umgangssprachliche Bezeichnungen automatisch auf die korrekten Retailer-IDs gemappt:
+
+```json
+{
+  "aldi": [127153, 127139],       // Findet automatisch ALDI SÜD und ALDI NORD
+  "aldi süd" / "aldi sued": [127153],
+  "aldi nord": [127139],
+  "lidl": [126679],
+  "edeka": [127093, 126699],      // EDEKA und E center
+  "rewe": [126802, 126804],       // REWE und REWE Center
+  "kaufland": [126654],
+  "penny": [126765],
+  "netto": [126735, 126734],      // Netto Marken-Discount und Netto (mit Hund)
+  "norma": [126745],
+  "globus": [127177],
+  "dm": [127057],
+  "rossmann": [126811],
+  "müller" / "mueller": [126720],
+  "hit": [126598],
+  "tegut": [126866],
+  "nahkauf": [137320]
+}
+```

.agents/skills/marktguru/scripts/marktguru_client.py

@@ -0,0 +1,577 @@
+#!/usr/bin/env python3
+"""
+Marktguru API Client (Production Grade)
+Fetches promotional offers and discounts from German supermarkets via the Marktguru API.
+
+Key Design Principles:
+- Zero external dependencies (uses standard library only: urllib, json, os, time, datetime).
+- Resilient: Retries with exponential backoff on transient errors.
+- Low Maintenance: Configurable keys and postal codes via environment variables.
+- Fail-Safe: Caching with stale fallback if upstream API is temporarily unreachable.
+- Defensive: Robust parsing protecting against upstream schema shifts and malformed payloads.
+"""
+
+import os
+import sys
+import time
+import json
+import logging
+import hashlib
+import tempfile
+import urllib.request
+import urllib.parse
+import urllib.error
+from datetime import datetime, timezone
+from typing import List, Dict, Any, Optional, Set, Tuple
+
+# Logger writing to stderr to keep stdout pure for JSON / CLI
+logger = logging.getLogger("marktguru_client")
+if not logger.handlers:
+    handler = logging.StreamHandler(sys.stderr)
+    handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
+    logger.addHandler(handler)
+    logger.setLevel(logging.INFO)
+
+API_BASE = "https://api.marktguru.de/api/v1"
+
+# Environment variable overrides for zero-code configuration changes
+DEFAULT_ZIP_CODE = os.environ.get("MARKTGURU_ZIP_CODE", "93059")
+CLIENT_KEY = os.environ.get("MARKTGURU_CLIENT_KEY", "WU/RH+PMGDi+gkZer3WbMelt6zcYHSTytNB7VpTia90=")
+API_KEY = os.environ.get("MARKTGURU_API_KEY", "8Kk+pmbf7TgJ9nVj2cXeA7P5zBGv8iuutVVMRfOfvNE=")
+CACHE_TTL = int(os.environ.get("MARKTGURU_CACHE_TTL", "1800"))  # 30 minutes
+CACHE_ENABLED = os.environ.get("MARKTGURU_CACHE_ENABLED", "true").lower() in ("true", "1", "yes")
+
+HEADERS = {
+    "x-clientkey": CLIENT_KEY,
+    "x-apikey": API_KEY,
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)",
+    "Accept": "application/json",
+}
+
+# Major supermarket chains and their Marktguru retailer IDs
+MAJOR_RETAILERS: Dict[str, Dict[str, Any]] = {
+    "ALDI SÜD": {"id": 127153, "type": "Discounter", "major": True},
+    "ALDI NORD": {"id": 127139, "type": "Discounter", "major": True},
+    "Lidl": {"id": 126679, "type": "Discounter", "major": True},
+    "EDEKA": {"id": 127093, "type": "Supermarkt", "major": True},
+    "E center": {"id": 126699, "type": "Supermarkt", "major": True},
+    "REWE": {"id": 126802, "type": "Supermarkt", "major": True},
+    "REWE Center": {"id": 126804, "type": "Supermarkt", "major": True},
+    "Kaufland": {"id": 126654, "type": "Großfläche/Hypermarkt", "major": True},
+    "PENNY": {"id": 126765, "type": "Discounter", "major": True},
+    "Netto Marken-Discount": {"id": 126735, "type": "Discounter", "major": True},
+    "Netto (mit Hund)": {"id": 126734, "type": "Discounter", "major": False},
+    "NORMA": {"id": 126745, "type": "Discounter", "major": True},
+    "Globus": {"id": 127177, "type": "Großfläche/Hypermarkt", "major": True},
+    "HIT": {"id": 126598, "type": "Supermarkt", "major": False},
+    "tegut": {"id": 126866, "type": "Supermarkt", "major": False},
+    "nahkauf": {"id": 137320, "type": "Supermarkt", "major": False},
+    "dm-drogerie markt": {"id": 127057, "type": "Drogerie", "major": True},
+    "ROSSMANN": {"id": 126811, "type": "Drogerie", "major": True},
+    "Müller Drogeriemarkt": {"id": 126720, "type": "Drogerie", "major": True},
+}
+
+# Aliases mapping user query strings to Marktguru retailer IDs
+RETAILER_ALIASES: Dict[str, List[int]] = {
+    "aldi": [127153, 127139],
+    "aldi süd": [127153],
+    "aldi sued": [127153],
+    "aldi nord": [127139],
+    "lidl": [126679],
+    "edeka": [127093, 126699],
+    "e center": [126699],
+    "ecenter": [126699],
+    "rewe": [126802, 126804],
+    "kaufland": [126654],
+    "penny": [126765],
+    "netto": [126735, 126734],
+    "norma": [126745],
+    "globus": [127177],
+    "hit": [126598],
+    "tegut": [126866],
+    "nahkauf": [137320],
+    "dm": [127057],
+    "rossmann": [126811],
+    "müller": [126720],
+    "mueller": [126720],
+}
+
+
+def _get_cache_dir() -> str:
+    """Returns directory path for local cached responses."""
+    cache_dir = os.path.join(tempfile.gettempdir(), "marktguru_cache")
+    try:
+        os.makedirs(cache_dir, exist_ok=True)
+    except Exception:
+        pass
+    return cache_dir
+
+
+def _read_cache(cache_key: str, max_age_seconds: int = CACHE_TTL) -> Tuple[Optional[Dict[str, Any]], bool]:
+    """
+    Reads cache file. Returns (data, is_fresh).
+    If cache is expired, returns (data, False) so caller can use it as emergency fallback.
+    """
+    if not CACHE_ENABLED:
+        return None, False
+
+    cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
+    if not os.path.isfile(cache_file):
+        return None, False
+
+    try:
+        with open(cache_file, "r", encoding="utf-8") as f:
+            payload = json.load(f)
+        cached_time = payload.get("_cached_at", 0)
+        age = time.time() - cached_time
+        data = payload.get("data")
+        is_fresh = age < max_age_seconds
+        return data, is_fresh
+    except Exception as e:
+        logger.debug(f"Failed to read cache {cache_key}: {e}")
+        return None, False
+
+
+def _write_cache(cache_key: str, data: Dict[str, Any]) -> None:
+    """Writes data to cache file atomically."""
+    if not CACHE_ENABLED:
+        return
+
+    cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
+    temp_file = f"{cache_file}.tmp.{os.getpid()}"
+    try:
+        payload = {
+            "_cached_at": time.time(),
+            "data": data,
+        }
+        with open(temp_file, "w", encoding="utf-8") as f:
+            json.dump(payload, f, ensure_ascii=False)
+        os.replace(temp_file, cache_file)
+    except Exception as e:
+        logger.debug(f"Failed to write cache {cache_key}: {e}")
+        if os.path.exists(temp_file):
+            try:
+                os.remove(temp_file)
+            except Exception:
+                pass
+
+
+def resolve_supermarket_ids(supermarkets: Optional[List[str]]) -> Optional[str]:
+    """Resolves human supermarket names/aliases to comma-separated Marktguru retailer IDs."""
+    if not supermarkets:
+        return None
+
+    # Handle string passed instead of list (e.g. "Aldi, Lidl")
+    if isinstance(supermarkets, str):
+        supermarkets = [s.strip() for s in supermarkets.split(",") if s.strip()]
+
+    resolved_ids: Set[int] = set()
+    for name in supermarkets:
+        if not isinstance(name, str):
+            continue
+        cleaned = name.strip().lower()
+        if not cleaned:
+            continue
+
+        if cleaned in RETAILER_ALIASES:
+            resolved_ids.update(RETAILER_ALIASES[cleaned])
+        elif cleaned.isdigit():
+            resolved_ids.add(int(cleaned))
+        else:
+            found = False
+            for r_name, r_info in MAJOR_RETAILERS.items():
+                if cleaned in r_name.lower():
+                    resolved_ids.add(r_info["id"])
+                    found = True
+            if not found:
+                logger.warning(f"Unknown supermarket name '{name}' skipped.")
+
+    return ",".join(str(i) for i in sorted(resolved_ids)) if resolved_ids else None
+
+
+def format_iso_date(iso_str: Optional[str]) -> str:
+    """Formats an ISO-8601 date string to human-readable DD.MM.YYYY format."""
+    if not iso_str or not isinstance(iso_str, str):
+        return "-"
+    try:
+        dt = datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
+        return dt.strftime("%d.%m.%Y")
+    except Exception:
+        return iso_str[:10] if len(iso_str) >= 10 else iso_str
+
+
+def parse_offer(raw: Any, zip_code: str) -> Optional[Dict[str, Any]]:
+    """
+    Defensively parses and standardizes an offer payload from Marktguru.
+    Protects against null fields, malformed dates, and missing keys.
+    """
+    if not isinstance(raw, dict):
+        return None
+
+    try:
+        offer_id = raw.get("id")
+        price = raw.get("price")
+        if isinstance(price, (int, float)):
+            price = round(float(price), 2)
+        else:
+            price = None
+
+        old_price = raw.get("oldPrice")
+        if isinstance(old_price, (int, float)):
+            old_price = round(float(old_price), 2)
+        else:
+            old_price = None
+
+        ref_price = raw.get("referencePrice")
+        if isinstance(ref_price, (int, float)):
+            ref_price = round(float(ref_price), 2)
+        else:
+            ref_price = None
+
+        discount_pct = None
+        if old_price and price and old_price > price:
+            discount_pct = round(((old_price - price) / old_price) * 100)
+
+        # Validity dates
+        validity = raw.get("validityDates")
+        valid_from_raw = None
+        valid_to_raw = None
+        is_valid_now = True
+        days_remaining = None
+
+        if isinstance(validity, list) and len(validity) > 0 and isinstance(validity[0], dict):
+            valid_from_raw = validity[0].get("from")
+            valid_to_raw = validity[0].get("to")
+
+            try:
+                now = datetime.now(timezone.utc)
+                if valid_to_raw and isinstance(valid_to_raw, str):
+                    dt_to = datetime.fromisoformat(valid_to_raw.replace("Z", "+00:00"))
+                    diff = (dt_to - now).days
+                    days_remaining = max(0, diff)
+                    if dt_to < now:
+                        is_valid_now = False
+
+                if valid_from_raw and isinstance(valid_from_raw, str):
+                    dt_from = datetime.fromisoformat(valid_from_raw.replace("Z", "+00:00"))
+                    if dt_from > now:
+                        is_valid_now = False
+            except Exception:
+                pass
+
+        # Supermarket / advertiser extraction
+        supermarket = "Unbekannt"
+        supermarket_id = None
+        advertisers = raw.get("advertisers")
+        if isinstance(advertisers, list) and len(advertisers) > 0 and isinstance(advertisers[0], dict):
+            supermarket = advertisers[0].get("name") or "Unbekannt"
+            supermarket_id = advertisers[0].get("id")
+
+        # High resolution CDN image
+        image_url = None
+        if offer_id:
+            image_url = f"https://mg2de.b-cdn.net/api/v1/offers/{offer_id}/images/default/0/medium.webp"
+
+        # Unit & volume
+        unit_info = raw.get("unit")
+        unit_name = None
+        if isinstance(unit_info, dict):
+            unit_name = unit_info.get("shortName") or unit_info.get("name")
+
+        product_info = raw.get("product")
+        product_name = None
+        if isinstance(product_info, dict):
+            product_name = product_info.get("name")
+
+        brand_info = raw.get("brand")
+        brand_name = None
+        if isinstance(brand_info, dict):
+            brand_name = brand_info.get("name")
+
+        return {
+            "id": offer_id,
+            "supermarket": supermarket,
+            "supermarket_id": supermarket_id,
+            "product_name": product_name,
+            "brand": brand_name,
+            "description": raw.get("description"),
+            "price": price,
+            "old_price": old_price,
+            "reference_price": ref_price,
+            "discount_percent": discount_pct,
+            "unit": unit_name,
+            "volume": raw.get("volume"),
+            "loyalty_app_required": bool(raw.get("requiresLoyalityMembership", False)),
+            "is_valid_now": is_valid_now,
+            "days_remaining": days_remaining,
+            "valid_from": format_iso_date(valid_from_raw),
+            "valid_to": format_iso_date(valid_to_raw),
+            "image_url": image_url,
+            "zip_code": str(zip_code),
+        }
+    except Exception as e:
+        logger.debug(f"Error parsing single offer {raw.get('id') if isinstance(raw, dict) else raw}: {e}")
+        return None
+
+
+def _fetch_with_retry(url: str, max_retries: int = 2) -> Dict[str, Any]:
+    """
+    Executes HTTP request with exponential backoff on transient errors (5xx / network blips).
+    """
+    last_err: Optional[Exception] = None
+
+    for attempt in range(max_retries + 1):
+        req = urllib.request.Request(url, headers=HEADERS)
+        try:
+            with urllib.request.urlopen(req, timeout=10) as resp:
+                raw_bytes = resp.read()
+                return json.loads(raw_bytes.decode("utf-8"))
+        except urllib.error.HTTPError as e:
+            last_err = e
+            # Retry only on transient server errors (500, 502, 503, 504) or rate limits (429)
+            if e.code in (429, 500, 502, 503, 504) and attempt < max_retries:
+                backoff = 0.5 * (2 ** attempt)
+                logger.warning(f"Marktguru HTTP {e.code}. Retrying in {backoff:.1f}s (attempt {attempt+1}/{max_retries})...")
+                time.sleep(backoff)
+                continue
+            raise
+        except (urllib.error.URLError, TimeoutError, OSError) as e:
+            last_err = e
+            if attempt < max_retries:
+                backoff = 0.5 * (2 ** attempt)
+                logger.warning(f"Connection issue ({e}). Retrying in {backoff:.1f}s...")
+                time.sleep(backoff)
+                continue
+            raise
+
+    raise last_err or RuntimeError("Max retries exceeded")
+
+
+def search_offers(
+    query: str,
+    zip_code: str = DEFAULT_ZIP_CODE,
+    supermarkets: Optional[List[str]] = None,
+    limit: int = 24,
+    offset: int = 0,
+    force_refresh: bool = False,
+) -> Dict[str, Any]:
+    """
+    Searches current supermarket offers in Germany via the Marktguru API.
+    
+    Args:
+        query: Search term (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch')
+        zip_code: German postal code (default: '93059' Regensburg)
+        supermarkets: List of supermarket names or aliases (e.g. ['Aldi', 'Lidl', 'Edeka'])
+        limit: Max number of items to return (default: 24, max: 50)
+        offset: Result pagination offset
+        force_refresh: If True, bypasses local TTL cache
+    """
+    cleaned_query = (query or "").strip()
+    cleaned_zip = str(zip_code or DEFAULT_ZIP_CODE).strip()
+    limit = max(1, min(int(limit), 50))
+    offset = max(0, int(offset))
+
+    retailer_ids = resolve_supermarket_ids(supermarkets)
+
+    # Compute deterministic cache key
+    cache_raw = f"{cleaned_query.lower()}:{cleaned_zip}:{retailer_ids or ''}:{limit}:{offset}"
+    cache_key = hashlib.sha256(cache_raw.encode("utf-8")).hexdigest()[:16]
+
+    if not force_refresh:
+        cached_data, is_fresh = _read_cache(cache_key)
+        if is_fresh and cached_data:
+            return cached_data
+
+    params: Dict[str, Any] = {
+        "as": "web",
+        "limit": limit,
+        "offset": offset,
+        "q": cleaned_query,
+        "zipCode": cleaned_zip,
+    }
+    if retailer_ids:
+        params["retailers"] = retailer_ids
+
+    url = f"{API_BASE}/offers/search?{urllib.parse.urlencode(params)}"
+
+    try:
+        data = _fetch_with_retry(url)
+    except Exception as e:
+        logger.error(f"Marktguru API query failed: {e}")
+        # Emergency fail-safe: Check if stale cache is available
+        cached_data, _ = _read_cache(cache_key, max_age_seconds=86400 * 7)
+        if cached_data:
+            logger.warning("Serving stale cached data due to API failure.")
+            cached_data["warning"] = "Stale cache returned due to temporary upstream API issue."
+            return cached_data
+
+        return {
+            "error": f"API request failed: {str(e)}",
+            "query": cleaned_query,
+            "zip_code": cleaned_zip,
+            "total_results": 0,
+            "returned_count": 0,
+            "offers": [],
+            "available_retailers": [],
+        }
+
+    raw_results = data.get("results") or []
+    parsed_offers: List[Dict[str, Any]] = []
+    for item in raw_results:
+        parsed = parse_offer(item, cleaned_zip)
+        if parsed is not None:
+            parsed_offers.append(parsed)
+
+    filters = data.get("filters") or {}
+    raw_retailers = filters.get("retailers") or []
+    available_retailers = [
+        {"name": r.get("name"), "id": r.get("id"), "count": r.get("resultsCount")}
+        for r in raw_retailers if isinstance(r, dict) and r.get("name")
+    ]
+
+    result = {
+        "query": cleaned_query,
+        "zip_code": cleaned_zip,
+        "total_results": data.get("totalResults", len(parsed_offers)),
+        "returned_count": len(parsed_offers),
+        "available_retailers": available_retailers,
+        "offers": parsed_offers,
+    }
+
+    _write_cache(cache_key, result)
+    return result
+
+
+def compare_prices(
+    query: str,
+    zip_code: str = DEFAULT_ZIP_CODE,
+    supermarkets: Optional[List[str]] = None,
+    force_refresh: bool = False,
+) -> Dict[str, Any]:
+    """
+    Compares prices for a given product query across supermarkets, sorted from cheapest to most expensive.
+    """
+    data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=50, force_refresh=force_refresh)
+    offers = data.get("offers", [])
+
+    # Filter out items without price and sort ascending
+    priced_offers = [o for o in offers if o.get("price") is not None]
+    priced_offers.sort(key=lambda x: x["price"])
+
+    return {
+        "query": query,
+        "zip_code": str(zip_code),
+        "total_compared": len(priced_offers),
+        "cheapest_offer": priced_offers[0] if priced_offers else None,
+        "all_offers_sorted": priced_offers,
+    }
+
+
+def list_supported_supermarkets() -> Dict[str, Any]:
+    """Returns a structured catalog of all major supermarket chains supported by Marktguru."""
+    categories: Dict[str, List[Dict[str, Any]]] = {
+        "Discounter": [],
+        "Supermarkt": [],
+        "Großfläche / Hypermarkt": [],
+        "Drogerie": [],
+    }
+
+    for name, info in MAJOR_RETAILERS.items():
+        cat = info.get("type", "Sonstige")
+        if cat not in categories:
+            categories[cat] = []
+        categories[cat].append({
+            "name": name,
+            "id": info["id"],
+            "major": info.get("major", False)
+        })
+
+    return {
+        "default_zip_code": DEFAULT_ZIP_CODE,
+        "cache_ttl_seconds": CACHE_TTL,
+        "supported_chains_count": len(MAJOR_RETAILERS),
+        "categories": categories,
+        "supported_aliases": list(RETAILER_ALIASES.keys()),
+    }
+
+
+def main():
+    import argparse
+    parser = argparse.ArgumentParser(description="Query German supermarket flyers/discounts via Marktguru")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    # Search
+    search_p = subparsers.add_parser("search", help="Search offers for a product")
+    search_p.add_argument("query", help="Product to search (e.g. Butter, Kaffee)")
+    search_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
+    search_p.add_argument("--supermarket", help="Comma-separated supermarkets (e.g. 'Aldi,Lidl,Edeka')")
+    search_p.add_argument("--limit", type=int, default=15, help="Max results (default 15)")
+    search_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
+    search_p.add_argument("--json", action="store_true", help="Output raw JSON")
+
+    # Compare
+    comp_p = subparsers.add_parser("compare", help="Compare prices across supermarkets")
+    comp_p.add_argument("query", help="Product to compare")
+    comp_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
+    comp_p.add_argument("--supermarket", help="Comma-separated supermarkets")
+    comp_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
+    comp_p.add_argument("--json", action="store_true", help="Output raw JSON")
+
+    # Supermarkets
+    subparsers.add_parser("supermarkets", help="List all supported supermarkets and chains")
+
+    args = parser.parse_args()
+
+    if args.command == "supermarkets":
+        data = list_supported_supermarkets()
+        print(json.dumps(data, indent=2, ensure_ascii=False))
+        return
+
+    sm_list = [s.strip() for s in args.supermarket.split(",")] if getattr(args, "supermarket", None) else None
+
+    if args.command == "search":
+        result = search_offers(
+            args.query,
+            zip_code=args.zip,
+            supermarkets=sm_list,
+            limit=args.limit,
+            force_refresh=args.refresh
+        )
+        if args.json:
+            print(json.dumps(result, indent=2, ensure_ascii=False))
+        else:
+            if "error" in result and not result.get("offers"):
+                print(f"❌ Fehler: {result['error']}")
+                return
+
+            print(f"\n🔍 Angebote für '{args.query}' in PLZ {args.zip} (Gefunden: {result['total_results']}):\n")
+            for o in result["offers"]:
+                disc_str = f" (-{o['discount_percent']}%)" if o.get("discount_percent") else ""
+                loyalty_str = " 📱 [App Coupon]" if o.get("loyalty_app_required") else ""
+                timing_str = f"noch {o['days_remaining']} Tage" if o.get("days_remaining") is not None else f"bis {o['valid_to']}"
+                print(f" • [{o['supermarket']}] {o['description'] or o['product_name'] or 'Angebot'}")
+                print(f"   Preis: {o['price']:.2f} €{disc_str}{loyalty_str} | Gültig: {timing_str}")
+                if o.get("reference_price"):
+                    print(f"   Grundpreis: {o['reference_price']:.2f} € / {o.get('unit') or 'Einheit'}")
+                print()
+
+    elif args.command == "compare":
+        result = compare_prices(
+            args.query,
+            zip_code=args.zip,
+            supermarkets=sm_list,
+            force_refresh=args.refresh
+        )
+        if args.json:
+            print(json.dumps(result, indent=2, ensure_ascii=False))
+        else:
+            print(f"\n🏷️ Preisvergleich für '{args.query}' in PLZ {args.zip}:\n")
+            for idx, o in enumerate(result["all_offers_sorted"], 1):
+                disc_str = f" (UVP {o['old_price']:.2f}€)" if o.get("old_price") else ""
+                loyalty_str = " [App erforderlich]" if o.get("loyalty_app_required") else ""
+                print(f" {idx:2d}. {o['price']:.2f} € - [{o['supermarket']}] {o['description'] or o['product_name']}{disc_str}{loyalty_str}")
+            print()
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/marktguru/scripts/marktguru_mcp.py

@@ -0,0 +1,319 @@
+#!/usr/bin/env python3
+"""
+Marktguru MCP (Model Context Protocol) Server (Production Grade)
+Exposes German supermarket flyers, discounts, and price comparison tools over standard JSON-RPC 2.0 stdio.
+
+Key Design Principles:
+- Zero external dependencies: Uses standard library only.
+- Strict Protocol Compliance: Conforms to MCP (2024-11-05) and JSON-RPC 2.0 specifications.
+- Defensive Argument Coercion: Tolerates string/number variations commonly output by LLMs.
+- Graceful Signal Handling: Clean exit on SIGINT, SIGTERM, and EOF.
+- Stream Isolation: All logs and diagnostics route to stderr, ensuring stdout is 100% clean JSON-RPC.
+"""
+
+import sys
+import json
+import signal
+import logging
+from typing import Dict, Any, List, Optional
+
+# Import business logic from hardened client script
+from marktguru_client import (
+    search_offers,
+    compare_prices,
+    list_supported_supermarkets,
+    DEFAULT_ZIP_CODE,
+    MAJOR_RETAILERS,
+    RETAILER_ALIASES,
+)
+
+# Configure logging strictly to stderr
+logging.basicConfig(
+    stream=sys.stderr,
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
+)
+logger = logging.getLogger("marktguru-mcp")
+
+PROTOCOL_VERSION = "2024-11-05"
+SERVER_NAME = "marktguru-mcp-server"
+SERVER_VERSION = "1.1.0"
+
+TOOLS = [
+    {
+        "name": "search_supermarket_offers",
+        "description": (
+            "Search current weekly supermarket offers and promotional discounts in Germany via the Marktguru API. "
+            "Supports filtering by postal code (PLZ) and specific supermarket chains (e.g. Aldi Süd, Aldi Nord, Lidl, Edeka, Rewe, Kaufland, Penny, Netto)."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "The product or keyword to search for (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch', 'Nutella', 'Schokolade')."
+                },
+                "zip_code": {
+                    "type": "string",
+                    "description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
+                },
+                "supermarkets": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Optional list of supermarket names to filter by (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe', 'Kaufland', 'Penny', 'Netto', 'dm', 'Rossmann']). If omitted, all available retailers are included."
+                },
+                "limit": {
+                    "type": "integer",
+                    "description": "Maximum number of offers to return (default: 20, max: 50)."
+                }
+            },
+            "required": ["query"]
+        }
+    },
+    {
+        "name": "compare_supermarket_prices",
+        "description": (
+            "Compare current prices for a product across German supermarkets in a given postal code. "
+            "Returns results sorted from lowest to highest price, highlighting the cheapest deal, regular prices (UVP), and required loyalty app memberships."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "The product or grocery item to compare (e.g. 'Kaffee', 'Butter', 'Bier', 'Olivenöl')."
+                },
+                "zip_code": {
+                    "type": "string",
+                    "description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
+                },
+                "supermarkets": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Optional list of supermarket names to compare (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe'])."
+                }
+            },
+            "required": ["query"]
+        }
+    },
+    {
+        "name": "list_supported_supermarkets",
+        "description": (
+            "List all supermarket, discounter, hypermarket, and drugstore chains supported by the Marktguru service in Germany, "
+            "including their categories (Discounter, Supermarkt, Großfläche, Drogerie) and accepted name aliases."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {}
+        }
+    }
+]
+
+
+def _sanitize_supermarkets(sm_input: Any) -> Optional[List[str]]:
+    """Coerces various LLM input formats (list, comma-string, single string) into a clean list."""
+    if not sm_input:
+        return None
+    if isinstance(sm_input, list):
+        return [str(s).strip() for s in sm_input if str(s).strip()]
+    if isinstance(sm_input, str):
+        return [s.strip() for s in sm_input.split(",") if s.strip()]
+    return None
+
+
+def handle_initialize(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
+    return {
+        "jsonrpc": "2.0",
+        "id": req_id,
+        "result": {
+            "protocolVersion": PROTOCOL_VERSION,
+            "capabilities": {
+                "tools": {
+                    "listChanged": False
+                }
+            },
+            "serverInfo": {
+                "name": SERVER_NAME,
+                "version": SERVER_VERSION
+            }
+        }
+    }
+
+
+def handle_tools_list(req_id: Any) -> Dict[str, Any]:
+    return {
+        "jsonrpc": "2.0",
+        "id": req_id,
+        "result": {
+            "tools": TOOLS
+        }
+    }
+
+
+def handle_tools_call(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
+    tool_name = params.get("name")
+    args = params.get("arguments", {})
+
+    logger.info(f"Tool call requested: {tool_name} with args={args}")
+
+    try:
+        if tool_name == "search_supermarket_offers":
+            query = str(args.get("query") or "").strip()
+            if not query:
+                return {
+                    "jsonrpc": "2.0",
+                    "id": req_id,
+                    "result": {
+                        "content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
+                        "isError": True
+                    }
+                }
+
+            zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
+            supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
+            try:
+                limit = int(args.get("limit") or 20)
+            except (ValueError, TypeError):
+                limit = 20
+
+            data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=limit)
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        elif tool_name == "compare_supermarket_prices":
+            query = str(args.get("query") or "").strip()
+            if not query:
+                return {
+                    "jsonrpc": "2.0",
+                    "id": req_id,
+                    "result": {
+                        "content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
+                        "isError": True
+                    }
+                }
+
+            zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
+            supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
+
+            data = compare_prices(query, zip_code=zip_code, supermarkets=supermarkets)
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        elif tool_name == "list_supported_supermarkets":
+            data = list_supported_supermarkets()
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        else:
+            return {
+                "jsonrpc": "2.0",
+                "id": req_id,
+                "error": {
+                    "code": -32601,
+                    "message": f"Method or tool '{tool_name}' not found."
+                }
+            }
+
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "result": {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": text_result
+                    }
+                ],
+                "isError": False
+            }
+        }
+
+    except Exception as e:
+        logger.exception(f"Exception during tool execution '{tool_name}': {e}")
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "result": {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": f"Tool execution failed: {type(e).__name__}: {str(e)}"
+                    }
+                ],
+                "isError": True
+            }
+        }
+
+
+def process_message(line: str) -> Optional[Dict[str, Any]]:
+    if not line.strip():
+        return None
+
+    try:
+        msg = json.loads(line)
+    except json.JSONDecodeError as e:
+        logger.error(f"Malformed JSON-RPC received: {e}")
+        return {
+            "jsonrpc": "2.0",
+            "id": None,
+            "error": {"code": -32700, "message": f"Parse error: {str(e)}"}
+        }
+
+    method = msg.get("method")
+    req_id = msg.get("id")
+    params = msg.get("params", {})
+
+    # Notifications do not return a response
+    if req_id is None:
+        if method == "notifications/initialized":
+            logger.info("Client acknowledged initialized notification.")
+        elif method == "notifications/cancelled":
+            logger.info(f"Client cancelled operation: {params}")
+        return None
+
+    # Standard Requests
+    if method == "initialize":
+        return handle_initialize(req_id, params)
+    elif method == "tools/list":
+        return handle_tools_list(req_id)
+    elif method == "tools/call":
+        return handle_tools_call(req_id, params)
+    elif method == "ping":
+        return {"jsonrpc": "2.0", "id": req_id, "result": {}}
+    else:
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "error": {
+                "code": -32601,
+                "message": f"Method '{method}' is not supported."
+            }
+        }
+
+
+def _sig_handler(sig, frame):
+    logger.info(f"Received termination signal {sig}. Exiting gracefully.")
+    sys.exit(0)
+
+
+def main():
+    signal.signal(signal.SIGINT, _sig_handler)
+    signal.signal(signal.SIGTERM, _sig_handler)
+
+    logger.info(f"Starting {SERVER_NAME} v{SERVER_VERSION} (stdio mode)...")
+
+    try:
+        for raw_line in sys.stdin:
+            response = process_message(raw_line)
+            if response is not None:
+                output = json.dumps(response, ensure_ascii=False) + "\n"
+                sys.stdout.write(output)
+                sys.stdout.flush()
+    except (KeyboardInterrupt, BrokenPipeError):
+        pass
+    except Exception as e:
+        logger.exception(f"Fatal error in stdio loop: {e}")
+        sys.exit(1)
+
+    logger.info("Marktguru MCP server shutdown complete.")
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/marktguru/tests/test_marktguru.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""
+Unit tests for the Marktguru skill & MCP Server.
+Runs completely offline using standard library unittest.
+"""
+
+import sys
+import os
+import json
+import unittest
+from datetime import datetime, timezone, timedelta
+
+# Add scripts directory to path
+SCRIPTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "scripts"))
+if SCRIPTS_DIR not in sys.path:
+    sys.path.insert(0, SCRIPTS_DIR)
+
+from marktguru_client import (
+    resolve_supermarket_ids,
+    format_iso_date,
+    parse_offer,
+    list_supported_supermarkets,
+    DEFAULT_ZIP_CODE,
+    _read_cache,
+    _write_cache,
+    _get_cache_dir,
+)
+from marktguru_mcp import process_message, SERVER_NAME, PROTOCOL_VERSION
+
+
+class TestMarktguruClient(unittest.TestCase):
+    def test_default_zip_code(self):
+        self.assertEqual(DEFAULT_ZIP_CODE, "93059")
+
+    def test_resolve_supermarket_ids_aliases(self):
+        # Aldi maps to Aldi Nord and Aldi Süd
+        aldi_ids = resolve_supermarket_ids(["aldi"])
+        self.assertIn("127153", aldi_ids)
+        self.assertIn("127139", aldi_ids)
+
+        # Edeka maps to Edeka and E center
+        edeka_ids = resolve_supermarket_ids(["edeka"])
+        self.assertIn("127093", edeka_ids)
+        self.assertIn("126699", edeka_ids)
+
+        # Lidl
+        lidl_ids = resolve_supermarket_ids(["lidl"])
+        self.assertEqual(lidl_ids, "126679")
+
+        # Comma-separated string support
+        combined = resolve_supermarket_ids("Lidl, Kaufland")
+        self.assertIn("126679", combined)
+        self.assertIn("126654", combined)
+
+        # Unknown supermarket is ignored gracefully
+        unknown = resolve_supermarket_ids(["NonExistentMarket123"])
+        self.assertIsNone(unknown)
+
+    def test_format_iso_date(self):
+        self.assertEqual(format_iso_date("2026-10-15T12:00:00Z"), "15.10.2026")
+        self.assertEqual(format_iso_date(None), "-")
+        self.assertEqual(format_iso_date(""), "-")
+        self.assertEqual(format_iso_date("invalid-date-format"), "invalid-da")
+
+    def test_parse_offer_defensive(self):
+        # 1. Complete payload
+        now = datetime.now(timezone.utc)
+        from_date = (now - timedelta(days=1)).isoformat()
+        to_date = (now + timedelta(days=4)).isoformat()
+
+        raw_complete = {
+            "id": 99999,
+            "price": 1.49,
+            "oldPrice": 2.29,
+            "referencePrice": 5.96,
+            "requiresLoyalityMembership": True,
+            "description": "250g Packung",
+            "product": {"name": "Süßrahmbutter"},
+            "brand": {"name": "Landliebe"},
+            "advertisers": [{"name": "EDEKA", "id": "retailers/127093"}],
+            "unit": {"shortName": "kg"},
+            "volume": 0.25,
+            "validityDates": [{"from": from_date, "to": to_date}]
+        }
+
+        parsed = parse_offer(raw_complete, "93059")
+        self.assertIsNotNone(parsed)
+        self.assertEqual(parsed["id"], 99999)
+        self.assertEqual(parsed["supermarket"], "EDEKA")
+        self.assertEqual(parsed["price"], 1.49)
+        self.assertEqual(parsed["old_price"], 2.29)
+        self.assertEqual(parsed["discount_percent"], 35)  # (2.29 - 1.49) / 2.29 = 34.9% -> 35%
+        self.assertTrue(parsed["loyalty_app_required"])
+        self.assertTrue(parsed["is_valid_now"])
+        self.assertGreaterEqual(parsed["days_remaining"], 3)
+        self.assertIn("https://mg2de.b-cdn.net/api/v1/offers/99999", parsed["image_url"])
+
+        # 2. Minimal / empty payload
+        raw_empty = {}
+        parsed_min = parse_offer(raw_empty, "93059")
+        self.assertIsNotNone(parsed_min)
+        self.assertEqual(parsed_min["supermarket"], "Unbekannt")
+        self.assertIsNone(parsed_min["price"])
+        self.assertIsNone(parsed_min["discount_percent"])
+
+        # 3. Invalid types (not a dict)
+        self.assertIsNone(parse_offer("not a dict", "93059"))
+        self.assertIsNone(parse_offer(None, "93059"))
+
+    def test_cache_mechanism(self):
+        test_key = "test_unit_key_123"
+        test_data = {"hello": "world", "timestamp": 12345}
+
+        _write_cache(test_key, test_data)
+        data, is_fresh = _read_cache(test_key, max_age_seconds=60)
+        self.assertTrue(is_fresh)
+        self.assertEqual(data, test_data)
+
+        # Stale cache test
+        data_stale, is_fresh_stale = _read_cache(test_key, max_age_seconds=-1)
+        self.assertFalse(is_fresh_stale)
+        self.assertEqual(data_stale, test_data)  # still returned as fallback!
+
+    def test_list_supported_supermarkets(self):
+        catalog = list_supported_supermarkets()
+        self.assertEqual(catalog["default_zip_code"], "93059")
+        self.assertIn("Discounter", catalog["categories"])
+        self.assertIn("Supermarkt", catalog["categories"])
+        self.assertIn("aldi", catalog["supported_aliases"])
+        self.assertIn("edeka", catalog["supported_aliases"])
+
+
+class TestMarktguruMCP(unittest.TestCase):
+    def test_mcp_initialize(self):
+        req = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}})
+        resp = process_message(req)
+        self.assertEqual(resp["id"], 1)
+        self.assertEqual(resp["result"]["protocolVersion"], PROTOCOL_VERSION)
+        self.assertEqual(resp["result"]["serverInfo"]["name"], SERVER_NAME)
+
+    def test_mcp_tools_list(self):
+        req = json.dumps({"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
+        resp = process_message(req)
+        tools = [t["name"] for t in resp["result"]["tools"]]
+        self.assertIn("search_supermarket_offers", tools)
+        self.assertIn("compare_supermarket_prices", tools)
+        self.assertIn("list_supported_supermarkets", tools)
+
+    def test_mcp_tools_call_supermarket_catalog(self):
+        req = json.dumps({
+            "jsonrpc": "2.0",
+            "id": 3,
+            "method": "tools/call",
+            "params": {"name": "list_supported_supermarkets", "arguments": {}}
+        })
+        resp = process_message(req)
+        self.assertFalse(resp["result"]["isError"])
+        content_text = resp["result"]["content"][0]["text"]
+        data = json.loads(content_text)
+        self.assertEqual(data["default_zip_code"], "93059")
+
+    def test_mcp_tools_call_missing_query(self):
+        req = json.dumps({
+            "jsonrpc": "2.0",
+            "id": 4,
+            "method": "tools/call",
+            "params": {"name": "search_supermarket_offers", "arguments": {"query": ""}}
+        })
+        resp = process_message(req)
+        self.assertTrue(resp["result"]["isError"])
+        self.assertIn("required", resp["result"]["content"][0]["text"])
+
+    def test_mcp_notification_silence(self):
+        req = json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"})
+        resp = process_message(req)
+        self.assertIsNone(resp)  # Notifications must not produce responses
+
+
+if __name__ == "__main__":
+    unittest.main()

.agents/skills/openclaw-ops/SKILL.md

@@ -0,0 +1,231 @@
+---
+name: openclaw-ops
+license: Apache-2.0
+description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram.
+---
+
+# OpenClaw Operations & Troubleshooting Runbook
+
+> **Target Environment**: Kubernetes cluster `haumdaucher`, namespace `openclaw`, pod `openclaw-0`.  
+> **Key References**: [`k8s/openclaw/AGENTS.md`](../../../k8s/openclaw/AGENTS.md) and [`k8s/openclaw/TROUBLESHOOTING.md`](../../../k8s/openclaw/TROUBLESHOOTING.md).
+
+---
+
+## 0. Instant One-Shot Triage (< 2 Seconds)
+
+Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**.
+Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem.
+
+### Primary Command:
+```bash
+# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research)
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicNumber>
+
+# Or by agent name
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent <agentId>
+```
+
+The script automatically reports:
+1. Ground-truth session status (`done`, `processing`, `running`) and token size.
+2. Inbound user prompt (including speech-to-text audio transcripts).
+3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`❌`), and final text output.
+4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`.
+
+### 🚫 Anti-Patterns & Operational Guardrails:
+- ❌ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks.
+- ❌ **NEVER guess SQLite table or column names.**
+  - **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)`
+  - **`transcript_events`**: `(session_id, seq, event_json, created_at)`
+- 📍 **Known Path Anchors (Use Directly):**
+  - Knowledge Wiki: `/home/node/.openclaw/wiki/main`
+  - Agent SQLite DBs: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
+  - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
+  - Agent Workspace: `/home/openclaw/.openclaw/workspace-<agent>`
+  - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
+
+---
+
+## 1. Diagnostic Playbook: "Bot is Not Answering / Silent"
+
+When a user reports that a Telegram forum topic or direct chat is unresponsive:
+
+### ⚠️ Golden Rule: Do NOT Assume Container Crash
+Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (`5/5 Running`).
+
+### Step 1: Check Live Logs for In-Flight Turns
+```bash
+kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100
+```
+Look for:
+- `compact:start reason=auto-threshold`: Inline safeguard compaction in progress (takes 30–60s).
+- `embedded run tool start ... tool=write|exec`: Agent is actively executing tool steps.
+- `[model-fetch] start ... status=200`: Waiting on LLM token streaming.
+- `heartbeat: webhooks=... active=X`: Active turn count in gateway.
+
+### Step 2: Query Ground-Truth Session Status in SQLite
+Each agent maintains its own SQLite database at `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`.
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, glob
+for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
+    agent = db.split('/')[5]
+    con = sqlite3.connect(db)
+    rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall()
+    if rows:
+        print(f'[{agent}]', rows)
+"
+```
+- If `status == "processing"`: Turn is actively executing.
+- If `status == "done"`: Turn completed. Check outbound Telegram dispatch.
+- If `status == "running"` with old `updated_at`: Session is wedged (see Recovery below).
+
+### Step 3: Inspect Recent Transcript Events
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, json
+con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
+sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('<sessionKey>',)).fetchone()[0]
+for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall():
+    print(seq, ca, ej)
+"
+```
+
+---
+
+## 2. Session Recovery & Lock Clearing
+
+### Un-wedge a Stuck Session Node
+If a turn was killed mid-execution (e.g. OOMKill or node reboot):
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3
+con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
+con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('<sessionKey>',))
+con.commit()
+print('Session un-wedged successfully.')
+"
+```
+
+### Clean Stale State Locks
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock
+```
+
+### Check Foreign Key Violations & Clear Quarantine
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, glob
+for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'):
+    con = sqlite3.connect(db)
+    fk = con.execute('PRAGMA foreign_key_check;').fetchall()
+    if fk:
+        print(f'FK violations in {db}:', fk)
+"
+# Clear quarantine table if present
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, os
+p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite'
+if os.path.exists(p):
+    c = sqlite3.connect(p)
+    c.execute('DELETE FROM quarantined_databases')
+    c.commit()
+    print('Quarantine cleared.')
+"
+```
+
+---
+
+## 3. Multi-Agent Compaction & Transcript Bloat Control
+
+### Pitfall: Stale Sessions Report `totalTokens: null`
+`openclaw sessions --all-agents --json` returns `totalTokens: null` (`totalTokensFresh: false`) for sessions that haven't received a message in several days. **Always inspect SQLite directly**:
+```bash
+# Check size of active sessions across all agents
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import glob, sqlite3, json
+for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
+    agent = db.split('/')[5]
+    con = sqlite3.connect(db)
+    for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall():
+        cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone()
+        if bsz > 300000 or cnt > 40:
+            print(f'{agent:12} {bsz//1024:6} KB  {cnt:4} events  {skey}')
+"
+```
+
+### Manually Compact a Stale or Bloated Session
+```bash
+# LLM summarization with 3-minute timeout
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --timeout 180000
+
+# Truncation fallback (if LLM summarization times out)
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --max-lines 80 --timeout 180000
+```
+
+### Ensure User Compaction Notification is Enabled
+To prevent silent user-perceived freezes during daytime turns:
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json
+```
+
+---
+
+## 4. Mechanical HDD / Page Cache Pre-Warming
+
+Node `haumdaucher` uses spinning HDDs (`rotational: 1`). Cold database opens take up to 20–50 minutes without page cache warming.
+
+### Warm Page Cache Instantly (<1s)
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \
+  "cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true"
+```
+
+### WAL Checkpoints & SQLite Optimization
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import glob, sqlite3
+dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')
+for db in dbs:
+    con = sqlite3.connect(db)
+    con.execute('PRAGMA wal_checkpoint(TRUNCATE);')
+    con.execute('PRAGMA optimize;')
+    con.close()
+print('WAL checkpoints and optimizations complete.')
+"
+```
+
+---
+
+## 5. Emergency Telegram Direct Message Bypass
+
+If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn:
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
+  --channel telegram \
+  --target "-1004326921510" \
+  --thread-id <topicNumber> \
+  --message "Dein Nachrichtentext hier..."
+```
+*(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).*
+
+---
+
+## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant
+
+At the end of **every single troubleshooting task, incident investigation, or bugfix**:
+
+### 🔄 Mandatory Dual-Reflection
+The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding:
+1. **Troubleshooting Process Optimization**:
+   - Did gathering information or isolating the root cause take too long?
+   - Did any query fail due to syntax, guessing, or slow filesystem sweeps?
+   - What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds?
+2. **System Maturity & Long-Term Hardening**:
+   - What fragile assumption or missing guardrail allowed this error to surface?
+   - How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs?
+
+### ⚖️ The Simplicity Invariant (The 98% Rule)
+- **Simplicity and clean, transparent architecture are the central design principles.**
+- Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%.
+- Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.
+

.agents/skills/promql/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: promql
+license: Apache-2.0
+description: Write, validate, and optimize PromQL for Prometheus / Grafana Mimir / Grafana Cloud Metrics. Covers `rate` vs `irate` vs `increase`, label matchers and regex, `sum / avg / topk / by / without` aggregation, classic + native `histogram_quantile`, ratios with divide-by-zero guards, `absent` / `changes` for staleness, time offsets and `predict_linear`, recording-rule naming, SLO + burn-rate math, and a cardinality-hunting playbook. Use when writing a metric query, fixing wrong p95s, building an error-budget alert, debugging "query is slow", finding the noisy label that blew up cardinality, or migrating a dashboard query to a recording rule — even when the user says "calculate the error rate", "p99 latency", "sum by service", "why is this query slow", or "what's filling Mimir" without naming PromQL.
+---
+
+# PromQL Query Patterns
+
+> **Docs**: https://prometheus.io/docs/prometheus/latest/querying/basics/
+
+PromQL returns either an **instant vector**, a **range vector**, or a **scalar**.
+
+**Golden rule:** `rate()` / `increase()` require a range vector ≥ 4× the scrape interval. 60s scrape → use `[5m]` minimum.
+
+## Prerequisites
+
+- A Prometheus / Mimir / Grafana Cloud endpoint to query (`/api/v1/query` or via Grafana Explore)
+- The PromQL pattern library in [`references/patterns.md`](references/patterns.md)
+
+## Common Workflows
+
+### 1. Write + validate a query
+
+```bash
+# 0. Point at your Prometheus/Mimir. For Grafana Cloud, use the metrics endpoint
+#    and add basic auth (-u "<metrics_user>:<token>") to each curl below.
+PROM=http://localhost:9090   # or https://prometheus-prod-XX.grafana.net/api/prom
+
+# 1. Sketch the query — for "5xx error rate per service":
+EXPR='sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)'
+
+# 2. Validate syntax + that the metric/labels exist
+curl -sG --data-urlencode "query=${EXPR}" \
+  "$PROM/api/v1/query" | jq '.status, (.data.result|length)'
+# Expect: "success" and result count > 0. If 0 — check label spelling and scrape activity:
+curl -sG --data-urlencode "match[]=http_requests_total" "$PROM/api/v1/series" | jq '.data | length'
+
+# 3. Sanity-check the magnitude — open Grafana Explore, paste the expr,
+#    confirm the values look right against a known ground truth (k6 run, log count, etc.)
+```
+
+### 2. Common patterns to copy
+
+**Per-status request rate** (aggregate AFTER rate):
+
+```promql
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+```
+
+**p95 latency** (must keep `le` in the inner aggregation):
+
+```promql
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+```
+
+**Error rate with divide-by-zero guard:**
+
+```promql
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / (sum(rate(http_requests_total[5m])) > 0)
+```
+
+Full library (recording rules, SLO burn-rate, offsets, cardinality hunt, native histograms): [`references/patterns.md`](references/patterns.md).
+
+### 3. Convert a slow dashboard query into a recording rule
+
+```yaml
+# 1. Pick the slow expression, give it a recording-rule name
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+```bash
+# 2. After rules load, verify the new metric exists
+curl -sG --data-urlencode "query=job:http_request_duration_p95:rate5m" \
+  "$PROM/api/v1/query" | jq '.data.result | length'   # → > 0
+
+# 3. Verify it matches the original expression for at least one sample window
+# (Both queries should produce the same value at the same timestamp.)
+
+# 4. Replace the dashboard panel expression with the recording-rule metric.
+```
+
+## Common bugs
+
+- `histogram_quantile` returns NaN → forgot `by (le)` in the inner aggregation
+- "No data" → check the metric exists (`/api/v1/series`) and the window ≥ 4× scrape interval
+- Wrong rate magnitude → counter was aggregated before `rate()` (always `rate()` first)
+- Query timeout → series count too high; use `topk(...)` + a recording rule + drop high-cardinality labels (see [`references/patterns.md`](references/patterns.md))
+
+## Resources
+
+- [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
+- [Operators](https://prometheus.io/docs/prometheus/latest/querying/operators/)
+- [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)
+- [Grafana Mimir](https://grafana.com/docs/mimir/latest/)

.agents/skills/promql/references/patterns.md

@@ -0,0 +1,174 @@
+# PromQL pattern library
+
+## Rate, counter, and aggregation
+
+```promql
+# rate() — per-second average
+rate(http_requests_total[5m])
+
+# CORRECT: rate first, then aggregate
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+
+# WRONG: never sum() a raw counter before rate()
+sum(http_requests_total) by (status_code)   # do NOT then rate()
+
+# increase() — total count over window
+increase(http_requests_total[1h])
+```
+
+`rate` vs `irate`:
+- `rate()` smooths the full window — use for dashboards + alerts
+- `irate()` uses the last two samples — for spike detection only, never alerting
+
+## Label matchers
+
+```promql
+http_requests_total{job="api", status_code="200"}
+http_requests_total{status_code=~"5.."}
+http_requests_total{status_code!~"2.."}
+http_requests_total{env=~"staging|production"}
+```
+
+## Aggregation
+
+```promql
+sum(rate(http_requests_total[5m])) by (service)
+avg(node_cpu_seconds_total{mode="idle"}) by (instance)
+
+# Top 5 by request rate
+topk(5, sum(rate(http_requests_total[5m])) by (service))
+
+# Count of distinct label values
+count(count(up) by (job)) by ()
+```
+
+`by` keeps only listed labels; `without` drops only listed labels.
+
+## Histogram quantiles
+
+```promql
+# Classic histogram
+histogram_quantile(0.99,
+  sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le))
+
+# Multi-service comparison — must include le AND service
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+
+# Native histograms (Prometheus 2.40+)
+histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m])))
+```
+
+Forgetting `by (le)` is the most common bug — `histogram_quantile` returns NaN or wrong values.
+
+## Ratios and error rates
+
+```promql
+# Error fraction
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / sum(rate(http_requests_total[5m]))
+
+# Success percentage
+(1 - sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+     / sum(rate(http_requests_total[5m]))) * 100
+
+# Avoid divide-by-zero
+sum(rate(errors_total[5m]))
+  / (sum(rate(requests_total[5m])) > 0)
+```
+
+## Absence and staleness
+
+```promql
+absent(up{job="api"})                  # metric disappeared
+changes(up{job="api"}[5m]) == 0        # value not changing → stale exporter
+count_over_time(up{job="api"}[5m]) > 0 # at least one sample in window
+```
+
+## Time offsets
+
+```promql
+# Current vs 1h ago
+rate(http_requests_total[5m])
+  - rate(http_requests_total[5m] offset 1h)
+
+# Day-over-day
+rate(http_requests_total[5m])
+  / rate(http_requests_total[5m] offset 1d)
+
+# Predict 2h ahead from 1h trend (linear regression)
+predict_linear(node_filesystem_avail_bytes[1h], 2 * 3600)
+```
+
+## Recording rules
+
+Naming: `<aggregation_level>:<metric_name>:<operation_and_window>`
+
+```yaml
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_requests_total:rate5m
+        expr:   sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_errors:ratio5m
+        expr: |
+          sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)
+          / sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+## SLO + burn-rate
+
+```promql
+# Availability SLO over 30 days
+1 - (sum(increase(http_requests_total{status_code=~"5.."}[30d]))
+     / sum(increase(http_requests_total[30d])))
+
+# 1h burn-rate vs target (replace 0.999 with your SLO target)
+( sum(rate(http_requests_total{status_code=~"5.."}[1h]))
+  / sum(rate(http_requests_total[1h])) )
+  / (1 - 0.999)
+```
+
+## Cardinality controls
+
+```promql
+# Top 10 metrics by series count
+topk(10, count by (__name__)({__name__=~".+"}))
+
+# Series count for one metric
+count(http_requests_total)
+
+# Cardinality of one label
+count(count by (user_id)(http_requests_total))
+```
+
+Drop a high-cardinality label at scrape time (Alloy):
+
+```alloy
+prometheus.scrape "api" {
+  targets = [...]
+  rule { source_labels = ["user_id"]
+         action        = "labeldrop" }
+}
+```
+
+## Other common patterns
+
+```promql
+# Service availability (alert)
+avg_over_time(up{job="api"}[5m]) < 0.9
+
+# Saturation — disk full in <4h
+predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
+
+# Throughput spike — current rate > 3x 1h average
+rate(http_requests_total[5m])
+  > 3 * avg_over_time(rate(http_requests_total[5m])[1h:5m])
+```

.gitattributes

@@ -1,5 +1,8 @@
 *.secret filter=git-crypt diff=git-crypt
 *.secret.yaml filter=git-crypt diff=git-crypt
+*.secret.yml filter=git-crypt diff=git-crypt
 *.secret.values filter=git-crypt diff=git-crypt
 *.secret.sh filter=git-crypt diff=git-crypt
 *.secret.conf filter=git-crypt diff=git-crypt
+*.secret.auto.tfvars filter=git-crypt diff=git-crypt
+*.secret.tfvars filter=git-crypt diff=git-crypt

.gitignore

@@ -2,3 +2,10 @@ k8s/gitea-chart/
 .vagrant
 k8s/tekton-pipelines/auth
 .DS_Store
+node_modules/
+**/node_modules/
+dist/
+**/dist/
+*.tsbuildinfo
+.venv/
+**/__pycache__/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "knowledge-wiki"]
+	path = knowledge-wiki
+	url = https://git.moritzgraf.de/moritz/knowledge-wiki.git

AGENTS.md

@@ -14,35 +14,115 @@ This repository is divided into distinct domains, each with its own `AGENTS.md`
 
 ### 2. [Kubernetes Workloads](./k8s/AGENTS.md) (`k8s/`)
 *   **Purpose**: Deploying applications and services to the cluster.
-*   **Method**: GitOps-style directory structure (Namespace = Folder).
-*   **Constraints**: Enforces `ingress-nginx`, `cert-manager` (Let's Encrypt), and `git-crypt` for secrets.
+*   **Strict Namespace Rule**: **Every top-level subdirectory in `k8s/` strictly corresponds to a Kubernetes namespace.** Non-namespace folders, loose scripts, or documentation do not belong here.
+*   **Constraints**: Enforces `ingress-nginx` / `traefik`, `cert-manager` (Let's Encrypt), and `git-crypt` for secrets.
+*   **OpenClaw**: For OpenClaw AI agent deployment and troubleshooting, see [k8s/openclaw/AGENTS.md](./k8s/openclaw/AGENTS.md) and [k8s/openclaw/TROUBLESHOOTING.md](./k8s/openclaw/TROUBLESHOOTING.md).
 *   **Read**: [k8s/AGENTS.md](./k8s/AGENTS.md)
 
-### 3. Terraform (`terraform/`)
-*   **Purpose**: Infrastructure as Code for external resources, primarily **DNS**.
-*   **Provider**: OVH (`ovh/ovh`).
-*   **Scope**:
-    *   **DNS Zones**: `haumdaucher.de`, `moritzgraf.de`, `ameliegraf.de`, and others.
-    *   **Mail Config**: Google Workspace MX/SPF/DKIM records.
-    *   **Cluster IP**: Points domains to the Haumdaucher node IP (`136.243.23.215`).
-*   **Usage**:
-    ```bash
-    cd terraform
-    terraform init
-    terraform apply
-    ```
+### 3. [Terraform](./terraform/AGENTS.md) (`terraform/`)
+*   **Purpose**: Declarative core cloud infrastructure as code.
+    *   **DNS Zones**: `haumdaucher.de`, `moritzgraf.de`, `ameliegraf.de`, and others via OVH (`ovh/ovh`).
+    *   **Cloud Services**: Authentik SSO (`authentik.tf`), GCP APIs, Velero backup SA (`gcp_backups.tf`), OpenClaw calendar SA (`gcp_calendar.tf`), and Google Workspace MX/SPF/DKIM (`google_workspace.tf`).
+    *   **Usage**: `cd terraform && terraform apply`
+*   **Read**: [terraform/AGENTS.md](./terraform/AGENTS.md)
 
 ### 4. Monitoring (`monitoring/`)
-*   **Purpose**: IoT and physical device monitoring, separate from K8s metrics.
+*   **Purpose**: Physical ambient BLE sensor fleet monitoring across living areas.
 *   **Tool**: [ESPHome](https://esphome.io).
-*   **Devices**: Temperature sensors, plant monitors (Xiaomi/BLE), ESP32 bridges.
+*   **Devices**: Temperature & humidity sensors, Xiaomi BLE plants, ESP32 bridges (`muc-kitchen`, `muc-living`, `rgbg-living`).
 *   **Usage**:
     ```bash
     cd monitoring
-    # Flash or update a device
     esphome run config/<device>.secret.yml
     ```
-*   **Network**: Devices typically operate on static IPs in the `192.168.x.x` range.
+*   **Read**: [monitoring/README.md](./monitoring/README.md)
+
+### 5. Knowledge Wiki & OpenClaw Tasks (`knowledge-wiki/`)
+*   **Purpose**: Shared personal knowledge wiki and task dispatch between OpenClaw (mobile/Telegram concierge) and Antigravity (local pair programming agent).
+*   **Location**: Git submodule `knowledge-wiki/` at root of `infrapuzzle` (Git repo: `https://git.moritzgraf.de/moritz/knowledge-wiki.git`, branch `main`).
+*   **Queue File**: `knowledge-wiki/tasks/antigravity.md`.
+*   **Workflow for Antigravity**:
+    - When the user asks to check tasks or pick up work from OpenClaw:
+      1. Pull latest: `git submodule update --remote knowledge-wiki` (or `git -C knowledge-wiki pull --rebase origin main`)
+      2. Read `knowledge-wiki/tasks/antigravity.md` to see pending items.
+      3. Implement the chosen task in the appropriate repository files.
+      4. Move the task to `## ✅ Completed Tasks` with completion details and commit info.
+      5. Push back: `git -C knowledge-wiki commit -am "chore(tasks): complete <task>" && git -C knowledge-wiki push origin main`.
+
+### 6. Architecture Decision Records (`docs/adr/`)
+*   **Purpose**: Formal record of architectural decisions, technical trade-offs, and design standards across the monorepo.
+*   **Format**: [MADR](https://adr.github.io/madr/) (Markdown Architectural Decision Records).
+*   **Consulting ADRs**: AI agents MUST consult `docs/adr/` before designing or altering system architecture, agent topologies, or tool integration patterns.
+*   **Key Records**:
+    *   [ADR 0001: Hybrid MCP Tool Search and Subagent Delegation for AI Agents](docs/adr/0001-hybrid-mcp-tool-search-and-subagents.md)
+    *   [ADR 0002: OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics](docs/adr/0002-openclaw-model-selection-tiering-and-economics.md)
+    *   [ADR 0003: Everything as Code (EaC) and Reproducible Declarative Infrastructure](docs/adr/0003-everything-as-code-and-reproducible-infrastructure.md)
+
+### 7. [Saschas Kronkorken Kreuzzug](./projects/sasch40/AGENTS.md) (`projects/sasch40/`)
+*   **Purpose**: Mobile-first 2.5D retro birthday adventure game for King Sascha's 40th jubilee (`https://sasch40.haumdaucher.de`).
+*   **Key Rules**: Prequel landing screen for QR codes, dynamic GameBoy arcade vs RPG layouts, real-photo character portraits, pure Web Audio synthesizer, competitive certificate scoring.
+*   **Read**: [projects/sasch40/AGENTS.md](./projects/sasch40/AGENTS.md)
+
+### 8. [Moritz Graf Personal Website](./projects/moritzgrafde/AGENTS.md) (`projects/moritzgrafde/`)
+*   **Purpose**: Personal homepage and portfolio for Moritz Graf (`https://moritzgraf.de`).
+*   **Manifest**: `k8s/web/moritzgrafde.yaml` (Namespace: `web`).
+*   **Image**: `registry.haumdaucher.de/moritzgrafde:latest`.
+*   **Key Rules**: Executive portfolio built with Vue 3, Vite, TypeScript, and Tailwind CSS served via Nginx in Docker.
+*   **Read**: [projects/moritzgrafde/AGENTS.md](./projects/moritzgrafde/AGENTS.md)
+
+### 9. [Haumdaucher Official Website](./projects/haumdaucherde/AGENTS.md) (`projects/haumdaucherde/`)
+*   **Purpose**: Official community platform and club portal for Haumdaucher Regensburg (`https://haumdaucher.de`).
+*   **Manifest**: `k8s/haumdaucher/haumdaucher.yaml` (Namespace: `haumdaucher`).
+*   **Image**: `registry.haumdaucher.de/haumdaucher-website:latest`.
+*   **Key Rules**: PWA built with Vue 3, Vite, TypeScript, and Firebase Auth; dynamic theming with NAT mode easter egg; strict compliance with German Vereinsrecht (§ 5 DDG Impressum for n.e.V.); user creation and mail forwarding Apps Script automations.
+*   **Read**: [projects/haumdaucherde/AGENTS.md](./projects/haumdaucherde/AGENTS.md)
+
+### 10. [Grabengasse 7 Home Portal](./projects/grabengasse/AGENTS.md) (`projects/grabengasse/`)
+*   **Purpose**: Local climate, weather, and environmental dashboard for Grabengasse 7 (`https://grabengasse7.de`).
+*   **Manifests**: `k8s/grabengasse/` (`frontend.yaml`, `backend.yaml`, `ingress.yaml` in namespace `grabengasse`).
+*   **Images**: `registry.haumdaucher.de/website/grabengasse_frontend:latest`, `registry.haumdaucher.de/website/grabengasse_backend:latest`.
+*   **Key Rules**: Vue 3 retro pixel art frontend + FastAPI backend querying cluster Prometheus for Shelly WS90 weather telemetry.
+*   **Read**: [projects/grabengasse/AGENTS.md](./projects/grabengasse/AGENTS.md)
+
+### 11. [Kitchen e-Paper Display](./projects/esphome_display_kitchen/AGENTS.md) (`projects/esphome_display_kitchen/`)
+*   **Purpose**: Dedicated ESP32 microcontroller driving the Waveshare e-Paper status display in the kitchen.
+*   **Integrations**: Home Assistant Native API (`input_text.epaper_message`), Prometheus client, SNTP time sync.
+*   **Key Tools**: Interactive `uv run flash.py` wizard and `./send_message.sh` CLI push messaging.
+*   **Read**: [projects/esphome_display_kitchen/AGENTS.md](./projects/esphome_display_kitchen/AGENTS.md)
+
+### 12. [DuckDB Backend](./projects/duckdb-backend/AGENTS.md) (`projects/duckdb-backend/`)
+*   **Purpose**: High-performance HTTP analytics and event tracking data service executing in-memory DuckDB queries against Knowledge Wiki JSONL datasets.
+*   **Manifests**: `k8s/family/` (`duckdb-backend.yaml`, `pvc.yaml`, `ingress.yaml` in namespace `family`).
+*   **Endpoints**: Curated dashboards (`/duckdb-api/v1/mila/dashboard`, `/duckdb-api/v1/moritz/dashboard`), generic SQL queries (`/duckdb-api/v1/query`), and event tracking writes with Git sync.
+*   **Read**: [projects/duckdb-backend/AGENTS.md](./projects/duckdb-backend/AGENTS.md)
+
+### 13. [Mila Tracking Web App](./projects/mila/AGENTS.md) (`projects/mila/`)
+*   **Purpose**: Reactive mobile-first PWA for Mila's baby sleep, feeding, diaper tracking, and growth percentiles (`https://family.haumdaucher.de/tracking/mila`).
+*   **Manifest**: `k8s/family/mila.yaml` (Namespace: `family`).
+*   **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; live wake timer; interactive 1-tap and custom-time tracking modal.
+*   **Read**: [projects/mila/AGENTS.md](./projects/mila/AGENTS.md)
+
+### 14. [Moritz Health & Nutrition Web App](./projects/moritz/AGENTS.md) (`projects/moritz/`)
+*   **Purpose**: Apple Health-inspired nutrition and metabolic balance dashboard (`https://family.haumdaucher.de/tracking/moritz`).
+*   **Manifest**: `k8s/family/moritz.yaml` (Namespace: `family`).
+*   **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; net calorie balance hero, Garmin health breakdown, and quick meal logger.
+*   **Read**: [projects/moritz/AGENTS.md](./projects/moritz/AGENTS.md)
+
+### 15. [Shelly Taupunktlüfter](./projects/shelly_taupunktluefter/AGENTS.md) (`projects/shelly_taupunktluefter/`)
+*   **Purpose**: Automated cellar ventilation controller ("Taupi") on Shelly Plug S Gen 3 to prevent mold growth using psychrometric calculations and BTHome BLE sensors.
+*   **Usage**: `cd projects/shelly_taupunktluefter && terraform apply`
+*   **Key Rules**: Sedlbauer LIM I decoupled mold danger evaluation, summer overheating and winter cold guards, optimal night drying window (03:00–07:00), and Prometheus metrics export.
+*   **Read**: [projects/shelly_taupunktluefter/AGENTS.md](./projects/shelly_taupunktluefter/AGENTS.md)
+
+### 16. Utility Scripts (`scripts/`)
+*   **Purpose**: Cross-cutting operational and administration CLI scripts that do not belong inside Kubernetes namespace directories.
+*   **Key Tools**:
+    *   `scripts/htpasswd.py`: Generates bcrypt-hashed htpasswd credentials for ingress basic auth secrets.
+
+### 17. Network & Router Integration (`fritzbox/`)
+*   **Purpose**: Site-to-Site WireGuard LAN coupling and DynDNS OVH synchronization for the home FritzBox router.
+*   **Cluster Endpoint**: `k8s/wireguard/` (`vpn.haumdaucher.de`).
+*   **Read**: [fritzbox/README.md](./fritzbox/README.md)
 
 ## Global Constraints
 *   **Secrets**: `git-crypt` is used extensively. Ensure you have the unlock key before editing `*.secret` files.

README.md

@@ -1,27 +1,26 @@
-# Infrapuzzle
+# Infrapuzzle 🧩
 
-Infrapuzzle is the newly restructured way of implementing my private infrastructure. Currently the following things are in project scope:
+Infrapuzzle is the monorepo for personal infrastructure, Kubernetes cluster operations, DNS, IoT monitoring, and hosted applications.
 
-* single node k8s "cluster" implemented with kubespray
-* development with Vagrant
-* host directory storage
-* Ingress via host network
+## Repository Layout
 
-## terraform
+- **[`bootstrap/`](bootstrap/)**: Provisioning the single-node Kubernetes cluster (`haumdaucher`) using Kubespray.
+- **[`k8s/`](k8s/)**: Kubernetes workloads. **Every subdirectory represents a Kubernetes namespace.**
+- **[`terraform/`](terraform/)**: Declarative Infrastructure as Code for core cloud infrastructure (OVH DNS zones, Authentik SSO, Google Workspace, and GCP APIs).
+- **[`projects/`](projects/)**: Self-contained hosted applications and projects:
+  - `projects/shelly_taupunktluefter/`: Smart home Shelly Plug S ventilation controller ("Taupi") for cellar mold prevention.
+  - `projects/sasch40/`: Retro birthday adventure game (`sasch40.haumdaucher.de`).
+  - `projects/moritzgrafde/`: Personal homepage and executive portfolio (`moritzgraf.de`).
+  - `projects/haumdaucherde/`: Official community website & portal (`haumdaucher.de`).
+  - `projects/grabengasse/`: Climate & weather dashboard (`grabengasse7.de`).
+  - `projects/esphome_display_kitchen/`: ESP32 e-Paper kitchen status display & flashing wizard.
+  - `projects/duckdb-backend/`: In-memory analytics and tracking query API.
+  - `projects/mila/`: Baby tracking PWA (`family.haumdaucher.de/tracking/mila`).
+  - `projects/moritz/`: Health and nutrition dashboard (`family.haumdaucher.de/tracking/moritz`).
+- **[`monitoring/`](monitoring/)**: ESPHome fleet for BLE ambient room temperature & humidity sensors.
+- **[`fritzbox/`](fritzbox/)**: Home router Site-to-Site WireGuard VPN coupling and DynDNS synchronization.
+- **[`scripts/`](scripts/)**: Operational utility scripts (e.g. `htpasswd.py`).
+- **[`knowledge-wiki/`](knowledge-wiki/)**: Shared personal knowledge base and OpenClaw task queue submodule.
+- **[`docs/`](docs/)**: Architecture Decision Records (ADRs) following the MADR standard.
 
-[Documentation in subfolder](./terraform/README.md). It takes care of setting DNS records on OVH nameservers.
-
-## bootstrap
-
-[Documentation in subfolder](./bootstrap/README.md). It takes care of setting up the k8s cluster itself, including a vagrant dev environment.
-
-## k8s
-
-[Documentation in subfolder](./k8s/README.md). The services themselfes.
-
-
-
-## Links used
-
-* [ingress via host network](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network)
-* [Install K8s with Kubespray](https://kubernetes.io/docs/setup/production-environment/tools/kubespray/)
+For architectural standards and operational guidelines, refer to [AGENTS.md](AGENTS.md).

bootstrap/.gitignore

@@ -1,2 +1,4 @@
 kubespray
 venv
+credentials/
+inventory/**/credentials/

bootstrap/AGENTS.md

@@ -1,49 +1,83 @@
 # AGENTS.md
 
 > [!NOTE]
-> This directory handles the **bootstrapping and provisioning** of the Haumdaucher Kubernetes cluster using **Kubespray**.
+> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.
 
 ## Project Overview
 *   **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
-*   **Method**: The local `inventory/` is the source of truth, which is synced into a checked-out Kubespray repository.
-*   **Idempotency**: The process is designed to be repeatable. The `kubespray` folder is treated as ephemeral and is re-created by `init.sh`.
+*   **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
+*   **Host OS**: Ubuntu 20.04.6 LTS.
+*   **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
+*   **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
+
+---
+
+## Architectural Invariants for Single-Node Upgrades
+
+> [!IMPORTANT]
+> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
+
+1. **NO DRAIN / NO CORDON**:
+   - Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
+   - `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
+   - Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
+2. **VERSION STRING FORMAT**:
+   - In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
+3. **UBUNTU 20.04 OS CONSTRAINT**:
+   - Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
+4. **ETCD BACKUPS**:
+   - Always take a manual etcd snapshot before initiating any cluster upgrade.
+
+---
 
 ## Workflow & Scripts
-The core workflow is encapsulated in `init.sh`.
 
-### `init.sh`
-**Purpose**: Prepares the environment and Kubespray for deployment.
-**Actions**:
-1.  **Clean Slate**: Deletes existing `kubespray/` directory.
-2.  **Clone**: Clones Kubespray (version defined in variable `VERSION`, e.g., `release-2.27`).
-3.  **Environment**: Sets up Python virtualenv via `pyenv` and installs `requirements.txt`.
-4.  **Sync**: Copies local `./inventory/` configurations into `./kubespray/inventory/`.
+### 1. Source `init.sh`
+```bash
+source init.sh
+```
+**Actions performed**:
+- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
+- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
+- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
+- Syncs `./inventory/` to `./kubespray/inventory/`.
+- Exports the `sync-inventory` alias.
+- Changes working directory to `./kubespray`.
 
-### Usage
-1.  **Source the script**:
-    ```bash
-    source init.sh
-    ```
-2.  **Deploy / Upgrade**:
-    After sourcing, go to the `kubespray` directory and run the Ansible playbooks as instructed by the script output.
-    *   **Standard Run**:
-        ```bash
-        cd kubespray
-        ansible-playbook -i inventory/prod/inventory.ini cluster.yml
-        ```
-    *   **Forced Upgrade**:
-        ```bash
-        cd kubespray
-        ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
-        ```
+### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
+
+#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
+```bash
+cd kubespray
+ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+```
+*Verification*: Check node readiness and that no application pods crashed.
+
+#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
+1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
+   ```yaml
+   kube_version: 1.32.13
+   ```
+2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
+   ```bash
+   sync-inventory
+   ```
+3. Run the in-place upgrade:
+   ```bash
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
+4. Verify the cluster:
+   ```bash
+   kubectl version
+   kubectl get nodes -o wide
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
+
+---
 
 ## Directory Structure
-*   `init.sh`: The entry point script. **Source of truth for Kubespray version.**
-*   `inventory/`: Contains cluster inventory configurations (hosts, variables). **Edit this, not the one in `kubespray/`**.
-*   `kubespray/`: (Ignored/Ephemeral) The checked-out Kubespray repository. **Do not edit files here directly**; they will be overwritten.
-
-## Configuration Updates
-To upgrade Kubespray or change cluster config:
-1.  **Version Upgrade**: Update `VERSION` in `init.sh` (e.g., to `release-2.28`).
-2.  **Config Changes**: Modify files in `./inventory/`.
-3.  **Apply**: Run `source init.sh` then execute the Ansible playbook.
+*   `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
+*   `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
+*   `inventory/`: Source of truth for cluster inventory and variables.
+*   `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.

bootstrap/README.md

@@ -1,114 +1,165 @@
 # Bootstrap
 
-The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder.
+This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
 
-The version to be initialized in configured in the `init.sh` script.
+---
+
+## Architecture & Single-Node Cluster Constraints
+
+`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
+
+> [!IMPORTANT]
+> ### Crucial Single-Node Invariants
+> 1. **Never Cordon or Drain the Node**:
+>    Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
+>    In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
+>    ```yaml
+>    drain_nodes: false
+>    upgrade_node_always_cordon: false
+>    upgrade_node_fail_if_drain_fails: false
+>    ```
+> 2. **In-Place Upgrades via `cluster.yml`**:
+>    Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
+>    ```sh
+>    ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+>    ```
+> 3. **Version Number Formatting**:
+>    Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
+>    ```yaml
+>    kube_version: 1.32.13   # Correct
+>    # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
+>    ```
+> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
+>    Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
+
+---
+
+## Quickstart
+
+### 1. Environment Initialization
+Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
 
 ```sh
-. ./init.sh
+source ./init.sh
 ```
 
-See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated.
+*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
 
-## Vagrant
+---
 
-```sh
-cd kubespray
-vagrant up
-# up and abkle to ssh
-vagrant ssh k8s-1
-```
+## Step-by-Step Upgrade Runbook
 
-## Prod
+Upgrades are conducted in two decoupled phases:
+1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
+2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
 
-Prepare server:
-* deactivate swap!
-* `moritz username     ALL=(ALL) NOPASSWD:ALL`
-* `
+### Step 0: Pre-Flight Checks & Backups
 
-```sh
-ssh centos@<ip>
-# auth via pw
-sudo su - root
-adduser moritz
-visudo # add as sudo user
-su - moritz
-sudo yum -y install vim python3
-ssh-keygen
-vim .ssh/authorized_users # paste key
-chmod 644 .ssh/authorized_keys
-# check whether login works with ssh key
-sudo vim /etc/ssh/sshd_config # remove pw auth & root login
-sudo yum upgrade -y && sudo reboot
-```
+1. **Verify SSH and sudo access**:
+   ```sh
+   ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
+   ```
 
-Install Kubernetes:
+2. **Check Pod Health**:
+   ```sh
+   ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
+   ```
 
-```sh
-. ./init.sh
-# follow instructions from output, sth like:
-cd kubespray
-ansible-playbook -i inventory/prod/inventory.ini cluster.yml
-```
+3. **Take a Manual etcd Snapshot**:
+   ```sh
+   ssh moritz@136.243.23.215 "sudo etcdctl \
+     --cacert=/etc/ssl/etcd/ssl/ca.pem \
+     --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
+     --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
+     snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
+   ```
 
-And get credentials:
+---
 
-```sh
-ssh <ip>
-sudo su - root
-cd
-cp -r .kube /home/moritz/
-chown -R moritz. /home/moritz/.kube
-#ctrl + d
-kubectl get ns # test connection
-#ctrl + d
-scp haumdaucher.de:/home/moritz/.kube/config .kube/config
-```
+### Step 1: Reconcile Kubespray Baseline (Phase 1)
 
-Foreward in k8s-directory.
+1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
+2. Run `source ./init.sh` to initialize the environment.
+3. Apply the baseline configuration:
+   ```sh
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+   ```
+4. Verify cluster health:
+   ```sh
+   kubectl get nodes -o wide
+   kubectl get pods -n kube-system
+   ```
 
-## Upgrade cluster
+---
 
-Check the current default value of `kube_version` in cloned repository.
+### Step 2: Upgrade Kubernetes Version (Phase 2)
 
-**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**
+1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
+   ```yaml
+   kube_version: 1.32.13
+   ```
 
-```sh
-cd kubespray
-ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
-# or just the newest version
-ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
-# upgrade to specific calico version (did not trigger/ failed)
-ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
-```
+2. Sync the configuration into the `kubespray/` workspace:
+   ```sh
+   sync-inventory
+   ```
 
-History:
+3. Run the in-place upgrade playbook:
+   ```sh
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
 
-* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12
+---
 
-## Add node
+### Step 3: Post-Upgrade Verification
 
-See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).
+1. **Check component versions**:
+   ```sh
+   kubectl version
+   ```
+   Ensure both `Client Version` and `Server Version` report `v1.32.13`.
 
-Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.
+2. **Verify Node status**:
+   ```sh
+   kubectl get nodes -o wide
+   ```
+   Ensure status is `Ready` and version matches `v1.32.13`.
 
-```sh
-ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
-ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
-```
+3. **Verify System Pods**:
+   ```sh
+   kubectl get pods -n kube-system
+   ```
+   All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
 
-This runs everything and is kind of idempotent:
+4. **Verify Workloads**:
+   ```sh
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
 
-```sh
-ansible-playbook -i inventory/prod/inventory.ini cluster.yml
-```
+---
 
-## Upgrade to 2.31.3
+## Troubleshooting & Edge Cases
 
-Required to execute: 
+### Kubeadm Preflight `CreateJob` Failure
+Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
+- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
+- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
 
-```
-ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e drain_nodes=false upgrade-cluster.yml
-```
+### Kubelet Housekeeping Stalls / Static Pod Timeout
+If `kubeadm upgrade apply` times out waiting for static pod hash changes:
+- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
+- If an old container unmount left a stale bind-mount busy, lazily detach it:
+  ```sh
+  sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
+  sudo systemctl restart kubelet
+  ```
 
-+ set a feature flag: https://github.com/kubernetes-sigs/kubespray/issues/11887
+---
+
+## Directory Layout
+* `init.sh`: Environment bootstrapping script and upgrade instructions.
+* `inventory/`: Source-of-truth inventory and variable files.
+* `patches/`: Git patches applied to upstream Kubespray on checkout.
+* `kubespray/`: (Ephemeral) Cloned Kubespray repository.

bootstrap/init.sh

@@ -1,60 +1,78 @@
 #!/bin/sh
 
-# check whether script is sourced
-[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "script is  executed, but needs to be sourced" && exit 1
+# Ensure script is sourced, not executed directly
+[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "Error: init.sh must be sourced: source ./init.sh (or . ./init.sh)" && exit 1
 
-echo "######################################################################################"
-echo "##  Reinit repository"
-rm -rf kubespray
-VERSION="release-2.27"
-git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git
+BOOTSTRAP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+VERSION="release-2.28" # Note: release-2.28 is the final release line supporting Ubuntu 20.04 LTS
 
-echo "######################################################################################"
-echo "##  Activating pyenv venv"
+echo "======================================================================================"
+echo "## [1/4] Checking / Cloning Kubespray ($VERSION)"
+echo "======================================================================================"
+if [ ! -d "$BOOTSTRAP_DIR/kubespray/.git" ]; then
+  rm -rf "$BOOTSTRAP_DIR/kubespray"
+  git clone --branch "$VERSION" --depth 1 https://github.com/kubernetes-sigs/kubespray.git "$BOOTSTRAP_DIR/kubespray"
+else
+  echo "Existing kubespray checkout detected. Ensuring correct branch..."
+  git -C "$BOOTSTRAP_DIR/kubespray" fetch origin "$VERSION" --depth 1
+  git -C "$BOOTSTRAP_DIR/kubespray" checkout "$VERSION"
+fi
+
+if [ -d "$BOOTSTRAP_DIR/patches" ]; then
+  for p in "$BOOTSTRAP_DIR"/patches/*.patch; do
+    if [ -f "$p" ]; then
+      echo "## Applying patch $(basename "$p")..."
+      git -C "$BOOTSTRAP_DIR/kubespray" apply --check "$p" 2>/dev/null && git -C "$BOOTSTRAP_DIR/kubespray" apply "$p" || echo "Patch $(basename "$p") already applied or skipped."
+    fi
+  done
+fi
+
+echo "======================================================================================"
+echo "## [2/4] Activating Python Virtualenv"
+echo "======================================================================================"
 eval "$(pyenv init -)"
 eval "$(pyenv virtualenv-init -)"
-pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
+pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
 pyenv activate infrapuzzle-bootstrap
-python -m pip install -r kubespray/requirements.txt
+python -m pip install -q -r "$BOOTSTRAP_DIR/kubespray/requirements.txt"
 
+echo "======================================================================================"
+echo "## [3/4] Syncing Inventory Configuration"
+echo "======================================================================================"
+rsync -a "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
 
-echo "######################################################################################"
-echo "##  Customizing vagrant dev env"
-mkdir -p kubespray/vagrant
-cat << EOF > kubespray/vagrant/config.rb
-\$instance_name_prefix = "k8s"
-\$vm_cpus = 4
-\$num_instances = 1
-\$os = "centos"
-\$subnet = "10.0.20"
-\$network_plugin = "calico"
-\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" }
-\$kube_node_instances_with_disks_number = 0
-EOF
+# Helper function for re-syncing changes made in ./inventory to ephemeral ./kubespray/inventory
+sync-inventory() {
+  rsync -av "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
+}
+export -f sync-inventory 2>/dev/null || true
+alias sync-inventory="rsync -av '$BOOTSTRAP_DIR/inventory/' '$BOOTSTRAP_DIR/kubespray/inventory/'"
 
-# make the rpm cache
-mkdir -p kubespray/temp/docker_rpms
-
-echo "###############"
-echo "Now cd to kubespray and 'vagrant up'"
+echo "======================================================================================"
+echo "## [4/4] Environment Ready! Single-Node Cluster Upgrade Workflow"
+echo "======================================================================================"
 echo ""
-echo "export KUBECONFIG=\"$( pwd )/kubespray/inventory/sample/artifacts/admin.conf\""
-
-
-echo "######################################################################################"
-echo "## * syncing config in kubespray dir"
-rsync -a ./inventory/ ./kubespray/inventory/
-echo "## * changing into kubespray dir"
-cd kubespray
-echo "## * execute the following command to force new settings"
-echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
-echo "## * execute the following command to force new settings with forced cluster upgrade"
-echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
-
-# cp -r kubespray/inventory/sample kubespray/inventory/prod
-# rm kubespray/inventory/prod/inventory.ini
-# cp ./prod.ini kubespray/inventory/prod/inventory.ini
-#gsed -i "s/kube_network_plugin: .*/kube_network_plugin: flannel/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml
-# gsed -i "s/# calico_iptables_backend: \"Legacy\"/calico_iptables_backend: \"NFT\"/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml
-#echo 'calico_iptables_backend: "NFT"' >> ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-cluster.yml
-#gsed -i "s/metrics_server_enabled: .*/metrics_server_enabled: true/" ./kubespray/inventory/prod/group_vars/k8s-cluster/addons.yml
+echo "Current directory changed to: $BOOTSTRAP_DIR/kubespray"
+cd "$BOOTSTRAP_DIR/kubespray"
+echo ""
+echo "=== PRE-UPGRADE CHECKLIST ==="
+echo "1. Verify SSH connection and sudo access:"
+echo "   ssh moritz@136.243.23.215 'sudo kubectl get nodes -o wide'"
+echo "2. Take manual etcd snapshot before upgrading:"
+echo "   ssh moritz@136.243.23.215 'sudo etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db'"
+echo ""
+echo "=== TWO-PHASE UPGRADE WORKFLOW ==="
+echo "Phase 1: Reconcile Kubespray baseline (keeps current k8s version):"
+echo "   ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
+echo ""
+echo "Phase 2: Upgrade Kubernetes version (e.g. to 1.32.13):"
+echo "   a) Update 'kube_version: <target>' in ../inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml"
+echo "   b) Sync inventory: sync-inventory"
+echo "   c) Run in-place single-node upgrade:"
+echo "      ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
+echo ""
+echo "=== POST-UPGRADE VERIFICATION ==="
+echo "   kubectl version"
+echo "   kubectl get nodes -o wide"
+echo "   kubectl get pods -A | grep -v -E 'Running|Completed'"
+echo "======================================================================================"

bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml

@@ -27,7 +27,17 @@ kube_kubeadm_apiserver_extra_args:
 kube_api_anonymous_auth: true
 
 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.31.9
+kube_version: 1.32.13
+
+# Single-node cluster settings: prevent draining or cordoning the only node
+drain_nodes: false
+upgrade_node_always_cordon: false
+upgrade_node_fail_if_drain_fails: false
+kubeadm_ignore_preflight_errors:
+  - ImagePull
+  - SystemVerification
+  - FileContent--proc-sys-net-bridge-bridge-nf-call-iptables
+  - CreateJob
 
 # kubernetes image repo define
 #kube_image_repo: "k8s.gcr.io"
@@ -122,6 +132,9 @@ kube_proxy_mode: iptables
 # Moritz: changed to iptables for tegra
 #kube_proxy_mode: ipvs
 
+# Expose kube-proxy metrics on all interfaces so Prometheus can scrape port 10249
+kube_proxy_metrics_bind_address: "0.0.0.0:10249"
+
 # configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
 # must be set to true for MetalLB to work
 kube_proxy_strict_arp: false

bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch

@@ -0,0 +1,20 @@
+diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+index 523956d1d..75723a6e0 100644
+--- a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
++++ b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+@@ -17,6 +17,7 @@
+     --force
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     {%- if kube_version is version('1.32.0', '>=') %}
+     --skip-phases={{ kubeadm_init_phases_skip | join(',') }}
+@@ -38,6 +39,7 @@
+     {% if kubeadm_patches | length > 0 %}--patches={{ kubeadm_patches_dir }}{% endif %}
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     --skip-phases={{ kubeadm_upgrade_node_phases_skip | join(',') }}
+   register: kubeadm_upgrade

docs/adr/0001-hybrid-mcp-tool-search-and-subagents.md

@@ -0,0 +1,148 @@
+---
+status: "accepted"
+date: 2026-09-12
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0001: Hybrid MCP Tool Search and Subagent Delegation for AI Agents
+
+## Context and Problem Statement
+
+OpenClaw operates as the personal AI concierge for the Graf family, communicating primarily via Telegram and web interfaces. It integrates external systems via the Model Context Protocol (MCP) (e.g., Home Assistant, Prometheus, Spotify, weather, shopping).
+
+When integrating comprehensive domain services like **Garmin Connect** via [`Taxuspt/garmin_mcp`](https://github.com/Taxuspt/garmin_mcp), the server exposes **148 distinct tools** spanning activity analysis, workout builders, physiological trends, sleep, stress, HRV, and gear tracking. 
+
+In standard MCP implementations, all tool schemas are transmitted in the LLM's `tools` array on *every single request turn*. For 148 tools, this imposes a **~25,000–30,000 token tax per turn** before a single word of user prompt is evaluated. This creates severe issues:
+1. **Severe Token & Cost Bloat**: Inflates inference costs on OpenRouter across thousands of daily family turns.
+2. **Turn Latency & Degradation**: Slows down response times on mobile Telegram and impairs the reasoning accuracy of fast conductor models (e.g. `deepseek-v4.1-flash`) due to tool schema noise.
+3. **Imbalance for Simple Checks**: A simple query like *"Wie war mein Schlaf?"* should not require dragging 148 tool definitions into context or, conversely, incurring the multi-second startup latency of a full subagent session.
+
+How do we grant OpenClaw full access to large, complex MCP tool catalogs without polluting everyday conversational context?
+
+---
+
+## Decision Drivers
+
+* **Token Economics**: Minimize idle token overhead on routine conversational turns.
+* **Low Latency**: Provide instant answers for routine, single-step lookups.
+* **Complex Multi-Step Capabilities**: Enable autonomous, deep reasoning for tasks like workout generation, periodization, and multi-source research without cluttering the primary conversation transcript.
+* **Protocol Standards**: Adhere to upstream MCP standards rather than writing fragile, non-standard CLI wrappers for every API.
+* **Architectural Consistency**: Establish an extensible pattern that can later be applied to refactor other domains (e.g. Supermarket shopping, Jellyfin media, and Regensburg Stadtrat local news research).
+
+---
+
+## Considered Options
+
+1. **Option 1: Global Eager MCP Registration**
+   * Register all 148 tools directly under `mcp.servers`.
+   * *Drawback*: Unacceptable ~30k token tax on every message; high cost and latency.
+
+2. **Option 2: Pure CLI / Script Wrapper Skill**
+   * Avoid MCP entirely; write custom Python CLI scripts in `mop-skills` executed via `bash`.
+   * *Drawback*: Re-invents the wheel, ignores the 1,150-star community MCP server, requires maintaining hundreds of lines of bespoke CLI parsing code.
+
+3. **Option 3: Subagent Delegation Only**
+   * Hide all domain tools from the main agent and mandate that *every* query spawns a subagent via `sessions_spawn`.
+   * *Drawback*: Adds unnecessary latency (2–4 seconds) and session overhead for trivial questions like *"Wie hoch ist meine Body Battery?"*.
+
+4. **Option 4: Hybrid Tool Search + Subagent Delegation (Selected)**
+   * Enable OpenClaw's native **Tool Search** (`tools.toolSearch.mode: "directory"`) for the main agent so large tool catalogs are deferred behind lightweight search/describe/call operations (~600 tokens).
+   * Empower the main conductor to dynamically decide:
+     * Execute directly via Tool Search for fast, single-step queries.
+     * Delegate to an isolated subagent (`sessions_spawn`) for deep, multi-step, stateful reasoning.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 4: Hybrid Tool Search + Subagent Delegation**.
+
+### 1. Dual-Path Architecture
+
+```
+User Query (Telegram)
+         │
+         ▼
+ ┌────────────────────────────────────────┐
+ │       OpenClaw Main Conductor          │
+ │       (DeepSeek v4.1 Flash)            │
+ └───────────────────┬────────────────────┘
+                     │
+         Is this a deep/multi-step task?
+        ┌────────────┴────────────┐
+        │ NO                      │ YES
+        ▼                         ▼
+ ┌───────────────┐        ┌─────────────────────────┐
+ │ Direct Tool   │        │ Subagent Delegation     │
+ │ Search        │        │ (sessions_spawn)        │
+ ├───────────────┤        ├─────────────────────────┤
+ │ • tool_search │        │ • Dedicated Context     │
+ │ • tool_call   │        │ • Full Tool Access      │
+ │ • Single Turn │        │ • Multi-Step Reasoning  │
+ └───────┬───────┘        └────────────┬────────────┘
+         │                             │
+         └──────────────┬──────────────┘
+                        ▼
+       ┌─────────────────────────────────┐
+       │   Taxuspt/garmin_mcp (148 Tools)│
+       │   Garmin Connect Cloud API      │
+       └─────────────────────────────────┘
+```
+
+### 2. Main Conductor Routing Heuristic
+
+The main conductor is guided by explicit system prompt rules:
+
+| Query Type | Routing Path | Examples |
+| :--- | :--- | :--- |
+| **Simple Vitals & Lookups** | **Direct Tool Search** | • *"Wie war mein Schlaf letzte Nacht?"*<br>• *"Wie ist meine Body Battery?"*<br>• *"Was habe ich gestern trainiert?"* |
+| **Deep Analysis & Creation** | **Subagent Delegation (`sessions_spawn`)** | • *"Erstelle mir ein 45-minütiges Oberkörper-Superset-Workout für heute auf meiner Uhr."*<br>• *"Analysiere mein Krafttraining der letzten 3 Wochen auf Progressive Overload und plane die nächste Woche."*<br>• *"Erstelle ein Periodisierungskonzept für den nächsten Monat."* |
+
+---
+
+## Consequences
+
+### Positive
+* **98% Token Reduction on Idle Turns**: The tool catalog is deferred behind OpenClaw's native Tool Search, dropping tool schema overhead from ~30,000 tokens to ~600 tokens.
+* **Instant Answers for Vitals**: Quick checks are answered in a single turn without waiting for subagent spin-up.
+* **Isolated Deep Reasoning**: Heavy multi-turn planning and JSON compilation occur inside temporary subagents, keeping the main Telegram conversation transcript lightweight and readable.
+* **Standard-Compliant**: Uses standard upstream MCP (`Taxuspt/garmin_mcp`) and standard OpenClaw features (`tools.toolSearch`, `sessions_spawn`).
+
+### Negative & Mitigations
+* **Two-Step Tool Search Latency**: For the first direct lookup of a tool, OpenClaw performs a search step before calling the tool.
+  * *Mitigation*: OpenClaw's `directory` mode caches known tool descriptors and allows direct calls when the exact tool name is known.
+* **Model Prompt Instruction**: Requires clear guidelines in the system prompt so the conductor consistently knows when to delegate vs. query directly.
+
+---
+
+## Future Roadmap: Monorepo Refactoring Blueprint
+
+This Hybrid MCP architecture serves as the official template for all agentic workflows in `infrapuzzle`. On 2026-09-16, this architecture was formally expanded into the **Full-Spectrum Multi-Agent Architecture** with 9 specialized subagents:
+
+### 1. Agents-as-Code in the Knowledge Wiki (`knowledge-wiki/agents/`)
+Subagent specifications, system prompts, tool assignments, and domain guidelines are decoupled from Kubernetes deployment manifests and stored as versioned Markdown files in `knowledge-wiki/agents/<name>.md`.
+* **Zero-Deployment Tuning**: Adding or refining an agent requires no Kubernetes pod restarts; changes synced via Git are immediately effective.
+* **Extensibility Blueprint**: Adding a 10th agent requires simply adding `agents/<new-agent>.md` with YAML frontmatter specifying model, tools, and owned concepts.
+
+### 2. The 9 Specialized Subagents Roster
+1. 🩺 **Digitale Hebamme Agent** (`agents/hebamme.md`): Baby milestones, sleep windows, BLW, pediatrician safety boundaries.
+2. 🍳 **Koch Agent** (`agents/koch.md`): Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization, exact recipes.
+3. 🏡 **Grabengasse Agent** (`agents/grabengasse.md`): Home Assistant, Philips Hue, Sonos multiroom, house systems.
+4. 🏰 **Regensburg Agent** (`agents/regensburg.md`): Regensburg & Stadtamhof insider, local places, RVV transit, city notices.
+5. ✈️ **Travel Agent** (`agents/travel.md`): Vacation planning, itineraries, packing checklists, on-the-road travel concierge.
+6. 🎬 **Media Agent** (`agents/media.md`): Jellyfin streaming/requests, Jellyseerr, Spotify audiobooks & playlists.
+7. 🔍 **Recherche Agent** (`agents/recherche.md`): Deep research engine (iterative multi-hop), ALLRIS Stadtrat, Friday bridge updates.
+8. 🛒 **Shopping Agent** (`agents/shopping.md`): Deal hunter, Geizhals all-time lows, supermarket deals, multi-tier webshop extraction.
+9. 🏋️ **Fitness Coach Agent** (`agents/fitness.md`): Garmin Connect (148 tools), Tacx Flux 2 (FTP 275 W), home gym supersets.
+
+### 3. Multi-Turn Continuity: Domain Stickiness & Rich State Bridge
+To prevent context loss during ongoing step-by-step dialogues (e.g. interactive cooking or workout design):
+* **Persistent Context Continuity**: Follow-up turns are routed within the dedicated subagent topic session using native conversational memory, keeping user-facing messages clean without embedding raw JSON/HTML metadata tags.
+
+### 4. 3-Tier Session Lifecycle (Preventing Unbounded Session Growth)
+1. **Conversational Close**: Topic shifts or explicit user completion (*"Danke, das war's"*) reset the active domain pointer.
+2. **Ephemeral Inactivity Cleanup**: Subagent child sessions inactive for > 60 minutes (`subagents.archiveAfterMinutes: 60`) are automatically archived.
+3. **Deterministic Nightly Compaction**: Nightly 03:30 AM CET compaction vacuums expired child sessions from the SQLite store.
+

docs/adr/0002-openclaw-model-selection-tiering-and-economics.md

@@ -0,0 +1,258 @@
+---
+status: "accepted (fleet-wide primary: DeepSeek-V4.1-Flash-1017; updated 2026-10-17)"
+date: 2026-09-13
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0002: OpenClaw Full-Spectrum Model Selection, Multimodal Tiering, and Cost Governance
+
+## Context and Problem Statement
+
+OpenClaw serves as the 24/7 personal AI concierge, family assistant, and operational orchestrator for Moritz Graf and his family. It interacts across Telegram, voice notes, webhooks, and background automations, bridging to personal knowledge (Obsidian wiki), smart home systems (Home Assistant, Sonos), fitness tracking (Garmin Connect), and cluster monitoring (Prometheus, Gitea).
+
+During early September 2026, OpenClaw consumed **143.26 million tokens** over 7 days, accumulating **\$12.55+** in raw OpenRouter API costs. Forensic analysis revealed that over **80% of this spend was architectural waste**:
+1. **Unbounded Context Re-transmission**: An uncompacted session (`agent:main:main`) reached 381k tokens (2,650 events, 5.7 MB JSON). Every minor turn re-ingested this entire history, costing \$12.20 (97.2% of total spend).
+2. **Heartbeat Amplification Loop**: Background heartbeats firing every 30 minutes targeted the active user session (`target: last`), re-evaluating the 381k context 48 times/day simply to output `NO_REPLY` (\$5.16 wasted).
+3. **Expensive & Deprecated Model Allocations**: Frontier models (`claude-3.7-sonnet` and `deepseek-v4-pro`) were configured without clear cost-benefit boundaries, with subagents frequently timing out (>300s). In addition, legacy model generations (e.g., Gemini 2.5) lingered in configuration despite superior, newer generations being available.
+4. **Modality Conflation (Image Input vs. Image Generation)**: Vision perception (analyzing an uploaded photo) was conflated with text-to-image synthesis (drawing an image). Configuring chat models like `z-ai/glm-5.3-flash` for image synthesis caused runtime HTTP 404 failures because pure LLMs lack diffusion/raster generation heads.
+
+Moritz established two foundational requirements for the repository:
+> 1. *"I want high quality, but it should not be too expensive."*
+> 2. *"We always want to use the newest models (as they usually are the best)."*
+
+How do we architect the full-spectrum model selection across **all modalities** (text, subagents, heartbeats, vision, image generation, audio transcription, text-to-speech, web search, and vector embeddings) to deliver premium quality, sub-10s latency, and a total operating budget of **<\$2.00 / week**?
+
+---
+
+## Decision Drivers
+
+* **State-of-the-Art Recency**: Exclusively employ the latest generation models (e.g. DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, Sonar Pro). Completely decommission legacy generations (e.g. Gemini 2.5).
+* **Conversational & Reasoning Quality**: High emotional intelligence, nuanced German fluency, personal context recall, and strict adherence to tool-calling protocols (ADR 0001).
+* **Strict Modality Decoupling**: Each capability must route to its dedicated, best-in-class engine:
+  - Text & Agentic Reasoning
+  - Native Multimodal Vision (reading user photos)
+  - Text-to-Image Generation (synthesizing new pictures)
+  - Speech-to-Text / Audio Input (transcribing voice messages)
+  - Text-to-Speech / Audio Output (synthesizing briefings and podcasts)
+  - Web Search & Retrieval (grounded search)
+  - Semantic Embeddings (vector memory)
+* **Inference Economics**: Low per-token costs for large conversational contexts (15k–80k tokens), high prompt-cache discount efficiency (`cacheRead`), zero background bleed, and predictable billing.
+* **Latency & Turn Responsiveness**: Conversational turns on Telegram must return within **4–10 seconds**, avoiding provider queuing or massive prefill bottlenecks.
+* **Resilience & Fault Tolerance**: Automated fallback chains and strict quantization/provider pins (`fp8`/`bf16`, `tool_choice` support) across independent providers on OpenRouter.
+
+---
+
+## Full-Spectrum Model Architecture & Concrete Allocations
+
+```
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                       OpenClaw Ingress Gateway                                         │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+         ┌───────────────────┬─────────────────────┼─────────────────────┬───────────────────┐
+         ▼                   ▼                     ▼                     ▼                   ▼
+┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
+│ Text Conductor  │ │ Subagent Engine │   │ Background &    │   │ Image Synthesis │ │ Voice & Audio   │
+│ & Vision Input  │ │ (sessions_spawn)│   │ Heartbeat       │   │ (Text-to-Image) │ │ (STT & TTS)     │
+├─────────────────┤ ├─────────────────┤   ├─────────────────┤   ├─────────────────┤ ├─────────────────┤
+│ Tier 1          │ │ Tier 2          │   │ Tier 3          │   │ Tier 4          │ │ Tier 5          │
+│ • Low Latency   │ │ • Deep Analysis │   │ • target: none  │   │ • Nano Banana 2 │ │ • Whisper V3    │
+│ • Tool Routing  │ │ • Autonomous    │   │ • Zero Bleed    │   │ • Dedicated     │ │ • Gemini 3.1    │
+│ • Decoupled OCR │ │ • Isolated Run  │   │ • 30m Polling   │   │   Raster Head   │ │   Flash TTS     │
+└────────┬────────┘ └────────┬────────┘   └────────┬────────┘   └────────┬────────┘ └────────┬────────┘
+         │                   │                     │                     │                   │
+         ▼                   ▼                     ▼                     ▼                   ▼
+┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
+│ Primary:        │ │ Primary:        │   │ Primary:        │   │ Primary:        │ │ STT:            │
+│ deepseek/       │ │ deepseek/       │   │ deepseek/       │   │ google/         │ │ openai/         │
+│ deepseek-v4.1-  │ │ deepseek-v4.1-  │   │ deepseek-v4.1-  │   │ gemini-3.1-     │ │ whisper-large-  │
+│ flash-1017      │ │ flash-1017      │   │ flash-1017      │   │ flash-image     │ │ v3-turbo        │
+│                 │ │                 │   │                 │   │                 │                   │
+│ Fallbacks:      │ │ Fallbacks:      │   │ Fallbacks:      │   │ Fallbacks:      │ │ TTS:            │
+│ 1. v4.1-flash   │ │ 1. v4.1-flash   │   │ 1. v4.1-flash   │   │ 1. gemini-3.1-  │ │ google/         │
+│ 2. glm-5.3-flash│ │ 2. glm-5.3-flash│   │ 2. glm-5.3-flash│   │    flash-preview│ │ gemini-3.1-     │
+│ 3. qwen3.8-flash│ │ 3. qwen3.8-flash│   │ 3. qwen3.8-flash│   │ 2. gemini-3-pro-│ │ flash-tts       │
+│                 │ │                 │   │                 │   │    image        │ │ (FB: MiniMax 2.8│
+│ Vision (OCR):   │ │                 │   │                 │   │                 │ │  Speech HD)     │
+│ google/         │ │                 │   │                 │   │                 │ │                 │
+│ gemini-3.8-     │ │                 │   │                 │   │                 │ │                 │
+│ flash           │ │                 │   │                 │   │                 │ │                 │
+└─────────────────┘ └─────────────────┘   └─────────────────┘   └─────────────────┘ └─────────────────┘
+```
+
+---
+
+## Modality Specifications & Cost-Benefit Analysis
+
+### 1. Interactive Conductor & Agentic Reasoning (Tier 1)
+* **Primary (Fleet-Wide Standard as of 2026-10-17)**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
+  - **1017 Snapshot Upgrade (Pareto Jump in Quality & Price)**: Released on October 17, 2026, `deepseek/deepseek-v4.1-flash-1017` boosts the Artificial Analysis Intelligence Index from **24.7 to 39.5** (+60%) via Reinforcement Learning with Self-Feedback (RLSF) while cutting DeepInfra `fp8` pricing by an additional **30%** (`$0.098/M` prompt, `$0.294/M` completion, and **`$0.00294/M` `cacheRead`** vs. `$0.140/$0.420/$0.0042` on the 0910 baseline).
+  - **Provider Pinning & Tool-Compatibility Verification**:
+    - Pinned `order: ["deepinfra", "cloudflare", "inference-net", "fireworks", "parasail"]`, `quantizations: ["fp8", "bf16", "unknown"]`, and `ignore: ["morph", "baseten", "deepseek", "open-inference", "decart", "sail-research", "wafer"]`.
+    - **Cloudflare (#2 Promotion)**: Promoted ahead of InferenceNet and Parasail because Cloudflare delivers a 97% cache discount (`$0.003/M` `cacheRead`), 3× lower completion costs (`$0.60/M` vs. `$1.20/M` on Parasail), and a massive **943,718** completion token limit.
+    - **Fireworks (#4 Addition)**: Onboarded as a high-reliability enterprise tier-1 provider (`$0.30/M` prompt, `$1.20/M` completion, `$0.006/M` `cacheRead`, 100% uptime, 943k max completion tokens, full tool choice and structured outputs support).
+    - **Completion Token Headroom Guardrail (`maxTokens: 65536`)**: Because reasoning models share the completion token budget between internal chain-of-thought (`reasoningTokens`, typically 2k–4k tokens) and the actual tool-call payload, omitting `maxTokens` previously allowed OpenClaw/OpenRouter to default to an 8,192 token ceiling. During large multi-kilobyte code or website generation, hitting 8,192 tokens abruptly truncated tool-call JSON mid-payload (`Provider returned an incomplete or malformed tool call`). Explicitly configuring `maxTokens: 65536` and `params.max_tokens: 65536` provides ample headroom for complex reasoning and large outputs.
+    - Live OpenRouter routing forensics verified that `morph/fp8` is filtered out by OpenRouter's `Filter by Tool Compatibility` when `tools` are passed, `deepseek` (official) is filtered by account privacy guardrails (`Paid model training violation`), and `baseten` returns `HTTP 429`. Meanwhile, `deepinfra`, `cloudflare`, `inference-net`, `fireworks`, and `parasail` deliver 100% tool-calling reliability and sub-second to 3s latency.
+  - **Post-Trial Evaluation of `z-ai/glm-5.3-flash` (2026-09-13 to 2026-10-17)**:
+    - While `z-ai/glm-5.3-flash` was trialed as primary conductor, 17-day production telemetry (Oct 1–17, 2026) revealed **12 mid-stream tool-call aborts** (`stopReason: "error"`, `errorMessage: "Provider returned an incomplete or malformed tool call"`) across 5 agents (`main`, `grabengasse_media`, `koch`, `hebamme`, `doktor`) when generating nested `tool_call({ id, args })` payloads after `tool_describe`, as well as occasional thinking-only whitespace stalls (Issue 23) and long-context attention drift (Issues 22 & 25).
+    - Promoting `deepseek/deepseek-v4.1-flash-1017` to primary across all agents eliminates malformed tool-call stream aborts AND lowers total monthly cost by >50% compared to GLM 5.3 Flash.
+  - **Decoupled Vision Override**: `deepseek/deepseek-v4.1-flash-1017`, `deepseek/deepseek-v4.1-flash`, and `z-ai/glm-5.3-flash` are registered with `input: ["text"]` under `models.providers.openrouter.models` so that OpenClaw always routes user photos through `tools.media.image` (`google/gemini-3.8-flash`) for high-precision German OCR before handing enriched text to the chat model (Issue 29).
+* **Fallbacks**:
+  1. `openrouter/deepseek/deepseek-v4.1-flash` (`0910` baseline across 25+ providers, pinned to `deepinfra`, `streamlake`, `inference-net`, `digitalocean`, `venice`, `nebius`).
+  2. `openrouter/z-ai/glm-5.3-flash`: Secondary fallback pinned to `fp8`/`bf16` providers (`morph`, `near-ai`, `digitalocean`, `venice`, `streamlake`).
+  3. `openrouter/qwen/qwen3.8-flash`: Provider diversity.
+
+### 2. Autonomous Subagents (Tier 2)
+* **All Subagents & Specialists**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
+  - **Execution**: Spawned via `sessions_spawn` with a 300-second timeout. Provides high-precision reasoning, rock-solid nested JSON tool loops (`tool_search` -> `tool_describe` -> `tool_call`), and zero syntax drift.
+* **Fallback**: `openrouter/deepseek/deepseek-v4.1-flash` / `openrouter/z-ai/glm-5.3-flash` / `openrouter/qwen/qwen3.8-flash`.
+
+### 3. Background Daemon & Heartbeats (Tier 3)
+* **Configuration**: `heartbeat.model: "openrouter/deepseek/deepseek-v4.1-flash-1017"`, `heartbeat.target: none`, `heartbeat.lightContext: true`.
+* **Rationale**: Because background heartbeats have a 75%+ `cacheRead` share (`22.86M` `cacheRead` vs. `7.73M` uncached input over 17 days), DeepSeek V4.1 Flash 1017's \$0.00294/M cache read rate on DeepInfra reduces 17-day heartbeat cost to <\$0.90 while keeping prompt cache warm on the primary provider.
+
+### 4. Vision Input vs. Text-to-Image Generation (Tier 4)
+* **Crucial Distinction**:
+  - **Vision Input (`imageModel` vs. Dedicated Primary)**: Reading photos sent to Telegram (e.g. food ingredients, nutrition tables, plants, receipts). Note: OpenClaw only invokes `imageModel` if the primary model *lacks* image input modality. Since `z-ai/glm-5.3-flash` advertises image modality on OpenRouter, OpenClaw does not auto-divert to `imageModel`. Therefore, vision-intensive agents (specifically `koch`) must explicitly configure `primary: "openrouter/google/gemini-3.8-flash"` to leverage Google's SOTA OCR and dense-table recognition.
+  - **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here.
+* **Image Synthesis Engine**:
+  - **Primary**: `openrouter/google/gemini-3.1-flash-image` (Nano Banana 2).
+    - **Cost**: \$0.00006 per generated image + \$0.50/1M prompt.
+    - **Latency**: ~4.5 seconds.
+  - **Fallbacks**:
+    1. `openrouter/google/gemini-3.1-flash-image-preview`
+    2. `openrouter/google/gemini-3-pro-image` (Nano Banana Pro)
+  - **Decommissioned**: Gemini 2.5 is completely removed from all image configurations.
+
+### 5. Voice, Audio & Podcast Synthesis (Tier 5)
+* **Speech-to-Text (Audio Input)**:
+  - **Engine**: `openrouter/openai/whisper-large-v3-turbo`
+  - **Capabilities**: Fast, highly accurate transcription of German and English Telegram voice notes, with local gateway fallback.
+* **Text-to-Speech (Audio Dossiers & Podcasts)**:
+  - **Engine**: `google/gemini-3.1-flash-tts-preview` via `text-to-audio` skill.
+  - **Voices**:
+    - German Solo/Host: `Puck` (warm, natural, dynamic).
+    - German Deep/Calm: `Charon`.
+    - German Co-Host / Expert: `Aoede`.
+    - English: `Zephyr`.
+  - **Fallback**: `minimax/speech-2.8-hd` for studio-grade multi-speaker podcasts when Google TTS is throttled.
+
+### 6. Grounded Web Search & Vector Embeddings
+* **Live Search**: `perplexity/sonar-pro` via OpenRouter (replacing raw web scraping with grounded citations).
+* **Memory Embeddings**: `text-embedding-3-small` via OpenRouter for the Obsidian personal knowledge wiki.
+
+---
+
+## Context Lifecycle & Daytime Headroom Policy
+
+To balance the user's desire to retain full conversation context across the entire day while avoiding the 70-second latency penalty that uncompacted transcripts impose on OpenRouter providers:
+
+1. **Daytime Headroom (`maxActiveTranscriptBytes: "3mb"`)**:
+   The active transcript is permitted to retain the day's back-and-forth interactions so OpenClaw recalls earlier topics and instructions naturally.
+2. **Deterministic Nightly Compaction**:
+   Scheduled daily at **03:30 AM Europe/Berlin** (`ce39bbf8-b83f-453e-be53-04e27db27b78`):
+   ```bash
+   openclaw sessions compact "agent:main:main" --max-lines 250
+   ```
+   Executes in **2.1 seconds** using **0 LLM tokens**, ensuring every morning begins with a lean, fast ~15k token transcript.
+
+---
+
+## Open Clarifying Questions for Moritz
+
+To align on remaining edge cases before final sign-off:
+
+1. **Active Trial with GLM-5.3-Flash as Primary Conductor**:
+   - As requested, `z-ai/glm-5.3-flash` is now deployed as the Primary Conductor for interactive chat and autonomous subagents, equipped with the prompt guardrails against direct tool calls and XML leakage.
+   - `deepseek/deepseek-v4.1-flash` stands ready as the zero-latency primary fallback.
+   - We will observe real-world performance, tool adherence, and cost over the next several days.
+2. **Thinking / Reasoning Mode on Telegram**:
+   - Currently, `thinking: medium` is enabled, emitting 100–300 reasoning tokens before generating conversational text.
+   - *Question*: Would you like to disable thinking mode (`thinking: none`) for standard Telegram chat turns (for near-instant replies) and keep reasoning active only for subagents?
+3. **On-Demand Frontier Model**:
+   - With all standard workloads handled by Flash-tier models, do you ever want an explicit escape hatch to a frontier model (e.g. `claude-3.7-sonnet` or `gemini-3.1-pro-preview` triggered exclusively via `/expert <query>`)?
+
+---
+
+## OpenRouter Provider Qualification Criteria & Re-Evaluation Playbook
+
+Whenever a new model snapshot (such as `deepseek/deepseek-v4.1-flash-1017`) is evaluated or re-checked as new hosting providers join OpenRouter, every candidate provider MUST pass the following **7-point qualification gate** before being added to `extra_body.provider.order` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml):
+
+### 1. Hard Technical Gates (Must Pass All)
+1. **Quantization Precision (`fp8` / `bf16` Only)**:
+   - **Rule**: Must run `fp8` or `bf16` (or verified full-precision `unknown` from trusted cloud hosts like Cloudflare).
+   - **Reject**: Any `fp4`, `nvfp4`, `mxfp4`, or `int4` quantization (`decart`, `sail-research`, `wafer`, `open-inference`, `phala` fp4). 4-bit quantizations severely degrade nested JSON tool-call syntax and German grammar.
+2. **OpenRouter Routing Funnel & Guardrail Compatibility (`tools` + `tool_choice: "auto"` Pass)**:
+   - **Rule**: Must not be stripped by OpenRouter's internal routing filters (`Filter by Tool Compatibility` or `Filter by Guardrails`) when a request includes `tools: [...]` and `tool_choice: "auto"`.
+   - **Why**: Metadata in `/api/v1/models/.../endpoints` can claim `supports_tool_choice: true` even when OpenRouter's live router blocks the provider on actual tool calls (e.g. `morph/fp8` is stripped by `Filter by Tool Compatibility`, and `deepseek` official is stripped by `Paid model training violation` account privacy settings).
+3. **Nested MCP `tool_call` Serialization Integrity (2/2 Live Pass)**:
+   - **Rule**: Must cleanly emit `finish_reason: "tool_calls"` with valid, parseable nested JSON arguments (`{"id": "mcp:...", "args": {...}}`) across back-to-back test runs without mid-stream SSE aborts (`Provider returned an incomplete or malformed tool call`) or HTTP `429` upstream rate limits.
+4. **Context Window & Completion Headroom**:
+   - **Rule**: `context_length >= 262,144` (1,048,576 preferred) and `max_completion_tokens >= 131,072`.
+   - **Why**: Active daytime sessions with system prompts and tool schemas reach 20k–80k tokens.
+5. **Uptime & Reliability**:
+   - **Rule**: `uptime_last_1d >= 95%` and `uptime_last_30m == 100%`.
+
+### 2. Economic & Latency Ranking (How to Order Passing Providers)
+Among providers passing all 5 technical gates, rank them in `extra_body.provider.order` by **effective production cost** and **turn latency**:
+1. **Implicit Prompt Cache Read Rate (`input_cache_read`) — Highest Weight**:
+   - Because **73%–94%** of OpenClaw's token volume (both interactive multi-turn chats and 30m background heartbeats) consists of cached prompt reads (`cacheRead`), a provider with `supports_implicit_caching: true` and a **95%–98% cache discount** (`<= $0.005 / 1M` tokens, e.g. `deepinfra` at `$0.00294/M` or `streamlake` at `$0.0030/M`) is vastly cheaper in net monthly spend than a provider with slightly cheaper uncached input but weak or missing `cacheRead` discounts (e.g. `nebius` with `cache_read: null` or `modal`/`primeintellect` at `$0.030/M`).
+2. **Base Prompt & Completion Pricing**:
+   - Target `prompt <= $0.18 / 1M` and `completion <= $0.65 / 1M` for top-priority slots; allow higher-priced providers (`<= $0.30 / $1.20`) only in lower fallback slots (#3/#4) for outage resilience.
+3. **End-to-End Tool-Call Latency**:
+   - Target **1.0s – 5.5s** for a ~300-token reasoning + tool-call turn.
+
+### 3. One-Shot Live Verification Command (For Future Checks)
+Any agent (including a fast/lightweight model) can audit all currently available OpenRouter endpoints for `deepseek/deepseek-v4.1-flash-1017` in one step by running:
+
+```bash
+# 1. Inspect current endpoint metadata, pricing, cache_read, and 24h uptime:
+curl -s https://openrouter.ai/api/v1/models/deepseek/deepseek-v4.1-flash-1017/endpoints | \
+  jq -r '.data.endpoints[] | "\(.provider_name) | tag=\(.tag) | quant=\(.quantization) | ctx=\(.context_length) | prompt=\(.pricing.prompt) | comp=\(.pricing.completion) | cache=\(.pricing.input_cache_read) | uptime1d=\(.uptime_last_1d)"'
+
+# 2. Live-test candidate providers inside openclaw-0 with real nested tool_call payloads:
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c '
+import os, json, time, urllib.request
+api_key = os.environ["OPENROUTER_API_KEY"]
+tools = [{"type": "function", "function": {"name": "tool_call", "parameters": {"type": "object", "properties": {"id": {"type": "string"}, "args": {"type": "object"}}, "required": ["id", "args"]}}}]
+candidates = ["deepinfra", "streamlake", "inference-net", "venice", "digitalocean", "novita", "parasail", "cloudflare", "nebius"]
+for prov in candidates:
+    req = urllib.request.Request(
+        "https://openrouter.ai/api/v1/chat/completions",
+        data=json.dumps({
+            "model": "deepseek/deepseek-v4.1-flash-1017",
+            "messages": [{"role": "user", "content": "Call tool_call with id=\"mcp:homeassistant:homeassistant__ha_get_state\" and args={\"entity_id\": \"media_player.bad\"}"}],
+            "tools": tools, "tool_choice": "auto", "max_tokens": 250,
+            "provider": {"order": [prov], "allow_fallbacks": False}
+        }).encode(),
+        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
+    )
+    t0 = time.time()
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            b = json.loads(r.read().decode())
+            tc = b["choices"][0]["message"].get("tool_calls")
+            args = tc[0]["function"]["arguments"] if tc else "NONE"
+            print(f"PASS | {prov:15s} | {time.time()-t0:4.2f}s | actual={b.get(\"provider\")} | args={args}")
+    except urllib.error.HTTPError as e:
+        print(f"FAIL | {prov:15s} | HTTP {e.code} | {e.read().decode()[:140]}")
+    except Exception as e:
+        print(f"FAIL | {prov:15s} | ERR {e}")
+'
+```
+
+---
+
+## Consequences
+
+### Positive
+* **Unified Latest-Generation Stack**: Completely eliminates legacy Gemini 2.5 models; standardizes exclusively on DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, and Sonar Pro.
+* **100% Reliable Image Generation**: Correctly routes image generation requests to Nano Banana 2 (`google/gemini-3.1-flash-image`), completely eliminating HTTP 404 generation errors.
+* **Cohesive Voice & Audio Pipeline**: Clear architectural documentation of the Whisper V3 Turbo and Gemini 3.1 Flash TTS integration for dossiers and podcasts.
+* **Guaranteed Cost Ceiling**: Drops weekly expenditure from **\$12.55 to ~\$0.80–\$1.40** (~90% savings) through DeepSeek V4.1 Flash 1017's `$0.00294/M` cache reads, decoupled heartbeats, and nightly compaction.
+
+### Negative & Mitigations
+* **OpenRouter Provider Rollout Lag on Day-0 Model Releases**: Newly released snapshots initially launch with 5–10 providers before the full fleet (25+ providers) joins over subsequent days.
+  * *Mitigation*: Codified the 7-point provider qualification gate and one-shot test script above, and configured automatic model fallback to the `0910` snapshot (`openrouter/deepseek/deepseek-v4.1-flash`).

docs/adr/0003-everything-as-code-and-reproducible-infrastructure.md

@@ -0,0 +1,153 @@
+---
+status: "accepted"
+date: 2026-09-17
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0003: Everything as Code (EaC) and Reproducible Declarative Infrastructure
+
+## Context and Problem Statement
+
+The **Infrapuzzle** monorepo governs the complete digital infrastructure for Moritz Graf and family: bare-metal server provisioning (Haumdaucher node), Kubernetes workloads, DNS management, mail routing, smart home/IoT edge devices, and AI concierge workflows (OpenClaw and specialized domain subagents).
+
+Historically, home labs and personal infrastructure suffer from the "snowflake" problem:
+1. **Undocumented ClickOps**: Settings altered manually in cloud provider consoles (e.g. OVH DNS, router web UIs) or via web dashboards (Home Assistant, Grafana) become unversioned, undocumented, and fragile.
+2. **Ephemeral Terminal Interventions**: Ad-hoc `kubectl edit`, hot-patched container environments, or manual secret injections deviate from repository state, rendering disaster recovery nearly impossible.
+3. **Agent Prompt Fragmentation**: As AI agents proliferate (e.g. OpenClaw conductor, domain subagents like Hebamme, Koch, Grabengasse, Fitness, Recherche), defining prompts or tool configs across disparate Helm values, ConfigMaps, and local files leads to inconsistencies, stale behavior, and duplicate definitions.
+
+How do we establish a strict, maintainable architectural standard that ensures **everything is defined as code as far as feasible and reproducible**, while accommodating dynamic runtime state and secrets safely?
+
+---
+
+## Decision Drivers
+
+* **Determinism & Disaster Recovery**: The entire single-node cluster (`haumdaucher.de`), along with all services, DNS records, and edge devices, must be rebuildable from scratch purely from git repositories.
+* **Single Source of Truth**: Eliminate configuration drift across all operational domains.
+* **Exclusive Agent Storage**: Establish an unambiguous, single home for all AI agent personas, system prompts, role boundaries, and tool guidance.
+* **Declarative Secret Management**: Store all secrets in version control without risking plain-text credential leaks.
+* **Pragmatic Feasibility Boundary**: Clearly separate declarative desired state (in Git) from legitimate mutable runtime data (databases, dynamic chat histories, live OAuth session tokens) without breaking stateful applications.
+
+---
+
+## Considered Options
+
+1. **Option 1: Ad-Hoc / Best-Effort Setup (Status Quo Antipattern)**
+   * Store major K8s manifests in git, but allow manual tweaks in web consoles (OVH, Home Assistant, Shelly) and ad-hoc `kubectl` patches.
+   * *Drawbacks*: Severe drift, snowflake server syndrome, impossible disaster recovery, zero audit trail.
+
+2. **Option 2: Pure GitOps Immutability with Zero Runtime Mutation**
+   * Force every state mutation (including session compacting, OAuth token refreshes, and dynamic settings) to generate a Git commit.
+   * *Drawbacks*: Impractical for stateful workloads. High friction, commit noise, rate limits on git providers, and potential race conditions when multi-agent systems rapidly update local session caches.
+
+3. **Option 3: Comprehensive Everything-as-Code with Pragmatic State Governance (Selected)**
+   * Define 100% of infrastructure, workloads, DNS, IoT edge devices, and AI agents declaratively in Git.
+   * Enforce `git-crypt` for declarative secrets.
+   * Isolate mutable runtime state (databases, chat histories, active OAuth tokens) to persistent volumes (PVCs) governed by zero-drift merge mechanics (`mergeMode: merge`).
+   * Mandate `knowledge-wiki/agents/` as the **exclusive** home for all AI agent prompt definitions.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 3: Comprehensive Everything-as-Code with Pragmatic State Governance**.
+
+### 1. The Five Tiers of Everything as Code (EaC)
+
+```
+┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                    Infrapuzzle Monorepo (Git)                                    │
+└────────────────────────────────────────────────┬─────────────────────────────────────────────────┘
+                                                 │
+      ┌──────────────────┬───────────────────────┼───────────────────────┬──────────────────┐
+      ▼                  ▼                       ▼                       ▼                  ▼
+┌───────────┐      ┌───────────┐           ┌───────────┐           ┌───────────┐      ┌───────────┐
+│  Tier 1   │      │  Tier 2   │           │  Tier 3   │           │  Tier 4   │      │  Tier 5   │
+│Bare-Metal │      │Kubernetes │           │Cloud & DNS│           │Smart Home │      │Agents as  │
+│Bootstrap  │      │Workloads  │           │           │           │& IoT Edge │      │Code (Wiki)│
+├───────────┤      ├───────────┤           ├───────────┤           ├───────────┤      ├───────────┤
+│• Kubespray│      │• Helm     │           │• Terraform│           │• ESPHome  │      │• Exclusive│
+│• Ansible  │      │• Manifests│           │  (OVH DNS)│           │• Shelly   │      │  prompts  │
+│• init.sh  │      │• Operators│           │• Mail DNS │           │  Terraform│      │• Tool     │
+│           │      │  (OpenClaw│           │  (MX/SPF/ │           │• BLE &    │      │  bindings │
+│           │      │   CRDs)   │           │   DKIM)   │           │  Sensors  │      │• git-sync │
+└─────┬─────┘      └─────┬─────┘           └─────┬─────┘           └─────┬─────┘      └─────┬─────┘
+      │                  │                       │                       │                  │
+      ▼                  ▼                       ▼                       ▼                  ▼
+  Hetzner Node      Haumdaucher K8s         Public DNS &            Grabengasse 7        OpenClaw
+  Base OS Setup     Cluster Engine          Domain Records          Physical Devices     Conductor
+```
+
+#### Tier 1: Bare-Metal Server Provisioning (`bootstrap/`)
+* **Tooling**: Kubespray, Ansible, Bash (`init.sh`).
+* **Scope**: Operating system hardening, kernel modules, firewall rules, container runtime (containerd), and Kubernetes control plane setup on the single bare-metal node (Haumdaucher, `136.243.23.215`).
+* **Source of Truth**: `bootstrap/init.sh` and version-pinned Kubespray playbooks.
+
+#### Tier 2: Kubernetes Workloads & Operators (`k8s/`)
+* **Tooling**: Raw Kubernetes manifests, Helm 3 values, and Kubernetes Custom Resource Definitions (CRDs).
+* **Conventions**:
+  - Directory structure enforces namespace partitioning (`k8s/<namespace>/`).
+  - Workloads managed via declarative specifications (e.g. OpenClaw Operator `OpenClawInstance` CRD in `k8s/openclaw/openclaw-instance.secret.yaml`).
+  - Standardized ingress controller (`ingress-nginx` / `traefik`) with automated Let's Encrypt certificates (`cert-manager`).
+
+#### Tier 3: External Infrastructure, DNS & Mail (`terraform/`)
+* **Tooling**: Terraform with `ovh/ovh` provider.
+* **Scope**:
+  - Authoritative DNS zones (`haumdaucher.de`, `moritzgraf.de`, `ameliegraf.de`, and others).
+  - Google Workspace mail routing (MX, SPF TXT, DKIM, DMARC records).
+  - Cluster IP mapping to the bare-metal ingress endpoint.
+* **Execution**: Fully automated via `terraform plan` / `terraform apply` directly from version-controlled state.
+
+#### Tier 4: Smart Home & IoT Edge Hardware (`projects/shelly_taupunktluefter/`, `esphome/`, `monitoring/`)
+* **Tooling**: ESPHome, Shelly Terraform Provider, YAML configs.
+* **Scope**:
+  - Shelly relays and smart switches provisioned declaratively via Terraform.
+  - ESPHome firmware configurations for ESP32 Bluetooth proxies, Xiaomi BLE temperature/hygrometer sensors, and plant moisture monitors.
+  - All device configurations, pinouts, and network parameters committed to git; flashing performed deterministically from code.
+
+#### Tier 5: Agents-as-Code & Knowledge Systems (`knowledge-wiki/agents/`)
+* **Core Rule — Exclusive Single Source of Truth**:
+  > **All AI agent definitions (prompts, personas, operational boundaries, and tool calling instructions) MUST reside exclusively in [`knowledge-wiki/agents/<name>.md`](../../knowledge-wiki/agents/).**
+* **Strict Architecture**:
+  - Agent behavior is never hardcoded inside container images, ad-hoc Helm values, or untracked local directories.
+  - Adding or modifying an agent is done purely by creating or editing a markdown specification in the `knowledge-wiki/agents/` directory (e.g. `hebamme.md`, `koch.md`, `grabengasse.md`, `fitness.md`, `recherche.md`).
+  - Workloads (such as OpenClaw) ingest these definitions dynamically via `git-sync` sidecars or submodules, enabling instantaneous updates without requiring pod restarts or image rebuilds.
+  - If agent prompts or specifications are discovered outside of `knowledge-wiki/agents/`, they must be refactored and consolidated into this directory.
+
+---
+
+## The Feasibility Boundary & State Governance
+
+The user requirement explicitly dictates: *"we want to do everything as code as far as it is feasible and reproducible"*. To balance pure declarative code with stateful services, the boundary is defined as follows:
+
+| Dimension | Managed as Code (Git) | Managed on Persistent Storage (PVC) |
+| :--- | :--- | :--- |
+| **Workload Topology** | Pod specs, replicas, resource limits, CRDs, Helm releases. | — |
+| **Secrets & Keys** | Encrypted via `git-crypt` (`*.secret`, `*.secret.yaml`, `*.secret.values`). | Mounted into runtime memory/containers. |
+| **Agent Behavior** | System prompts, personas, roles, tool manifests in `knowledge-wiki/agents/`. | Dynamic short-term conversation context, user memory (`MEMORY.md`). |
+| **External Auth Tokens** | Initial static API keys and client credentials. | Volatile session tokens refreshed dynamically at runtime (e.g. Garmin OAuth tokens on PVC). |
+| **Data Storage** | Schema migrations, DB connection configs, backup cronjobs. | Raw databases (PostgreSQL, SQLite), media files, Nextcloud files. |
+| **Operator Integration** | Declarative base configurations (`spec.config.raw`). | Runtime mutations deep-merged safely via `mergeMode: merge` without overwriting runtime state. |
+
+### Operational Governance Rules
+
+1. **No Ad-Hoc `kubectl edit`**: Never modify cluster objects directly with `kubectl edit` or unrecorded commands. All modifications must be committed to git and applied via `kubectl apply -f <manifest>` or Helm.
+2. **Git-Crypt Security Enforcement**: Any file containing API keys, private passwords, or private identities must match the `.gitattributes` pattern (`*.secret*`) and be locked with `git-crypt`.
+3. **Reproducibility Guarantee**: In the event of total hardware loss, cloning `infrapuzzle` and unlocking `git-crypt` must be sufficient to run `bootstrap/init.sh`, apply Terraform, restore PVC backups from Velero/MinIO, and restore full operational state.
+
+---
+
+## Consequences
+
+### Positive
+* **100% Deterministic Disaster Recovery**: Eliminates mysterious configuration drift and allows full rebuilds of bare-metal, Kubernetes, DNS, and IoT systems.
+* **Unified AI Agent Lifecycle**: Developers and AI assistants edit agent personalities, prompts, and skills in a single, version-controlled markdown directory (`knowledge-wiki/agents/`).
+* **High Operational Velocity**: Changes to agent logic apply instantly via `git-sync` without container restarts or downtime.
+* **Audit Trail**: Every change to infrastructure, DNS, smart home switches, or agent system prompts has an author, timestamp, and commit message.
+
+### Negative & Mitigations
+* **Discipline Overhead**: Even minor configuration changes require a git commit and deployment workflow rather than a quick UI click.
+  * *Mitigation*: Tooling like OpenClaw Operator automated rolling updates and `git-sync` minimizes deployment friction to a single `git push`.
+* **Git-Crypt Dependency**: Operators and agents require the repository encryption key to modify secrets.
+  * *Mitigation*: Standardized symmetric key distribution and clear documentation in `k8s/AGENTS.md`.

docs/adr/0004-structured-personal-data-datastore.md

@@ -0,0 +1,141 @@
+---
+status: "accepted"
+date: 2026-10-06
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)
+
+## Context and Problem Statement
+
+OpenClaw manages personal logging and family assistance for the Graf family across several daily domains:
+1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`.
+2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`.
+3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect.
+4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings.
+5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg.
+
+Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes:
+- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals.
+- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning).
+- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text.
+
+How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture?
+
+---
+
+## Decision Drivers
+
+* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine.
+* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly.
+* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM.
+* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza).
+* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`.
+* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead.
+
+---
+
+## Considered Options
+
+1. **Option 1: Stricter Markdown Prompts + Python Lint Rules**
+   * Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums.
+   * *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries.
+
+2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)**
+   * Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC.
+   * *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git.
+
+3. **Option 3: Pure Event Sourcing for All Data**
+   * Force every piece of data (including stores, recipes, and food labels) into an immutable event log.
+   * *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams.
+
+4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)**
+   * Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine.
+   * Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**.
+
+### 1. The Three Data Archetypes
+
+```
+┌────────────────────────────────────────────────────────────────────────┐
+│                        Three Data Archetypes                           │
+├─────────────────────────┬─────────────────────────┬────────────────────┤
+│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs)  │ 3. DOCUMENTS (Prose│
+├─────────────────────────┼─────────────────────────┼────────────────────┤
+│ • data/events/nutrition/│ • data/entities/foods/  │ • entities/recipes/│
+│ • data/events/sleep/    │ • data/entities/recipes/│ • syntheses/places/│
+│ • data/events/feeding/  │ • data/entities/places/ │ • sources/*.pdf    │
+│ • data/events/body/     │ • data/entities/people/ │                    │
+│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│
+└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘
+             │                         │                       │
+             ▼                         ▼                       │
+    ┌──────────────────────────────────────────────┐           │
+    │         In-Memory DuckDB SQL Engine          │           │
+    │  • Schema Validation (STRUCT & CHECK)        │           │
+    │  • SQL Aggregations & Daily Deficit Joins    │           │
+    │  • Spatial Queries (ST_DWithin / Haversine)  │           │
+    └──────────────────────┬───────────────────────┘           │
+                           │                                   │
+              ┌────────────┴────────────┐                      │
+              ▼                         ▼                      │
+     ┌──────────────────┐      ┌──────────────────┐            │
+     │ Telegram Output  │      │ Generated Views  │            │
+     │ • Text Summaries │      │ • Markdown Views ◄────────────┘
+     │ • chart-to-chat  │      │ • HTML Dashboards│ (Frontmatter:
+     └──────────────────┘      │   (/family/...)  │  datastore_ref)
+                               └──────────────────┘
+```
+
+1. **Events (Time-Series Logs):**
+   - Stored in `data/events/<domain>/<YYYY-MM>.jsonl`, sorted by `(start, id)`.
+   - Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`.
+2. **Entities (Structured Catalogs):**
+   - Stored in `data/entities/<collection>.jsonl`, sorted by slug `id`.
+   - `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density.
+   - `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros.
+   - `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents.
+   - `people`: Family profile registry.
+3. **Documents (Prose & Media):**
+   - Stored as standard Markdown in `knowledge-wiki/`.
+   - Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: "<collection>:<id>"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`.
+
+### 2. Dedicated Raw Garmin Mirror (`data/garmin/`)
+- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into:
+  - `data/garmin/activities/<YYYY-MM>.jsonl`
+  - `data/garmin/daily/<YYYY-MM>.jsonl`
+  - `data/garmin/sleep/<YYYY-MM>.jsonl`
+- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure.
+
+### 3. Spatial & Geospatial Radius Lookups
+- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`).
+- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment.
+
+### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`)
+- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers.
+- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts).
+- **Private (`/private/*`):** Protected by Authentik for personal vital logs.
+
+---
+
+## Consequences
+
+### Positive
+- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors.
+- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs.
+- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds.
+- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database.
+
+### Negative & Mitigations
+- **Generated Markdown Views Must Not Be Hand-Edited:**
+  - *Mitigation*: Generated files include a prominent `<!-- DO NOT EDIT: GENERATED BY DATASTORE -->` warning comment.
+  - *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data.
+- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed.
+  - *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt.

docs/adr/0005-personal-data-isolation-and-authentik-boundary.md

@@ -0,0 +1,156 @@
+---
+status: "accepted"
+date: 2026-10-11
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0005: Personal Data Isolation, Authentik Authentication Boundaries, and Hosting Segregation
+
+## Context and Problem Statement
+
+On 2026-10-11, during the development of mobile-first PWA dashboards for baby monitoring (Mila Graf) and personal nutrition/Garmin energy balance (Moritz Graf), personal health, sleep, diaper, and nutrition data was temporarily made available at unauthenticated root URLs:
+- `https://hosting.haumdaucher.de/mila.html`
+- `https://hosting.haumdaucher.de/moritz.html`
+
+While [ADR 0004](0004-structured-personal-data-datastore.md) established the conceptual routing design (`/family/*` protected by Traefik ForwardAuth, public recipes at `/recipes/*`), a gap analysis revealed critical systemic vulnerabilities:
+1. **Unapplied Kubernetes Ingress:** The ForwardAuth Ingress (`minio-hosting-family`) declared in `k8s/minio/hosting-ingress.yaml` was never applied to the Kubernetes cluster.
+2. **Missing Authentik Identity Provider:** Authentik had no Proxy Provider or Application configured for `hosting.haumdaucher.de`, returning HTTP 404 on ForwardAuth checks.
+3. **Dual-Publishing Anti-Pattern:** Dashboard generator scripts (`update_website.py`, `update_moritz_website.py`) mirrored files to root (`/mila.html`, `/moritz.html`) for convenience, completely bypassing the planned `/family/` path boundary.
+4. **Lack of Automated Ingress & Storage Auditing:** No automated test or CI/Cron audit prevented personal data from being uploaded to public prefixes.
+
+How do we establish a permanent, verifiable, and automated security boundary that guarantees personal and family data can never be exposed publicly without authentication?
+
+---
+
+## Decision Drivers
+
+* **Zero Public Exposure for Personal Data:** Health, sleep, nutrition, body, and family data must NEVER be accessible without authentication.
+* **Defense-in-Depth:** Security must not rely on a single layer (e.g. secret URLs or developer memory). Protection must be enforced at identity provider, reverse proxy, storage bucket, and generator script levels.
+* **Declarative Infrastructure as Code (ADR 0003):** All Authentik providers, policies, groups, and Kubernetes Ingresses must be versioned in Git and applied via Terraform/Kubectl.
+* **Automated Guardrails & Fast Feedback:** Continuous audits must detect and quarantine any unauthorized files outside protected paths.
+* **Seamless PWA Experience:** The PWA capabilities (manifest, service worker, offline shell) must function within the authenticated cookie session without breaking ForwardAuth.
+
+---
+
+## Decision Outcome
+
+Chosen Strategy: **Multi-Layer Boundary Enforcement with ForwardAuth, Strict Storage Segregation, and Continuous Auditing**.
+
+### 1. Storage & URL Path Segmentation Architecture
+
+```
+Internet Request to https://hosting.haumdaucher.de
+                    │
+                    ▼
+          ┌───────────────────┐
+          │ Traefik Ingress   │
+          └─────────┬─────────┘
+                    │
+         Path Inspection & Priority
+                    │
+   ┌────────────────┴────────────────┐
+   │                                 │
+Path: /family/*                   Path: /* (Fallback)
+Priority: 100                     Priority: 10
+   │                                 │
+   ▼                                 ▼
+┌─────────────────────────┐       ┌─────────────────────────┐
+│ Authentik ForwardAuth   │       │ Public Ingress          │
+│ • Host: hosting.haumd...│       │ • ONLY /recipes/*       │
+│ • Allowed: 'family' &   │       │ • ONLY /showcase/*      │
+│   'infra-admins' groups │       │ • NO personal files     │
+└────────────┬────────────┘       └────────────┬────────────┘
+             │                                 │
+     HTTP 200 (Authed)                         │
+             │                                 │
+             ▼                                 ▼
+   ┌──────────────────┐               ┌──────────────────┐
+   │ MinIO Bucket     │               │ MinIO Bucket     │
+   │ key: family/*    │               │ key: public/*    │
+   │ (Protected Data) │               │ (Public Static)  │
+   └──────────────────┘               └──────────────────┘
+```
+
+#### Rule 1: Canonical Protected Prefix `/family/`
+- All dashboards containing personal, familial, or physiological metrics MUST reside under `/family/`:
+  - `https://hosting.haumdaucher.de/family/mila.html` (Baby sleep, feeds, diapers)
+  - `https://hosting.haumdaucher.de/family/moritz.html` (Personal nutrition, Garmin deficits, health)
+  - `https://hosting.haumdaucher.de/family/nutrition.html` (Family nutrition overview)
+  - `https://hosting.haumdaucher.de/family/sleep.html` (Sleep analytics)
+- Associated PWA assets (manifests, service workers) MUST also be scoped within `/family/` (e.g. `/family/mila.webmanifest`, `/family/sw.js`).
+
+#### Rule 2: Root Prefix `/` Strictly Restricted
+- The root of `hosting.haumdaucher.de` is restricted to public static assets (`/recipes/*`, `/showcase/*`).
+- Publishing scripts MUST NEVER write personal dashboards to the bucket root.
+
+---
+
+### 2. Declarative Authentik ForwardAuth Configuration
+
+Authentik configuration is codified in `terraform/authentik.tf`:
+1. **Proxy Provider (`authentik_provider_proxy.hosting_family`):**
+   - External Host: `https://hosting.haumdaucher.de`
+   - Mode: `forward_single`
+   - Bound to `authentik_outpost.embedded`
+   - Session duration: 30 days (`access_token_validity = "days=30"`)
+2. **Application (`authentik_application.hosting_family`):**
+   - Slug: `hosting-family`
+   - Protocol Provider: `authentik_provider_proxy.hosting_family.id`
+3. **Role-Based Policy Bindings:**
+   - Group `family` (Moritz & Amelie Graf): Granted access
+   - Group `infra-admins`: Granted access
+   - Unauthenticated requests are redirected (HTTP 302) to `https://auth.haumdaucher.de/application/o/authorize/...`
+
+---
+
+### 3. Kubernetes Ingress Deployment
+
+`k8s/minio/hosting-ingress.yaml` defines the two ingress tiers:
+- **`minio-hosting-family` (Priority 100):**
+  - Paths:
+    - `/family` (Prefix) -> Protected personal dashboards
+    - `/outpost.goauthentik.io` (Prefix) -> Authentik ForwardAuth OAuth callback & session negotiation
+  - Middlewares: `auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd`
+- **`minio-hosting` (Priority 10):**
+  - Path: `/` (Prefix)
+  - Middlewares: `minio-hosting-add-prefix@kubernetescrd`
+
+---
+
+### 4. Generator Scripts & Codebase Hardening
+
+1. **Elimination of Root Duplication:**
+   - `skills/datastore/scripts/update_website.py`: Removed all S3 upload and local sync calls to root `mila.html`. Target is strictly `family/mila.html`.
+   - `skills/datastore/scripts/update_moritz_website.py`: Removed S3 upload and local sync calls to root `moritz.html`. Target is strictly `family/moritz.html` (and `family/nutrition.html`).
+2. **PWA Scope Alignment:**
+   - Web App Manifests configure `start_url: "/family/mila.html"` and `scope: "/family/"`.
+   - Service workers are registered at `/family/sw.js`.
+3. **Telegram Link Consistency:**
+   - Agent prompts for `mila` (Topic 20) and `koch` (Topic 21) only broadcast the secure URL: `https://hosting.haumdaucher.de/family/...`.
+
+---
+
+### 5. Automated Continuous Leak Audit Guard
+
+An automated audit check runs continuously (via OpenClaw periodic audit and datastore maintenance):
+- Inspects the MinIO `hosting` bucket root (`prefix=""`).
+- Detects any forbidden files matching patterns: `*mila*`, `*moritz*`, `*nutrition*`, `*sleep*`, `*schlaf*`, `*health*`, `*diaper*`.
+- If detected:
+  1. Immediately deletes/quarantines the offending object from the public bucket root.
+  2. Raises a critical notification/event to the system log.
+
+---
+
+## Consequences
+
+### Positive
+- **Guaranteed Confidentiality:** Family and personal health data is physically impossible to access without passing Authentik ForwardAuth.
+- **Reproducible Infrastructure:** The entire authentication chain is codified in Terraform and Kubernetes manifests.
+- **Persistent Mobile PWA:** Authentik's 30-day session lifetime ensures seamless standalone mobile app operation without repetitive login prompts.
+- **Automated Safety Net:** Even if a developer or agent script mistakenly attempts to mirror a file to root, the automated audit deletes it.
+
+### Negative & Mitigations
+- **Authentication Step on First Mobile Launch:**
+  - *Mitigation*: When installing the PWA on iOS/Android, the user authenticates once via Authentik. The persistent session cookie maintains login across app launches.

docs/adr/README.md

@@ -0,0 +1,43 @@
+# Architecture Decision Records (ADRs)
+
+This directory documents the key architectural decisions made within the **Infrapuzzle** monorepo and its associated workloads (Kubernetes, OpenClaw AI agent, networking, DNS, and smart home systems).
+
+We follow the [Markdown Architectural Decision Records (MADR)](https://adr.github.io/madr/) format to capture intentional technical decisions, their context, considered options, trade-offs, and consequences.
+
+---
+
+## 📋 Record Index
+
+| ADR | Title | Status | Date | Area |
+| :--- | :--- | :---: | :---: | :--- |
+| [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP |
+| [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models |
+| [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC |
+| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage |
+| [**0005**](0005-personal-data-isolation-and-authentik-boundary.md) | **Personal Data Isolation, Authentik Authentication Boundaries, and Hosting Segregation** | `accepted` | 2026-10-11 | Security / Storage / Authentik |
+
+---
+
+## ✍️ How to Create a New ADR
+
+1. Copy the template structure into a new numbered markdown file:
+   ```bash
+   docs/adr/000X-short-title-in-kebab-case.md
+   ```
+2. Include the YAML frontmatter:
+   ```yaml
+   ---
+   status: "proposed | accepted | superseded | deprecated"
+   date: YYYY-MM-DD
+   decision-makers: ["Moritz Graf", "Antigravity"]
+   consulted: []
+   informed: []
+   ---
+   ```
+3. Clearly define:
+   - **Context & Problem Statement** (Why was this decision needed?)
+   - **Decision Drivers** (Constraints, costs, token limits, latency, maintainability)
+   - **Considered Options** (Alternatives evaluated)
+   - **Decision Outcome** (Chosen path with justification)
+   - **Consequences** (Positive effects, negative trade-offs, follow-up actions)
+4. Add the new record to the table in this `README.md`.

k8s/AGENTS.md

@@ -11,8 +11,9 @@ This directory contains the Kubernetes manifests and Helm charts for a single-no
 *   **SSL**: `cert-manager` (LetsEncrypt)
 
 ## Directory Structure
-*   **Top-level folders**: Each folder corresponds to a Kubernetes **namespace**.
-    *   Example: `mailu/` contains resources for the `mailu` namespace.
+*   **Strict Namespace Mapping**: **Every top-level subdirectory in `k8s/` MUST correspond strictly to a Kubernetes namespace.**
+    *   Example: `mailu/` contains resources for the `mailu` namespace, `grabengasse/` for `grabengasse`, `haumdaucher/` for `haumdaucher`, etc.
+    *   **NO non-namespace folders**: Loose utility scripts belong in the root `scripts/` directory, application source code belongs in `projects/`, and general documentation belongs in namespace READMEs or `docs/`.
 *   **Documentation**: `README.md` is the **authoritative source** for deployment commands. Always check it before running commands.
 
 ## Code Style & Conventions
@@ -89,16 +90,16 @@ ssh -t moritz@haumdaucher.de "sudo df -h"
     ```
 
 ## Ingress Configuration
-Ingress resources **must** follow these strict conventions to work with the cluster's ingress controller (`nginx`) and certificate manager (`cert-manager`).
+Ingress resources **must** follow these strict conventions to work with the cluster's ingress controller (`traefik`) and certificate manager (`cert-manager`).
 
 ### Annotations
 All Ingress resources must include:
 ```yaml
 annotations:
-  kubernetes.io/ingress.class: "nginx"
+  kubernetes.io/ingress.class: "traefik"
   cert-manager.io/cluster-issuer: "letsencrypt-prod"
   kubernetes.io/tls-acme: "true"
-  # Standard nginx tweaks
+  # Standard nginx tweaks (if using dual class) or traefik configurations
   nginx.ingress.kubernetes.io/proxy-body-size: "0"
   nginx.ingress.kubernetes.io/ssl-redirect: "true"
   nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
@@ -107,13 +108,13 @@ annotations:
 ### Hostnames & TLS
 *   **Domain**: Use a subdomain of `haumdaucher.de` or `moritzgraf.de`.
 *   **TLS Secret Name**: Must use **hyphens** instead of dots.
-    *   Pattern: `<subdomain>-<domain>-<tld>`
-    *   Example: `n8n.moritzgraf.de` -> `n8n-moritzgraf-de`
+*   **Pattern**: `<subdomain>-<domain>-<tld>`
+*   **Example**: `n8n.moritzgraf.de` -> `n8n-moritzgraf-de`
 
 ### Example
 ```yaml
 spec:
-  ingressClassName: nginx
+  ingressClassName: traefik
   tls:
     - hosts:
         - n8n.moritzgraf.de
@@ -160,3 +161,53 @@ spec:
 *   **Namespaces**: Every application gets its own namespace.
 *   **Secrets**: Encrypt all secrets using `git-crypt`.
 
+## Network Setup (VPN & Routing)
+
+The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.
+
+```mermaid
+graph TD
+    subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
+        subgraph "monitoring namespace"
+            Exporter[fritzbox-exporter Pod]
+        end
+        HostNet[Host Network Namespace]
+        WgPod[wireguard Pod hostNetwork: true]
+    end
+
+    subgraph "Home Network (192.168.10.0/24)"
+        FB[FRITZ!Box Gateway - 192.168.10.1]
+        Taupi[Taupi Fan Shelly - 192.168.10.168]
+    end
+
+    Exporter -- "Queries 192.168.10.1" --> HostNet
+    HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
+    WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
+    FB -- "Accesses local subnet" --> Taupi
+```
+
+### Components & Routing
+1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`.
+2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`).
+3. **WireGuard VPN Link (`wg0`)**:
+   - The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace.
+   - The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`.
+   - The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`.
+   - The host routing table directs home network traffic over the tunnel:
+     ```bash
+     192.168.10.0/24 dev wg0 scope link
+     ```
+   - Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box.
+
+### FritzBox Exporter Egress Reject Workaround (Option B)
+
+**The Problem**:
+When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`). 
+When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts.
+
+**The Solution**:
+Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.
+- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges.
+- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace.
+- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately.
+- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.

k8s/README.md

@@ -34,14 +34,33 @@ kubcetl edit -n mailu secret sh.helm.release.v1.mailu.v8
 
 # Deployment (non persistent stuff)
 
-## [ingress-nginx](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx)
+## [Traefik Ingress Controller](https://github.com/traefik/traefik-helm-chart) (Replaces retired ingress-nginx)
 
-Apply with helm:
+Apply Dual IngressClass configuration (supporting both legacy `nginx` and new `traefik` names on the same controller):
 
 ```bash
-helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+kubectl apply -f traefik/ingress-classes.yaml
+```
+
+Apply with Helm:
+
+```bash
+helm repo add traefik https://traefik.github.io/charts
 helm repo update
-helm upgrade --install --create-namespace ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx -f ingress-nginx/ingress-nginx.yaml
+helm upgrade --install traefik traefik/traefik -n kube-system -f traefik/traefik-values.yaml
+```
+
+### [Legacy ingress-nginx (RETIRED)](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx)
+
+The community ingress-nginx is **retired** (EOL March 2026). During hot-swap, it was scaled down to 0 replicas. Keep it scaled down for a few days before deleting:
+
+```bash
+# To scale down (executed during migration)
+kubectl scale daemonset ingress-nginx-controller -n ingress-nginx --replicas=0
+
+# Permanent cleanup (execute after 3-7 days safety buffer)
+# helm uninstall ingress-nginx -n ingress-nginx
+# kubectl delete ns ingress-nginx
 ```
 
 ## [cert-manager](https://cert-manager.io/docs/tutorials/acme/ingress/)
@@ -102,19 +121,64 @@ Backup tool. See chart [README](https://github.com/vmware-tanzu/helm-charts/blob
 ```sh
 helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
 helm repo update
-helm upgrade --install --create-namespace --namespace velero -f ./velero/velero.secret.yaml velero vmware-tanzu/velero --version 5.3.0
-kubectl create secret generic rclone-config --from-file=./velero/rclone.secret
-kubectl apply -f velero/dropbox_sync.yml
-# #
+# Apply CRDs first (required on upgrades across major versions)
+helm show crds vmware-tanzu/velero --version 12.1.0 | kubectl apply -f -
+helm upgrade --install --create-namespace --namespace velero -f ./velero/velero.secret.yaml velero vmware-tanzu/velero --version 12.1.0
+# Secrets for rclone (minio + gdrive)
+kubectl create secret generic rclone-config --from-file=./velero/rclone.secret -n velero
+kubectl create secret generic gdrive-sa-key --from-file=gdrive_sa.json.secret=./velero/gdrive_sa.json.secret -n velero
+# Deploy GDrive sync cronjob
+kubectl apply -f velero/gdrive_sync.yml
+# Teardown
 helm delete velero -n velero
 kubectl delete ns velero
 ```
 
-A manual backup may be created executing the following command. **Note: Keep backuped namespaces in sync with config from helm chart!!!**
+A manual backup may be created executing the following command. **Note: Keep backed-up namespaces in sync with config from helm chart!!!**
 
 ```sh
 DATE=$( date +%Y%m%d )
-velero backup create $DATE --include-namespaces datalab,development,nextcloud,tt-rss,mailu --wait
+velero backup create $DATE --include-namespaces coder,home-assistant,n8n,influxdb,development,nextcloud,openclaw,auth,zigbee --wait
+```
+
+### Backup Retention Policy
+
+Three schedules run automatically, providing a tiered retention policy. Kopia (used for fs-backup) deduplicates at the block level, so each backup only stores changed data — incrementals come for free.
+
+| Schedule | Cron | TTL | Keeps |
+|---|---|---|---|
+| `velero-daily-backup` | `0 2 * * *` | 8 days | Last 7 daily backups |
+| `velero-weekly-backup` | `0 3 * * 0` | 43 days | Last 6 weekly backups (Sundays) |
+| `velero-monthly-backup` | `0 4 1 * *` | 183 days | Last 6 monthly backups (1st of month) |
+
+```sh
+# Check all schedules and their last run
+velero schedule get
+
+# Check current backups and their expiry
+velero backup get
+```
+
+
+## authentik (auth)
+
+Authentik is the central SSO and identity provider deployed at [auth.haumdaucher.de](https://auth.haumdaucher.de).
+See full documentation in [k8s/auth/README.md](./auth/README.md) and [k8s/auth/MACHINE_AUTH.md](./auth/MACHINE_AUTH.md).
+
+```sh
+# 1. Create namespace & CNPG PostgreSQL cluster
+kubectl create namespace auth
+kubectl apply -f auth/authentik-db.yaml
+
+# 2. Deploy Authentik via Helm
+helm repo add authentik https://charts.goauthentik.io
+helm repo update
+helm upgrade --install authentik authentik/authentik \
+  --namespace auth \
+  -f auth/authentik.secret.values.yaml
+
+# 3. Deploy Traefik ForwardAuth Middleware
+kubectl apply -f auth/traefik-forwardauth-middleware.yaml
 ```
 
 ## coder
@@ -322,12 +386,21 @@ helm repo add prometheus-community https://prometheus-community.github.io/helm-c
 helm repo update
 helm upgrade --install --create-namespace prometheus-operator prometheus-community/kube-prometheus-stack -n monitoring -f monitoring/prometheus-operator.secret.yml --version 56.6.1
 # alert configuration
-NAMESPACES_TO_ALERT=( kube-system monitoring cert-manager datalab ingress-nginx mailu minio velero web openebs )
+NAMESPACES_TO_ALERT=( kube-system monitoring cert-manager datalab ingress-nginx mailu minio velero web openebs n8n influxdb home-assistant )
 for i in "${NAMESPACES_TO_ALERT[@]}"; do
   kubectl apply -f monitoring/alertmanagerconfig.secret.yaml -n $i
 done
 ```
 
+### Checking alerts history
+
+A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod:
+
+```bash
+python3 k8s/monitoring/scripts/analyze_alerts.py
+```
+
+
 ### influxdb
 
 Used to store hass data long term.
@@ -774,3 +847,62 @@ An autonomous AI agent platform.
     *   **Encrypt**: Ensure the file is encrypted with `git crypt` before committing!
 3.  **Deploy**: `kubectl apply -f openclaw/openclaw.secret.yaml`
 4.  **Access**: `https://openclaw.haumdaucher.de` (Authentication required)
+
+## Zigbee2MQTT
+
+Zigbee2MQTT manages Zigbee devices via the SLZB-MR1U coordinator. It is deployed via the official Helm chart in the `zigbee` namespace.
+
+```bash
+helm repo add zigbee2mqtt https://charts.zigbee2mqtt.io
+helm repo update
+# Apply the secrets first for the Traefik basic auth middleware
+kubectl apply -f zigbee/zigbee2mqtt.secret.yaml
+# Install or upgrade the release
+helm upgrade --install zigbee2mqtt zigbee2mqtt/zigbee2mqtt --create-namespace -n zigbee -f zigbee/zigbee2mqtt.secret.values.yaml
+```
+
+To modify configurations (like `devices`, `throttle`, or `mqtt`), edit `zigbee/zigbee2mqtt.secret.values.yaml` and re-run the `helm upgrade` command. The chart automatically handles merging these values into the persistent volume on startup.
+
+## Grabengasse 7 Website (`grabengasse7.de`)
+
+Grabengasse 7 home website and backend service deployed in the `grabengasse` namespace. Exposes outdoor weather metrics from the Shelly WS90 station with interactive 24h charts.
+
+### Build and Push Container Images
+
+```bash
+# Login to local registry (if not already logged in)
+docker login registry.haumdaucher.de -u moritz -p "PPf10musSh!!"
+
+# Build and push images via root projects/grabengasse Makefile
+cd projects/grabengasse
+make build push
+
+# Or build individual components:
+cd projects/grabengasse/frontend && make all
+cd ../backend && make all
+```
+
+### Deploy to Kubernetes
+
+```bash
+# Create namespace and registry secret
+kubectl apply -f k8s/grabengasse/namespace.yaml
+kubectl apply -f k8s/grabengasse/registry.secret.yaml
+
+# Deploy backend, frontend, and ingress
+kubectl apply -f k8s/grabengasse/backend.yaml
+kubectl apply -f k8s/grabengasse/frontend.yaml
+kubectl apply -f k8s/grabengasse/ingress.yaml
+```
+
+### Verification
+
+```bash
+# Verify pod statuses
+kubectl get pods -n grabengasse
+
+# Check backend health
+curl -k https://grabengasse7.de/api/v1/healthz
+curl -k https://grabengasse7.de/api/v1/weather/current
+```
+

k8s/auth/MACHINE_AUTH.md

@@ -0,0 +1,65 @@
+# Machine & AI Assistant Authentication Guide
+
+This guide describes how to configure programmatic, machine-to-machine, and AI assistant authentication across services protected by **Authentik** and **Traefik ForwardAuth**.
+
+---
+
+## 🎯 Use Case
+When an AI assistant (or automation script, CLI tool, or cron job) needs to:
+* Inspect Zigbee2MQTT logs or device states via `https://zigbee2mqtt.moritzgraf.de`
+* Scrape or query metrics from `https://prometheus.haumdaucher.de`
+* Trigger or query APIs protected behind the Traefik ForwardAuth gateway
+
+Instead of interactive browser logins or hardcoding static basic auth credentials into individual ingress configs, Authentik supports **Service Accounts & API Tokens**.
+
+---
+
+## ⚙️ Setup Instructions
+
+### 1. Create a Service Account in Authentik
+1. Navigate to **Directory > Users** in Authentik (`https://auth.haumdaucher.de`).
+2. Click **Create Service Account**:
+   * **Username**: `ai-assistant` (or `mopbot`, `cli-agent`)
+   * **Create API Token**: Check this box.
+   * **Token Expiration**: Set to never expire (or desired TTL).
+3. Copy the generated **Token Secret** (e.g. `ak_tok_xxxx...`).
+
+### 2. Grant Access Permissions
+1. Go to **Directory > Groups**.
+2. Add the `ai-assistant` service account to the appropriate group (e.g. `infra-admins` or `zigbee-readers`).
+3. Ensure the Application/Proxy Provider policy allows this group.
+
+---
+
+## 💻 Making Authenticated API Calls
+
+### HTTP Bearer Token (Recommended)
+Pass the API token in the `Authorization` header:
+
+```bash
+curl -H "Authorization: Bearer <your-authentik-token>" \
+     https://zigbee2mqtt.moritzgraf.de/api/bridge/info
+```
+
+When Traefik ForwardAuth queries Authentik with this header, Authentik detects the Bearer token, authenticates the `ai-assistant` service account, checks permissions, and responds with HTTP `200 OK`. Traefik then proxies the request directly to the backend.
+
+### Python / Scripting Example
+
+```python
+import requests
+
+AUTH_TOKEN = "your-authentik-token"
+headers = {
+    "Authorization": f"Bearer {AUTH_TOKEN}",
+    "Accept": "application/json"
+}
+
+response = requests.get("https://zigbee2mqtt.moritzgraf.de/api/bridge/devices", headers=headers)
+print(response.json())
+```
+
+---
+
+## 🔄 Rotating & Revoking Tokens
+* To rotate a token, navigate to **Directory > Tokens and App passwords**.
+* You can view active tokens, inspect last-used timestamps, and revoke compromised tokens instantly with one click without restarting any Kubernetes pods or modifying ingress secrets.

k8s/auth/README.md

@@ -0,0 +1,188 @@
+# Authentik Identity & Access Management (SSO)
+
+Centralized authentication and identity provider for the `haumdaucher` Kubernetes cluster.
+
+* **Domain**: `https://auth.haumdaucher.de`
+* **Helm Chart**: `authentik/authentik` (Version 2026.8.0)
+* **Database**: CloudNative-PG (`cnpg`) PostgreSQL cluster (`authentik-db`)
+* **Ingress**: Traefik with Let's Encrypt TLS certificate (`auth-haumdaucher-de`)
+* **Capabilities**: OIDC/OAuth2 Provider, Traefik ForwardAuth, 2FA (WebAuthn/Passkey, TOTP), User Invitations, Machine/AI Service Accounts.
+
+---
+
+## 🏗️ Architecture Layout
+
+```mermaid
+graph TD
+    Client[Browser / User] -->|HTTPS| Traefik[Traefik Ingress]
+    AI[AI Assistant / CLI Script] -->|HTTPS + Bearer Token| Traefik
+    
+    subgraph "ForwardAuth Ingresses"
+        Traefik -->|ForwardAuth Check| ForwardAuthMW[Traefik ForwardAuth Middleware]
+        ForwardAuthMW -->|Verify Session or Token| AuthServer[Authentik Server]
+        Traefik -->|Allow| Z2M[Zigbee2MQTT]
+        Traefik -->|Allow| Prom[Prometheus & Alertmanager]
+        Traefik -->|Allow| PMA[phpMyAdmin]
+        Traefik -->|Allow| OC[OpenClaw]
+        Traefik -->|Allow| N8N[n8n Web UI]
+    end
+
+    subgraph "Native OIDC Workloads"
+        Grafana[Grafana] -->|OIDC Auth Code Flow| AuthServer
+        Coder[Coder v2] -->|OIDC Auth| AuthServer
+        Gitea[Gitea] -->|OIDC Auth| AuthServer
+        Nextcloud[Nextcloud user_oidc] -->|OIDC Auth| AuthServer
+        MinIO[MinIO Console] -->|OIDC Auth| AuthServer
+    end
+
+    subgraph "auth Namespace"
+        AuthServer --> DB[(CNPG Postgres: authentik-db)]
+        AuthWorker[Authentik Worker] --> DB
+    end
+```
+
+---
+
+## 🚀 Deployment Instructions
+
+### 1. Create Namespace & Provision CNPG Database
+
+```bash
+# Create auth namespace
+kubectl create namespace auth
+
+# Apply CNPG PostgreSQL cluster
+kubectl apply -f k8s/auth/authentik-db.yaml
+
+# Verify PostgreSQL cluster is ready (Status: Cluster in healthy state)
+kubectl get cluster -n auth authentik-db
+```
+
+### 2. Deploy Authentik via Helm
+
+```bash
+# Add Authentik repository
+helm repo add authentik https://charts.goauthentik.io
+helm repo update
+
+# Install or upgrade Authentik release
+helm upgrade --install authentik authentik/authentik \
+  --namespace auth \
+  -f k8s/auth/authentik.secret.values.yaml
+```
+
+### 3. Deploy Traefik ForwardAuth Middleware
+
+```bash
+kubectl apply -f k8s/auth/traefik-forwardauth-middleware.yaml
+```
+
+---
+
+## 🔑 Initial Setup & Admin Onboarding
+
+1. Open your browser and navigate to:
+   ```
+   https://auth.haumdaucher.de/if/flow/initial-setup/
+   ```
+2. Set a secure password for the default `akadmin` account.
+3. Configure your admin user profile and optionally enroll a WebAuthn / Passkey / TOTP 2FA device.
+
+---
+
+## 👥 User Management & Invitations
+
+### Creating Users & Sending Invites
+1. Navigate to **Directory > Users** in the Authentik Admin interface.
+2. You can create users manually or upload/provision users in batches.
+3. To send enrollment links, go to **Directory > Invitations**:
+   * Click **Create**.
+   * Set the target email and assign user groups (e.g. `family`, `friends`, `admins`).
+   * Copy the generated invitation URL or send it to the user.
+
+### Group Permissions
+* `authentik Admins`: Full administrative control over Authentik.
+* `infra-admins`: Access to infrastructure tools (Prometheus, Alertmanager, phpMyAdmin, Zigbee2MQTT).
+* `users`: Standard user group with access to shared web services (Grafana dashboards, Coder, Nextcloud, Gitea).
+
+👉 **For the complete guide on Terraform user management, bulk Excel/CSV imports, custom email templates, and RBAC/PBAC access limiting, see [USER_MANAGEMENT.md](./USER_MANAGEMENT.md).**
+
+---
+
+## 🤖 Machine & AI Agent Access (Programmatic Authentication)
+
+For automated tools, scripts, or AI assistants that need to inspect protected applications (e.g. querying Zigbee2MQTT API/logs or Prometheus metrics):
+
+1. **Create a Service Account**:
+   * In Authentik Admin, navigate to **Directory > Users**.
+   * Click **Create Service Account** (e.g. `ai-agent` or `automation-bot`).
+   * Create a permanent API token for the service account.
+2. **Accessing Protected Endpoints**:
+   * Pass the token in the HTTP `Authorization` header:
+     ```bash
+     curl -H "Authorization: Bearer <your-authentik-token>" https://zigbee2mqtt.moritzgraf.de/api/...
+     ```
+   * Authentik's ForwardAuth middleware validates the Bearer token and allows the request without redirecting to a web login page.
+
+See [MACHINE_AUTH.md](./MACHINE_AUTH.md) for full details.
+
+---
+
+## 🔒 Protecting Applications (Rollout Guide)
+
+### Pattern 1: Traefik ForwardAuth (Proxy Protection)
+Use this for applications without native OIDC (e.g. Zigbee2MQTT, Prometheus, phpMyAdmin, OpenClaw).
+
+1. In Authentik Admin, go to **Applications > Providers**:
+   * Create a **Proxy Provider**.
+   * Name: `<App Name> Proxy` (e.g. `Zigbee2MQTT Proxy`).
+   * Mode: `Forward auth (single application)`.
+   * External Host: `https://<app-domain>` (e.g. `https://zigbee2mqtt.moritzgraf.de`).
+2. Go to **Applications > Applications**:
+   * Create Application, link the Provider, and set access policies/groups.
+3. Add the middleware annotation to the application's Kubernetes Ingress:
+   ```yaml
+   annotations:
+     traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+   ```
+
+### Pattern 2: Native OIDC SSO
+Use this for applications supporting OpenID Connect (Grafana, Coder, Gitea, Nextcloud).
+
+1. In Authentik Admin, create an **OAuth2/OpenID Provider**:
+   * Name: `<App Name> OIDC`
+   * Client Type: `Confidential`
+   * Redirect URI: e.g. `https://grafana.haumdaucher.de/login/generic_oauth`
+2. Create an **Application** in Authentik and assign it to the Provider.
+3. Configure the client credentials (`client_id` and `client_secret`) in the application's Helm values / ConfigMap.
+
+---
+
+## 🎨 Branding & Haumdaucher UI Theme
+
+The Authentik login screen is customized with a **Haumdaucher** glassmorphism theme matching `haumdaucher.de`.
+
+* **Styling Highlights**:
+  * **Glassmorphic Card**: Frosted glass panel (`backdrop-filter: blur(20px)`), smooth rounded corners (`24px`), and elegant ambient background aura.
+  * **Typography**: Clean, modern typography using Google Font **Outfit**.
+  * **Logo & Entrance Animation**: Dynamic 3D scale-and-fade entry animation for the Haumdaucher crest logo (`https://haumdaucher.de/images/logo_classic.png`).
+  * **Dual Light & Dark Theme**: Automatic adaptation with dark mode styling featuring obsidian surfaces, gold accents (`#d4af37`), and glowing tactile buttons.
+  * **Branded Footer**: "HAUMDAUCHER Wurst und Spezialitäten n.e.V." subtle watermark.
+
+### Managing Theme via Terraform
+Branding is provisioned via the `authentik_brand` resource in `terraform/authentik.tf` using the stylesheet `terraform/haumdaucher-theme.css`:
+
+```bash
+cd terraform
+terraform apply -target=authentik_brand.haumdaucher
+```
+
+---
+
+## 💾 Backup & Disaster Recovery
+
+* **Database**: Managed by CloudNative-PG on `openebs-hostpath` persistent storage.
+* **Velero Integration**: The `auth` namespace is backed up during daily/weekly Velero schedules:
+  ```bash
+  velero backup create auth-backup-$(date +%Y%m%d) --include-namespaces auth --wait
+  ```

k8s/auth/USER_MANAGEMENT.md

@@ -0,0 +1,629 @@
+# Authentik User Management: Complete Roundup & Operations Guide
+
+This guide covers everything you need to know about managing users in Authentik for the **Haumdaucher** infrastructure (`https://auth.haumdaucher.de`): from manual creation and invitation workflows to Terraform IaC, custom email branding, fine-grained access control, handling custom attributes (e.g., Vereins-Mitgliedsnummer, second email), and bulk-importing users from an Excel / CSV spreadsheet.
+
+---
+
+## 📑 Table of Contents
+
+1. [Executive Summary: Quick Answers](#-executive-summary-quick-answers)
+2. [Method 1: Manual Web UI & Invitation Links](#-method-1-manual-web-ui--invitation-links)
+3. [Method 2: User Provisioning via Terraform](#-method-2-user-provisioning-via-terraform)
+4. [Method 3: Bulk Import from Excel / CSV (Python Script)](#-method-3-bulk-import-from-excel--csv-python-script)
+5. [Customizing Registration & Onboarding Emails](#-customizing-registration--onboarding-emails)
+6. [Access Control & Authorization Policies (Limiting App Access)](#-access-control--authorization-policies-limiting-app-access)
+7. [Custom Attributes & OIDC Claim Mappings](#-custom-attributes--oidc-claim-mappings)
+8. [Production Checklist & Best Practices](#-production-checklist--best-practices)
+
+---
+
+## ⚡ Executive Summary: Quick Answers
+
+| Question | Answer & Key Mechanism |
+| :--- | :--- |
+| **How to add new users?** | **4 Ways**: (1) Admin Web UI (`Directory > Users`), (2) **Invitations flow** (`Directory > Invitations` - recommended for self-service), (3) **Terraform** (`authentik_user` resource), or (4) **REST API / Python script** for bulk imports. |
+| **Can this be done in Terraform?** | **Yes.** Use `resource "authentik_user"` in `terraform/authentik.tf`. Supports custom attributes via `jsonencode()`, group assignments, and active status. |
+| **Can I change the email sent to the user?** | **Yes.** Configured in **Flows & Stages > Stages > Email Stage**. You can customize the subject line, token expiration, and HTML template (written in Django template syntax with access to user fields and custom attributes). |
+| **Can I limit what a user has access to?** | **Yes.** Through **Policy Bindings** attached to Applications: Group Membership Policies (e.g., `infra-admins` vs `users`), IP/Geo Policies, or Python Expression Policies checking custom attributes (e.g., only active club members). |
+| **How to handle Excel tables with custom attributes?** | Authentik stores arbitrary key-value pairs in `user.attributes` (JSON). You can import an Excel/CSV file with a Python script or Terraform `csvdecode()`, map fields like `haumdaucher_mitgliedsnummer` or `second_email`, and expose them to downstream apps (Nextcloud, Grafana, Gitea) via OIDC Scope Mappings. |
+
+---
+
+## 👤 Method 1: Manual Web UI & Invitation Links
+
+Authentik offers two UI methods to onboard users:
+
+### Option A: Direct User Creation (Admin sets password or triggers reset)
+1. Go to `https://auth.haumdaucher.de` and log in with your admin account.
+2. Navigate to **Directory > Users** and click **Create**.
+3. Fill in:
+   * **Username**: `max.mustermann`
+   * **Name**: `Max Mustermann`
+   * **Email**: `max@example.com`
+   * **Type**: `Internal`
+   * **Groups**: Select groups (e.g., `users`, `verein-mitglieder`).
+   * **Attributes (YAML/JSON)**:
+     ```json
+     {
+       "second_email": "max.privat@gmail.com",
+       "haumdaucher_mitgliedsnummer": "HM-2024-042",
+       "phone": "+49 170 1234567"
+     }
+     ```
+4. Click **Create**.
+5. To set credentials: Click on the user -> **Actions** -> **Set Password** (or **Explicitly trigger password reset flow** so the user receives a link via email).
+
+---
+
+---
+
+## ⚠️ Troubleshooting: "Flow does not apply to current user"
+
+If you encounter this screen when opening an invite link:
+
+> **"Flow does not apply to current user."**
+
+### Why this happens:
+1. **Wrong Flow Selected (`default-source-enrollment`)**:
+   In your URL (`/if/flow/default-source-enrollment/?itoken=...`), the flow used was `default-source-enrollment`. That flow is strictly designed for external identity providers (OAuth/SAML like Google/GitHub) and binds a policy that checks for an active external source in the context. When accessed directly with an `?itoken=...`, Authentik evaluates this policy, fails, and displays this error.
+2. **Missing Invitation Stage**:
+   Standard enrollment flows reject direct token access unless an **`Invitation Stage`** is bound to the beginning of the flow to validate `?itoken=...`.
+3. **Active Admin Session**:
+   If an administrator is currently logged in to the browser and clicks the enrollment link, flows with an "unauthenticated-only" policy will reject the session. Always test invitation URLs in an **Incognito / Private window**.
+
+---
+
+### Solution: The Haumdaucher Custom Invitation Flow (`haumdaucher-invitation-flow`)
+
+We provision a dedicated, streamlined flow in `terraform/authentik.tf`:
+
+```mermaid
+graph LR
+    Link[Invite Link with ?itoken=...] --> Stage1[1. Invitation Stage<br/>Validates Token]
+    Stage1 --> Stage2[2. Prompt Stage<br/>User inputs Username, Name, Password]
+    Stage2 --> Stage3[3. User Write Stage<br/>Writes to DB & assigns Groups]
+    Stage3 --> Stage4[4. User Login Stage<br/>Logs user into session]
+    Stage4 --> Stage5[5. Redirect Stage<br/>Redirects to haumdaucher.de]
+```
+
+#### Flow Behavior (Tailored to your setup):
+* **User Inputs**: Prompts the user to choose their preferred **Username**, **Full Name**, and **Password** (with confirmation).
+* **Email & Custom Attributes**: Pre-filled from the admin invitation (e.g. `haumdaucher_mitgliedsnummer`, `second_email`).
+* **MFA / 2FA**: Optional. The user registers immediately with their password and can enroll Passkeys/TOTP anytime in their profile settings.
+* **Automatic Redirect**: Upon completing registration, the user is automatically logged in and redirected to **`https://haumdaucher.de`**.
+
+---
+
+### How to Create an Invitation using the Custom Flow
+
+1. Go to `https://auth.haumdaucher.de` > **Directory > Invitations**.
+2. Click **Create** (or **New Invitation**).
+3. Set the fields:
+   * **Name**: `Invite - Max Mustermann`
+   * **Expires**: Set expiration (e.g., 7 days).
+   * **Flow**: Select **`Haumdaucher Invitation Flow`** (`haumdaucher-invitation-flow`).
+   * **Single use**: Checked (`true`).
+   * **Fixed Data / Attributes (JSON)**:
+     ```json
+     {
+       "email": "max@example.com",
+       "haumdaucher_mitgliedsnummer": "HM-2024-042",
+       "second_email": "max.privat@gmail.com"
+     }
+     ```
+4. Click **Create**.
+5. Copy the generated URL:
+   ```
+   https://auth.haumdaucher.de/if/flow/haumdaucher-invitation-flow/?itoken=<generated-token>
+   ```
+6. Send the link to the user. When they open it, they will see the branded Haumdaucher registration form asking for their Username, Name, and Password!
+
+---
+
+## 🏗️ Method 2: User Provisioning via Terraform
+
+You can manage users, groups, and permissions directly in `terraform/authentik.tf`.
+
+### 1. Single User Definition
+```hcl
+# Group for club members
+resource "authentik_group" "verein_mitglieder" {
+  name         = "verein-mitglieder"
+  is_superuser = false
+}
+
+# User with custom attributes
+resource "authentik_user" "max_mustermann" {
+  username   = "max.mustermann"
+  name       = "Max Mustermann"
+  email      = "max@example.com"
+  type       = "internal"
+  is_active  = true
+  groups     = [
+    authentik_group.users.id,
+    authentik_group.verein_mitglieder.id
+  ]
+  attributes = jsonencode({
+    "second_email"                 = "max.privat@gmail.com",
+    "haumdaucher_mitgliedsnummer" = "HM-2024-042",
+    "eintrittsdatum"               = "2024-01-15",
+    "status"                       = "aktiv"
+  })
+}
+```
+
+### 2. Batch Provisioning from CSV in Terraform
+If you maintain users in a CSV file (`terraform/users.csv`):
+
+```csv
+username,name,email,second_email,mitgliedsnummer,is_admin
+max.mustermann,Max Mustermann,max@example.com,max.privat@gmail.com,HM-2024-042,false
+anna.schmidt,Anna Schmidt,anna@example.com,anna.work@company.de,HM-2024-043,true
+```
+
+In `terraform/authentik.tf`:
+```hcl
+locals {
+  csv_users = csvdecode(file("${path.module}/users.csv"))
+}
+
+resource "authentik_user" "csv_members" {
+  for_each = { for u in local.csv_users : u.username => u }
+
+  username   = each.value.username
+  name       = each.value.name
+  email      = each.value.email
+  type       = "internal"
+  is_active  = true
+  groups = compact([
+    authentik_group.users.id,
+    authentik_group.verein_mitglieder.id,
+    each.value.is_admin == "true" ? authentik_group.infra_admins.id : null
+  ])
+  attributes = jsonencode({
+    "second_email"                 = each.value.second_email,
+    "haumdaucher_mitgliedsnummer" = each.value.mitgliedsnummer
+  })
+}
+```
+
+### ⚖️ Trade-offs: When to use Terraform for Users
+* **Pros**: Version-controlled in Git, declarative, reproducible, ideal for admins and service accounts (`ai_assistant`).
+* **Cons / Considerations**: Terraform cannot trigger interactive password onboarding emails out of the box; passwords must be set via recovery flow or first login reset.
+
+---
+
+## 📊 Method 3: Bulk Import from Excel / CSV (Python Script)
+
+If you have an Excel file (`mitglieder.xlsx`) or CSV, using Authentik's REST API is the cleanest approach because it can **create users, attach custom attributes, and automatically email them an invitation link to set their own password**.
+
+### Ready-to-Run Python Import Script
+Save this script (e.g. `k8s/auth/scripts/import_users_excel.py`):
+
+```python
+#!/usr/bin/env python3
+"""
+Bulk User Importer for Authentik (auth.haumdaucher.de)
+Reads an Excel (.xlsx) or CSV file and creates users with custom attributes and invitation links.
+"""
+
+import sys
+import os
+import requests
+import pandas as pd
+
+# Configuration
+AUTHENTIK_URL = os.getenv("AUTHENTIK_URL", "https://auth.haumdaucher.de")
+AUTHENTIK_TOKEN = os.getenv("AUTHENTIK_TOKEN")  # API Token (e.g. ai_assistant_token)
+
+if not AUTHENTIK_TOKEN:
+    print("❌ Error: AUTHENTIK_TOKEN environment variable is required.")
+    sys.exit(1)
+
+HEADERS = {
+    "Authorization": f"Bearer {AUTHENTIK_TOKEN}",
+    "Content-Type": "application/json",
+    "Accept": "application/json"
+}
+
+def get_or_create_group(group_name: str) -> str:
+    """Fetch group ID or create if it doesn't exist."""
+    res = requests.get(f"{AUTHENTIK_URL}/api/v3/core/groups/?name={group_name}", headers=HEADERS)
+    res.raise_for_status()
+    results = res.json().get("results", [])
+    if results:
+        return results[0]["pk"]
+    
+    # Create group
+    create_res = requests.post(f"{AUTHENTIK_URL}/api/v3/core/groups/", headers=HEADERS, json={"name": group_name})
+    create_res.raise_for_status()
+    return create_res.json()["pk"]
+
+def import_users_from_file(file_path: str, send_invitation: bool = True):
+    # Read Excel or CSV
+    if file_path.endswith((".xlsx", ".xls")):
+        df = pd.read_excel(file_path)
+    else:
+        df = pd.read_csv(file_path)
+
+    # Standardize column names (lowercase & stripped)
+    df.columns = [c.strip().lower() for c in df.columns]
+
+    print(f"📋 Found {len(df)} rows to process.")
+    
+    users_group_pk = get_or_create_group("users")
+    verein_group_pk = get_or_create_group("verein-mitglieder")
+
+    for idx, row in df.iterrows():
+        name = str(row.get("name", "")).strip()
+        email = str(row.get("email", "")).strip()
+        username = str(row.get("username", "")).strip() or email.split("@")[0].lower()
+        second_email = str(row.get("second_email", row.get("zweite_email", ""))).strip()
+        mitgliedsnummer = str(row.get("mitgliedsnummer", row.get("verein_mitgliedsnummer", ""))).strip()
+
+        if not email or email == "nan":
+            print(f"⚠️ Skipping row {idx+1}: missing email")
+            continue
+
+        # Prepare user payload with custom attributes
+        user_payload = {
+            "username": username,
+            "name": name if name != "nan" else username,
+            "email": email,
+            "type": "internal",
+            "is_active": True,
+            "groups": [users_group_pk, verein_group_pk],
+            "attributes": {
+                "second_email": second_email if second_email != "nan" else "",
+                "haumdaucher_mitgliedsnummer": mitgliedsnummer if mitgliedsnummer != "nan" else "",
+                "source": "excel_import"
+            }
+        }
+
+        # 1. Create or Update User in Authentik
+        res = requests.get(f"{AUTHENTIK_URL}/api/v3/core/users/?username={username}", headers=HEADERS)
+        res.raise_for_status()
+        existing = res.json().get("results", [])
+
+        if existing:
+            user_pk = existing[0]["pk"]
+            patch_res = requests.patch(f"{AUTHENTIK_URL}/api/v3/core/users/{user_pk}/", headers=HEADERS, json=user_payload)
+            patch_res.raise_for_status()
+            print(f"🔄 Updated existing user: {username} ({email})")
+        else:
+            create_res = requests.post(f"{AUTHENTIK_URL}/api/v3/core/users/", headers=HEADERS, json=user_payload)
+            if not create_res.ok:
+                print(f"❌ Failed to create {username}: {create_res.text}")
+                continue
+            user_pk = create_res.json()["pk"]
+            print(f"✅ Created user: {username} ({email}) [ID: {user_pk}]")
+
+        # 2. Generate Invitation / Password Reset Link
+        if send_invitation:
+            invitation_payload = {
+                "name": f"Onboarding {username}",
+                "single_use": True,
+                "fixed_data": {
+                    "email": email,
+                    "username": username
+                }
+            }
+            inv_res = requests.post(f"{AUTHENTIK_URL}/api/v3/stages/invitation/invitations/", headers=HEADERS, json=invitation_payload)
+            if inv_res.ok:
+                inv_key = inv_res.json()["pk"]
+                link = f"{AUTHENTIK_URL}/if/flow/default-enrollment-flow/?itoken={inv_key}"
+                print(f"   ✉️ Invitation Link: {link}")
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: python3 import_users_excel.py <path-to-excel-or-csv>")
+        sys.exit(1)
+    import_users_from_file(sys.argv[1])
+```
+
+---
+
+## ✉️ Customizing Registration & Onboarding Emails
+
+When an admin creates a user or triggers an enrollment/password recovery, Authentik sends transactional emails. You can customize their sender address, subject line, and full HTML appearance.
+
+### 1. SMTP & Sender Settings
+Authentik sends emails using the SMTP configuration defined in `k8s/auth/authentik.secret.values.yaml` or global environment variables:
+* `AUTHENTIK_EMAIL__HOST`: `smtp.gmail.com` (or Google Workspace / OVH relay)
+* `AUTHENTIK_EMAIL__PORT`: `587`
+* `AUTHENTIK_EMAIL__USE_TLS`: `true`
+* `AUTHENTIK_EMAIL__USERNAME`: `service@haumdaucher.de`
+* `AUTHENTIK_EMAIL__PASSWORD`: `<app-password>`
+* `AUTHENTIK_EMAIL__FROM`: `HAUMDAUCHER Identity <service@haumdaucher.de>`
+
+---
+
+### 2. Customizing Email Stages in Authentik
+Authentik controls outgoing emails via **Email Stages**:
+1. Open Admin UI: **Flows & Stages > Stages**.
+2. Locate the relevant email stage:
+   * `default-password-reset-email` (for password recovery & manual user activation)
+   * `default-enrollment-email` / `invitation-email` (for new user invitations)
+   * `default-user-settings-password-reset-email`
+3. Click **Edit** on the stage:
+   * **Subject**: `Willkommen bei Haumdaucher e.V. – Aktiviere deinen Zugang`
+   * **Token Expiry**: `Minutes=1440` (24 hours) or `Minutes=10080` (7 days)
+   * **Template**: Select standard template or your custom template path.
+
+---
+
+### 3. Customizing the HTML Email Template (Django Syntax)
+Authentik email templates use Django template syntax. Available variables in email templates include:
+* `{{ user.name }}` / `{{ user.username }}` / `{{ user.email }}`
+* `{{ user.attributes.haumdaucher_mitgliedsnummer }}`
+* `{{ url }}` (The activation / password reset link)
+* `{{ token_expires }}` (Expiry timestamp)
+* `{{ brand.branding_title }}` (`HAUMDAUCHER`)
+
+#### Example Custom Haumdaucher Welcome Template
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <style>
+    body { font-family: 'Outfit', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background-color: #0d0d12; color: #f0f0f5; margin: 0; padding: 40px; }
+    .card { background: #181824; border: 1px solid rgba(212, 175, 55, 0.3); border-radius: 16px; max-width: 540px; margin: 0 auto; padding: 32px; box-shadow: 0 12px 30px rgba(0,0,0,0.5); }
+    .header { text-align: center; margin-bottom: 24px; }
+    .logo { height: 60px; }
+    h1 { color: #d4af37; font-size: 22px; margin-top: 12px; }
+    p { color: #c0c0d0; line-height: 1.6; font-size: 15px; }
+    .btn { display: block; text-align: center; background: linear-gradient(135deg, #d4af37, #aa8520); color: #000; font-weight: bold; text-decoration: none; padding: 14px 24px; border-radius: 10px; margin: 28px 0; }
+    .meta { background: rgba(255,255,255,0.05); border-radius: 8px; padding: 12px 16px; font-size: 13px; color: #a0a0b0; }
+    .footer { text-align: center; font-size: 12px; color: #606070; margin-top: 30px; }
+  </style>
+</head>
+<body>
+  <div class="card">
+    <div class="header">
+      <img class="logo" src="https://haumdaucher.de/images/logo_classic.png" alt="Haumdaucher Logo" />
+      <h1>Willkommen beim Haumdaucher e.V.</h1>
+    </div>
+    
+    <p>Hallo <strong>{{ user.name }}</strong>,</p>
+    <p>dein zentraler Zugang für die Haumdaucher Cloud & Services wurde eingerichtet.</p>
+    
+    {% if user.attributes.haumdaucher_mitgliedsnummer %}
+    <div class="meta">
+      <strong>Mitgliedsnummer:</strong> {{ user.attributes.haumdaucher_mitgliedsnummer }}<br/>
+      <strong>Benutzername:</strong> {{ user.username }}
+    </div>
+    {% endif %}
+
+    <p>Klicke auf den Button unten, um dein Passwort festzulegen und deinen Zugang zu aktivieren:</p>
+    
+    <a class="btn" href="{{ url }}">Zugang aktivieren & Passwort setzen</a>
+    
+    <p style="font-size: 12px; color: #888;">Dieser Link ist 7 Tage lang gültig. Falls du die Registrierung nicht angefordert hast, kannst du diese E-Mail ignorieren.</p>
+    
+    <div class="footer">
+      HAUMDAUCHER e.V. &bull; <a href="https://haumdaucher.de" style="color: #d4af37;">haumdaucher.de</a>
+    </div>
+  </div>
+</body>
+</html>
+```
+
+---
+
+## 🔒 Access Control & Authorization Policies (Limiting App Access)
+
+By default, an application without an attached policy allows **any authenticated user** to access it. Authentik provides fine-grained Access Control (RBAC & PBAC) to restrict access per application.
+
+```mermaid
+graph TD
+    subgraph "Authentik Directory Groups"
+        Admin["👑 <b>infra-admins</b><br/>(Moritz & Wife / Full Admin)"]
+        Verein["🍺 <b>verein-mitglieder</b><br/>(Club Members)"]
+        Normal["👤 <b>users</b><br/>(Normal Authenticated Users)"]
+    end
+
+    subgraph "Tier 1: Admin & Ops Tools (ForwardAuth / OIDC)"
+        Z2M["Zigbee2MQTT"]
+        Prom["Prometheus & Alertmanager"]
+        PMA["phpMyAdmin (Nextcloud & Gitea)"]
+        N8N["n8n Automation"]
+        MinIO["MinIO Console"]
+        Coder["Coder v2"]
+        OpenClaw["OpenClaw Engine"]
+    end
+
+    subgraph "Tier 2: Daily Shared Services (OIDC)"
+        HASS["Home Assistant"]
+        Grafana["Grafana Dashboards"]
+        Nextcloud["Nextcloud Cloud"]
+        Gitea["Gitea Git Platform"]
+        ClubWeb["haumdaucher.de"]
+    end
+
+    %% Admin Links (Full Superadmin access everywhere)
+    Admin ==>|Full Access| Z2M
+    Admin ==>|Full Access| OpenClaw
+    Admin ==>|Admin| HASS
+    Admin ==>|Admin| Grafana
+    Admin ==>|Admin| Nextcloud
+    Admin ==>|Admin| Gitea
+    Admin ==>|Admin| ClubWeb
+
+    %% Normal User Links (Restricted)
+    Normal -->|Viewer (Read-Only)| Grafana
+    Normal -->|Dashboard View| HASS
+    Normal -->|Standard User| Nextcloud
+    Normal -->|Developer| Gitea
+
+    %% Club Members Links
+    Verein -->|Member Portal| ClubWeb
+    Verein -->|Club Shared Folder| Nextcloud
+    Verein -->|Viewer (Read-Only)| Grafana
+```
+
+### Master Permission Matrix
+
+| Application | Domain | Auth Type | `infra-admins` (Moritz & Wife) | `verein-mitglieder` (Club) | `users` (Normal Users) |
+| :--- | :--- | :--- | :---: | :---: | :---: |
+| **OpenClaw** | `openclaw.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Zigbee2MQTT** | `zigbee2mqtt.moritzgraf.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Prometheus / Alertmanager** | `prometheus.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **phpMyAdmin (All DBs)** | `*.phpmyadmin.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **n8n** | `n8n.moritzgraf.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **MinIO Console** | `minio-console.haumdaucher.de` | Native OIDC | **`consoleAdmin`** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Coder v2** | `coder.moritzgraf.de` | Native OIDC | **`Owner / Admin`** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Home Assistant** | `hass.moritzgraf.de` | Native OIDC | **`Administrator`** ✅ | ❌ *Denied* | **`Standard User`** |
+| **Grafana** | `grafana.haumdaucher.de` | Native OIDC | **`Admin`** ✅ | **`Viewer` (Read-Only)** | **`Viewer` (Read-Only)** |
+| **Nextcloud** | `cloud.haumdaucher.de` | Native OIDC | **`Admin`** ✅ | **`User` (Club Folder)** | **`User` (Standard)** |
+| **Gitea** | `git.moritzgraf.de` | Native OIDC | **`Admin`** ✅ | ❌ / Member | **`Developer`** |
+| **haumdaucher.de** | `haumdaucher.de` | ForwardAuth / OIDC | **`Admin`** ✅ | **`Member Access`** ✅ | ❌ *Denied* |
+
+### Policy Bindings in Terraform
+All policy bindings are enforced directly in `terraform/authentik.tf`:
+
+```hcl
+# 1. Admin-Only Tools (Restricted strictly to infra-admins)
+resource "authentik_policy_binding" "openclaw_admin_only" {
+  target = authentik_application.openclaw.uuid
+  group  = authentik_group.infra_admins.id
+  order  = 0
+}
+
+resource "authentik_policy_binding" "zigbee2mqtt_admin_only" {
+  target = authentik_application.zigbee2mqtt.uuid
+  group  = authentik_group.infra_admins.id
+  order  = 0
+}
+
+# 2. Shared & Member Applications (Accessible to users group)
+resource "authentik_policy_binding" "grafana_users" {
+  target = authentik_application.grafana.uuid
+  group  = authentik_group.users.id
+  order  = 0
+}
+
+resource "authentik_policy_binding" "nextcloud_users" {
+  target = authentik_application.nextcloud.uuid
+  group  = authentik_group.users.id
+  order  = 0
+}
+```
+
+---
+
+### 2. Attribute-Based Policy (Python Expression Policy)
+You can write Python expressions that evaluate custom user attributes dynamically:
+
+```hcl
+resource "authentik_policy_expression" "require_active_membership" {
+  name       = "require-active-haumdaucher-membership"
+  expression = <<EOF
+# Must have a valid Mitgliedsnummer and active status
+mitgliedsnummer = request.user.attributes.get('haumdaucher_mitgliedsnummer')
+status = request.user.attributes.get('status', 'aktiv')
+
+if not mitgliedsnummer or not mitgliedsnummer.startswith('HM-'):
+    ak_message("Keine gültige Haumdaucher Mitgliedsnummer hinterlegt.")
+    return False
+
+if status != 'aktiv':
+    ak_message("Deine Mitgliedschaft ist derzeit nicht aktiv.")
+    return False
+
+return True
+EOF
+}
+
+# Attach policy to Nextcloud application
+resource "authentik_policy_binding" "nextcloud_active_members" {
+  target = authentik_application.nextcloud.uuid
+  policy = authentik_policy_expression.require_active_membership.id
+  order  = 1
+}
+```
+
+---
+
+## 🏷️ Custom Attributes & OIDC Claim Mappings
+
+Authentik allows storing any arbitrary JSON object inside `user.attributes`.
+
+### 1. Recommended Attribute Schema for Haumdaucher
+```json
+{
+  "haumdaucher_mitgliedsnummer": "HM-2024-042",
+  "second_email": "max.privat@gmail.com",
+  "eintrittsdatum": "2024-01-15",
+  "verein_rolle": "Vorstand / Kassenwart / Mitglied",
+  "phone": "+49 170 1234567",
+  "telegram_handle": "@max_mustermann"
+}
+```
+
+---
+
+### 2. Exposing Custom Attributes via OIDC Claims (Terraform)
+To pass `haumdaucher_mitgliedsnummer` or `second_email` to downstream applications like **Nextcloud**, **Grafana**, or **Gitea**:
+
+```hcl
+# Define a custom Scope Mapping in Terraform
+resource "authentik_property_mapping_provider_scope" "haumdaucher_profile" {
+  name        = "Haumdaucher Extended Profile"
+  scope_name  = "haumdaucher_profile"
+  description = "Provides Vereins-Mitgliedsnummer and secondary email to OIDC clients"
+  expression  = <<EOF
+return {
+    "haumdaucher_mitgliedsnummer": request.user.attributes.get("haumdaucher_mitgliedsnummer", ""),
+    "second_email": request.user.attributes.get("second_email", ""),
+    "verein_rolle": request.user.attributes.get("verein_rolle", "Mitglied")
+}
+EOF
+}
+
+# Add this mapping to the OAuth2 provider (e.g. Nextcloud or Grafana)
+resource "authentik_provider_oauth2" "nextcloud" {
+  name               = "Nextcloud"
+  client_id          = "nextcloud"
+  client_secret      = random_password.nextcloud_oauth_client_secret.result
+  client_type        = "confidential"
+  authorization_flow = data.authentik_flow.default_authorization_flow.id
+  invalidation_flow  = data.authentik_flow.default_invalidation_flow.id
+  signing_key        = data.authentik_certificate_key_pair.generated.id
+  
+  property_mappings  = [
+    data.authentik_property_mapping_provider_scope.openid.id,
+    data.authentik_property_mapping_provider_scope.email.id,
+    data.authentik_property_mapping_provider_scope.profile.id,
+    authentik_property_mapping_provider_scope.haumdaucher_profile.id, # <-- Custom scope
+  ]
+}
+```
+
+---
+
+## 🛡️ Production Checklist & Best Practices
+
+When managing users in production, keep the following considerations in mind:
+
+### 1. Passwordless & Multi-Factor Authentication (MFA / 2FA)
+* Enforce **WebAuthn / Passkeys (FIDO2)** or **TOTP (Authenticator App)** for users in the `infra-admins` group.
+* In Authentik, this is done by binding an Authenticator Validation Stage to the default authentication flow or creating an Admin Authorization Flow.
+
+### 2. User Self-Service & Attribute Editing
+* Authentik provides a self-service portal at `https://auth.haumdaucher.de/if/user/#/settings`.
+* Users can change their password, register Passkeys, and edit user profile fields.
+* **Security Rule**: Keep administrative attributes (like `haumdaucher_mitgliedsnummer`) read-only by not exposing them to the user edit flow, while allowing users to freely edit personal contact details like `second_email` or `phone`.
+
+### 3. Offboarding & Member Deactivation
+* If a member leaves the club:
+  1. Set `is_active = false` (or remove from `verein-mitglieder` group).
+  2. Authentik immediately terminates active sessions and revokes ForwardAuth access and OIDC refresh tokens.
+  3. No need to manually reset passwords on Nextcloud, Grafana, or Gitea—SSO handles deprovisioning instantly.
+
+### 4. GDPR / DSGVO & Privacy
+* Store only necessary data in user attributes.
+* If a member requests account deletion, deleting the user in Authentik cascades identity termination across all connected services.
+
+### 5. Backups & Audit Logs
+* User database changes are stored in the PostgreSQL cluster (`authentik-db`) and backed up with daily Velero snapshot schedules.
+* Authentik logs all login attempts, failed logins, and administrative modifications under **Events > Logs**.

k8s/auth/authentik-db.yaml

@@ -0,0 +1,24 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: authentik-db
+  namespace: auth
+spec:
+  instances: 1
+  bootstrap:
+    initdb:
+      database: authentik
+      owner: authentik
+  postgresql:
+    parameters:
+      shared_buffers: "128MB"
+      max_connections: "100"
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 2Gi
+  storage:
+    storageClass: openebs-hostpath
+    size: 5Gi

k8s/auth/authentik.secret.values.yaml

@@ -0,0 +1,64 @@
+# --- Authentik Helm Configuration ---
+# Uses CloudNative-PG cluster (authentik-db) for PostgreSQL
+
+global:
+  env:
+    - name: AUTHENTIK_POSTGRESQL__HOST
+      value: "authentik-db-rw.auth.svc.cluster.local"
+    - name: AUTHENTIK_POSTGRESQL__NAME
+      value: "authentik"
+    - name: AUTHENTIK_POSTGRESQL__USER
+      value: "authentik"
+    - name: AUTHENTIK_POSTGRESQL__PASSWORD
+      valueFrom:
+        secretKeyRef:
+          name: authentik-db-app
+          key: password
+    - name: AUTHENTIK_SESSION__DURATION
+      value: "days=30"
+
+authentik:
+  # Secret key for cookie signing and session encryption
+  secret_key: "gU2ohe8eighaiph4quaeCh1ieM3iechoh2ohNg5yahz6eephoh"
+  error_reporting:
+    enabled: false
+  log_level: info
+
+server:
+  replicas: 1
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 1536Mi
+
+  ingress:
+    enabled: true
+    ingressClassName: "traefik"
+    annotations:
+      kubernetes.io/ingress.class: "traefik"
+      cert-manager.io/cluster-issuer: "letsencrypt-prod"
+      kubernetes.io/tls-acme: "true"
+      nginx.ingress.kubernetes.io/ssl-redirect: "true"
+      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+      nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    hosts:
+      - auth.haumdaucher.de
+    tls:
+      - hosts:
+          - auth.haumdaucher.de
+        secretName: auth-haumdaucher-de
+
+worker:
+  replicas: 1
+  resources:
+    requests:
+      cpu: 100m
+      memory: 384Mi
+    limits:
+      memory: 768Mi
+
+# Disable bundled subchart databases (managed by CNPG)
+postgresql:
+  enabled: false

k8s/auth/phpldapadmin.yaml

@@ -1,25 +0,0 @@
-# # status: removed again
-
-# apiVersion: helm.fluxcd.io/v1
-# kind: HelmRelease
-# metadata:
-#   name: helm-phpldapadmin
-#   namespace: flux
-# spec:
-#   releaseName: phpldapadmin
-#   targetNamespace: auth
-#   chart:
-#     repository: https://cetic.github.io/helm-charts
-#     version: 0.1.3
-#     name: phpldapadmin
-#   values:
-#     service:
-#       type: ClusterIP
-#     ingress:
-#       enabled: true
-#       hosts:
-#         - "ldap.haumdaucher.de"
-#       annotations:
-#         cert-manager.io/cluster-issuer: "letsencrypt-prod"
-#       tls:
-#         - ldap-haumdaucher-de

k8s/auth/traefik-forwardauth-middleware.yaml

@@ -0,0 +1,26 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: authentik-forwardauth
+  namespace: auth
+spec:
+  forwardAuth:
+    address: http://authentik-server.auth.svc.cluster.local/outpost.goauthentik.io/auth/traefik
+    trustForwardHeader: true
+    authRequestHeaders:
+      - Authorization
+      - Cookie
+      - Accept
+    authResponseHeaders:
+      - X-authentik-username
+      - X-authentik-groups
+      - X-authentik-email
+      - X-authentik-name
+      - X-authentik-uid
+      - X-authentik-jwt
+      - X-authentik-meta-jwks
+      - X-authentik-meta-outpost
+      - X-authentik-meta-provider
+      - X-authentik-meta-app
+      - X-authentik-meta-version
+      - authorization

k8s/backup/job-gdrive-to-gphotos.yaml

@@ -0,0 +1,29 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: gdrive-to-gphotos
+  namespace: backup
+spec:
+  template:
+    spec:
+      containers:
+      - name: sync-agent
+        image: iptizer/swiss
+        volumeMounts:
+        - name: rclone-gdrive-config
+          mountPath: /rclone.conf
+          subPath: "rclone.conf"
+        command:
+        - "/bin/bash"
+        - "-c"
+        args:
+        - |
+          echo "Starting copy operation from Google Drive to Google Photos..."
+          cd /home/devbox && devbox run rclone --config /rclone.conf -v --stats 10s --stats-one-line copy --drive-root-folder-id="1f43AtWvfb7qd-eGr9nw-8vHraA5k9mVl" gdrive: "gphotos:album/All pictures of the last years"
+      volumes:
+      - name: rclone-gdrive-config
+        configMap:
+          name: rclone-gdrive-config
+      restartPolicy: Never
+  activeDeadlineSeconds: 86400
+  backoffLimit: 3

k8s/backup/verify_migration.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""
+Google Drive to Google Photos Migration Verification Tool
+
+This script analyzes the migration status of pictures from Google Drive to Google Photos.
+It compares Google Drive folder contents with Pod logs from Kubernetes (and/or Google Photos API)
+to determine exact file counts, total sizes, and completion status per subfolder.
+"""
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from collections import defaultdict
+
+DEFAULT_SECRET_FILE = os.path.join(os.path.dirname(__file__), "rclone.secret.yml")
+DEFAULT_DRIVE_ROOT_ID = "1f43AtWvfb7qd-eGr9nw-8vHraA5k9mVl"
+DEFAULT_PHOTOS_ALBUM = "All pictures of the last years"
+
+def extract_rclone_conf(secret_file_path):
+    """Extract rclone.conf from Kubernetes ConfigMap secret YAML file."""
+    if not os.path.exists(secret_file_path):
+        print(f"Error: Secret file not found at {secret_file_path}")
+        sys.exit(1)
+    
+    with open(secret_file_path, "r", encoding="utf-8") as f:
+        content = f.read()
+    
+    match = re.search(r"rclone\.conf:\s*\|\n(.*)", content, re.DOTALL)
+    if not match:
+        print("Error: Could not find rclone.conf data in secret file.")
+        sys.exit(1)
+    
+    conf_data = match.group(1)
+    # Remove leading 4-space indentation if present
+    cleaned_lines = []
+    for line in conf_data.split("\n"):
+        if line.startswith("    "):
+            cleaned_lines.append(line[4:])
+        else:
+            cleaned_lines.append(line)
+    
+    tmp_conf = tempfile.NamedTemporaryFile(mode="w", suffix=".conf", delete=False)
+    tmp_conf.write("\n".join(cleaned_lines))
+    tmp_conf.close()
+    return tmp_conf.name
+
+def get_k8s_job_logs(namespace="backup", job_name="gdrive-to-gphotos"):
+    """Fetch logs from all pods associated with the Kubernetes job."""
+    print("Fetching execution logs from Kubernetes pods...")
+    try:
+        res = subprocess.run(
+            ["kubectl", "get", "pods", "-n", namespace, "-l", f"job-name={job_name}", "-o", "jsonpath={.items[*].metadata.name}"],
+            capture_output=True, text=True, check=True
+        )
+        pod_names = res.stdout.strip().split()
+        if not pod_names:
+            print(f"Warning: No pods found for job '{job_name}' in namespace '{namespace}'.")
+            return ""
+        
+        all_logs = []
+        for pod in pod_names:
+            print(f"  Retrieving logs for pod: {pod}")
+            log_res = subprocess.run(
+                ["kubectl", "logs", "-n", namespace, pod],
+                capture_output=True, text=True
+            )
+            all_logs.append(log_res.stdout)
+        return "\n".join(all_logs)
+    except Exception as e:
+        print(f"Warning: Could not fetch k8s logs directly ({e}).")
+        return ""
+
+def parse_copied_files_from_logs(log_text):
+    """Parse log text for confirmed uploaded files per subfolder."""
+    copied_files = defaultdict(set)
+    for line in log_text.splitlines():
+        if "Copied (new)" in line or "Copied (replaced)" in line:
+            m = re.search(r"INFO\s+:\s+(.*?)/(.*?):\s+Copied", line)
+            if m:
+                folder, filename = m.group(1), m.group(2)
+                copied_files[folder].add(filename)
+    return copied_files
+
+def get_gdrive_subfolders(config_path, root_id):
+    """List subdirectories in the source Google Drive folder."""
+    print("Querying subfolders from Google Drive...")
+    res = subprocess.run(
+        ["rclone", "--config", config_path, "lsd", f"--drive-root-folder-id={root_id}", "gdrive:"],
+        capture_output=True, text=True
+    )
+    if res.returncode != 0:
+        print(f"Error listing Google Drive folders: {res.stderr}")
+        return []
+    
+    subfolders = []
+    for line in res.stdout.strip().splitlines():
+        if line:
+            parts = line.split(maxsplit=4)
+            if len(parts) >= 5:
+                subfolders.append(parts[4])
+    return sorted(subfolders)
+
+def get_gdrive_folder_stats(config_path, root_id, subfolder):
+    """Get file count and size for a Google Drive subfolder using rclone size."""
+    res = subprocess.run(
+        ["rclone", "--config", config_path, "size", f"--drive-root-folder-id={root_id}", f"gdrive:{subfolder}"],
+        capture_output=True, text=True
+    )
+    if res.returncode != 0:
+        return {"count": 0, "size_str": "Unknown", "bytes": 0}
+    
+    output = res.stdout.strip()
+    # Handle formats like "Total objects: 2.563k (2563)" or "Total objects: 524"
+    count_match = re.search(r"Total objects:\s*(?:[\d\.\w]+\s*\()?(\d+)\)?", output)
+    size_match = re.search(r"Total size:\s*([\d\.]+\s*[KMGT]?i?B)\s*\((.*?)\s*Bytes\)", output)
+    
+    count = int(count_match.group(1)) if count_match else 0
+    size_str = size_match.group(1) if size_match else "0 B"
+    size_bytes = int(size_match.group(2)) if size_match else 0
+    
+    return {"count": count, "size_str": size_str, "bytes": size_bytes}
+
+def main():
+    parser = argparse.ArgumentParser(description="Verify Google Drive to Google Photos migration status.")
+    parser.add_argument("--secret-file", default=DEFAULT_SECRET_FILE, help="Path to rclone.secret.yml")
+    parser.add_argument("--root-id", default=DEFAULT_DRIVE_ROOT_ID, help="Google Drive root folder ID")
+    parser.add_argument("--local-log", help="Path to local pod log file (optional override)")
+    parser.add_argument("--csv", help="Path to save CSV report output (optional)")
+    args = parser.parse_args()
+
+    print("==========================================================")
+    print(" Google Drive to Google Photos Migration Verification Tool")
+    print("==========================================================")
+    
+    # 1. Extract Config
+    config_path = extract_rclone_conf(args.secret_file)
+    
+    try:
+        # 2. Get Logs
+        if args.local_log and os.path.exists(args.local_log):
+            print(f"Reading logs from local file: {args.local_log}")
+            with open(args.local_log, "r", errors="ignore") as f:
+                log_text = f.read()
+        else:
+            log_text = get_k8s_job_logs()
+        
+        copied_map = parse_copied_files_from_logs(log_text)
+        
+        # 3. Get Google Drive Subfolders
+        subfolders = get_gdrive_subfolders(config_path, args.root_id)
+        if not subfolders:
+            print("No subfolders found or unable to connect to Google Drive.")
+            sys.exit(1)
+        
+        print(f"\nAnalyzing {len(subfolders)} subfolders...\n")
+        
+        results = []
+        total_drive_files = 0
+        total_drive_bytes = 0
+        total_copied_files = 0
+        
+        hdr = f"{'Subfolder Name':<35} | {'GDrive Files':<12} | {'GDrive Size':<12} | {'Uploaded':<10} | {'Status':<15}"
+        print(hdr)
+        print("-" * len(hdr))
+        
+        for folder in subfolders:
+            stats = get_gdrive_folder_stats(config_path, args.root_id, folder)
+            copied_count = len(copied_map.get(folder, set()))
+            
+            drive_count = stats["count"]
+            drive_size = stats["size_str"]
+            total_drive_files += drive_count
+            total_drive_bytes += stats["bytes"]
+            total_copied_files += copied_count
+            
+            if drive_count > 0 and copied_count >= drive_count:
+                status = "COMPLETE"
+            elif copied_count > 0:
+                pct = (copied_count / drive_count * 100) if drive_count > 0 else 0
+                status = f"IN PROGRESS ({pct:.1f}%)"
+            else:
+                status = "QUEUED / WAITING"
+            
+            print(f"{folder:<35} | {drive_count:<12} | {drive_size:<12} | {copied_count:<10} | {status:<15}")
+            
+            results.append({
+                "folder": folder,
+                "drive_files": drive_count,
+                "drive_size": drive_size,
+                "uploaded_files": copied_count,
+                "status": status
+            })
+        
+        print("-" * len(hdr))
+        total_gb = total_drive_bytes / (1024 ** 3)
+        summary_str = f"TOTAL: {total_drive_files} files ({total_gb:.2f} GiB) | Uploaded: {total_copied_files} files"
+        print(f"\n{summary_str}")
+        
+        # Save CSV if requested
+        if args.csv:
+            import csv
+            with open(args.csv, "w", newline="", encoding="utf-8") as csvfile:
+                writer = csv.DictWriter(csvfile, fieldnames=["folder", "drive_files", "drive_size", "uploaded_files", "status"])
+                writer.writeheader()
+                writer.writerows(results)
+            print(f"Saved CSV report to: {args.csv}")
+
+    finally:
+        if os.path.exists(config_path):
+            os.remove(config_path)
+
+if __name__ == "__main__":
+    main()

k8s/cert-manager/production-issuer.yaml

@@ -18,4 +18,4 @@ spec:
     solvers:
     - http01:
         ingress:
-          class: nginx
+          ingressClassName: traefik

k8s/cert-manager/staging-issuer.yaml

@@ -18,4 +18,4 @@ spec:
     solvers:
     - http01:
         ingress:
-          class:  nginx
+          ingressClassName: traefik

k8s/datalab/corona-api.yaml

@@ -1,86 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: corona-api
-  labels:
-    app: corona-api
-  namespace: datalab
-spec:
-  selector:
-    matchLabels:
-      app: corona-api
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: corona-api
-    spec:
-      imagePullSecrets:
-      - name: "registry-haumdaucher-de"
-      containers:
-      - image: registry.haumdaucher.de/corona_plumber:latest
-        imagePullPolicy: Always
-        name: corona-api
-        ports:
-        - containerPort: 8000
-
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: corona-api
-  namespace: datalab
-spec:
-  ports:
-  - port: 8000
-    targetPort: 8000
-    protocol: TCP
-  selector:
-    app: corona-api
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-api-haumdaucher
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona-api.haumdaucher.de"
-     secretName: corona-api-haumdaucher-de
-  rules:
-  - host: corona-api.haumdaucher.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona-api
-          servicePort: 8000
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-api-moritzgraf
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona-api.moritzgraf.de"
-     secretName: corona-api-moritzgraf-de
-  rules:
-  - host: corona-api.moritzgraf.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona-api
-          servicePort: 8000

k8s/datalab/corona.yaml

@@ -1,86 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: corona
-  labels:
-    app: corona
-  namespace: datalab
-spec:
-  selector:
-    matchLabels:
-      app: corona
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: corona
-    spec:
-      imagePullSecrets:
-      - name: "registry-haumdaucher-de"
-      containers:
-      - image: registry.haumdaucher.de/corona:latest
-        imagePullPolicy: Always
-        name: corona
-        ports:
-        - containerPort: 3838
-
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: corona
-  namespace: datalab
-spec:
-  ports:
-  - port: 3838
-    targetPort: 3838
-    protocol: TCP
-  selector:
-    app: corona
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-haumdaucher
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona.haumdaucher.de"
-     secretName: corona-haumdaucher-de
-  rules:
-  - host: corona.haumdaucher.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona
-          servicePort: 3838
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-moritzgraf
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona.moritzgraf.de"
-     secretName: corona-moritzgraf-de
-  rules:
-  - host: corona.moritzgraf.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona
-          servicePort: 3838

k8s/datalab/redeploy.yaml

@@ -1,79 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  creationTimestamp: null
-  name: redeploy
-  namespace: datalab
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  namespace: datalab
-  name: redeploy
-rules:
-- apiGroups: ["apps"]
-  resources: ["deployments"]
-  verbs: ["get",list","patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-# This role binding allows "jane" to read pods in the "default" namespace.
-# You need to already have a Role named "pod-reader" in that namespace.
-kind: RoleBinding
-metadata:
-  name: redeploy-pods
-  namespace: datalab
-subjects:
-# You can specify more than one "subject"
-- kind: ServiceAccount
-  name: redeploy
-roleRef:
-  # "roleRef" specifies the binding to a Role / ClusterRole
-  kind: Role #this must be Role or ClusterRole
-  name: redeploy
-  apiGroup: rbac.authorization.k8s.io
-
----
-apiVersion: batch/v1beta1
-kind: CronJob
-metadata:
-  name: redeploy-corona-api
-  namespace: datalab
-spec:
-  schedule: "0 5/12 * * *"
-  jobTemplate:
-    spec:
-      template:
-        spec:
-          serviceAccountName: redeploy
-          imagePullSecrets:
-          - name: "registry-haumdaucher-de"
-          containers:
-          - name: redeploy-corona-api
-            image: registry.haumdaucher.de/redeploy:latest
-            env:
-            - name: DEPLOYMENT
-              value: "corona-api"
-          restartPolicy: Never
-
----
-apiVersion: batch/v1beta1
-kind: CronJob
-metadata:
-  name: redeploy-corona
-  namespace: datalab
-spec:
-  schedule: "15 5/12 * * *"
-  jobTemplate:
-    spec:
-      template:
-        spec:
-          serviceAccountName: redeploy
-          imagePullSecrets:
-          - name: "registry-haumdaucher-de"
-          containers:
-          - name: redeploy-corona
-            image: registry.haumdaucher.de/redeploy:latest
-            env:
-            - name: DEPLOYMENT
-              value: "corona"
-          restartPolicy: Never

k8s/development/gitea-phpmyadmin.yml

@@ -1,13 +1,74 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gitea-db-adminer
+  namespace: development
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: gitea-db-adminer
+  template:
+    metadata:
+      labels:
+        app: gitea-db-adminer
+    spec:
+      containers:
+        - name: adminer
+          image: adminer:latest
+          ports:
+            - containerPort: 8080
+              name: http
+          env:
+            - name: ADMINER_DEFAULT_SERVER
+              value: "gitea-postgresql"
+            - name: ADMINER_DESIGN
+              value: "dracula"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
 ---
-db:
-  host: gitea-mariadb
-ingress:
-  enabled: true
-  hosts:
-  - path: "/"
-    tls: true
-    name: "gitea.phpmyadmin.haumdaucher.de"
-    tlsSecret: "gitea-phpmyadmin-haumdaucher-de"
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea-db-adminer
+  namespace: development
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 8080
+      name: http
+  selector:
+    app: gitea-db-adminer
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea-phpmyadmin
+  namespace: development
   annotations:
     cert-manager.io/cluster-issuer: "letsencrypt-prod"
-    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    kubernetes.io/ingress.class: "traefik"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitea.phpmyadmin.haumdaucher.de
+      secretName: gitea-phpmyadmin-haumdaucher-de
+  rules:
+    - host: gitea.phpmyadmin.haumdaucher.de
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea-db-adminer
+                port:
+                  number: 80

k8s/development/registry_ingress.yaml

@@ -8,7 +8,7 @@ metadata:
     kubernetes.io/tls-acme: "true"
     # ----------------------------------------------
     cert-manager.io/cluster-issuer: letsencrypt-prod
-    kubernetes.io/ingress.class: nginx
+    kubernetes.io/ingress.class: traefik
     meta.helm.sh/release-name: docker-registry
     meta.helm.sh/release-namespace: development
     nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
@@ -21,7 +21,7 @@ metadata:
     release: docker-registry
 spec:
   # --- ADDED: Critical for modern K8s ---
-  ingressClassName: nginx 
+  ingressClassName: traefik 
   # --------------------------------------
   rules:
   - host: registry.haumdaucher.de

k8s/family/duckdb-backend.yaml

@@ -0,0 +1,154 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: duckdb-backend
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: duckdb-backend
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: duckdb-backend
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      initContainers:
+        - name: git-init
+          image: alpine/git:latest
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -e
+              export HOME=/tmp
+              cd /wiki
+              if [ ! -d ".git" ]; then
+                echo "Initializing wiki repository on dedicated PVC..."
+                REPO_URL="https://${GIT_SYNC_USERNAME}:${GIT_SYNC_PASSWORD}@git.moritzgraf.de/moritz/knowledge-wiki.git"
+                git config --global http.sslVerify false
+                git config --global --add safe.directory /wiki
+                git config --global --add safe.directory '*'
+                git init
+                git remote add origin "$REPO_URL" || git remote set-url origin "$REPO_URL"
+                git fetch origin main
+                git checkout -f -B main origin/main
+                echo "Clone completed successfully."
+              else
+                echo "Repository already initialized on PVC, updating..."
+                REPO_URL="https://${GIT_SYNC_USERNAME}:${GIT_SYNC_PASSWORD}@git.moritzgraf.de/moritz/knowledge-wiki.git"
+                git config --global http.sslVerify false
+                git config --global --add safe.directory /wiki
+                git config --global --add safe.directory '*'
+                git remote set-url origin "$REPO_URL"
+                if [ -f .git/shallow ]; then
+                  echo "Unshallowing existing repository..."
+                  git fetch --unshallow origin main || git fetch origin main
+                else
+                  git fetch origin main
+                fi
+                git rebase --abort 2>/dev/null || true
+                git merge --abort 2>/dev/null || true
+                git reset --hard origin/main
+                git clean -fd
+              fi
+          env:
+            - name: GIT_SYNC_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: git-sync-username
+            - name: GIT_SYNC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: wiki-sync-token
+          volumeMounts:
+            - name: duckdb-wiki-data
+              mountPath: /wiki
+      containers:
+        - name: duckdb-backend
+          image: registry.haumdaucher.de/family/duckdb-backend:latest
+          imagePullPolicy: Always
+          env:
+            - name: DATA_DIR
+              value: "/wiki/data"
+            - name: DUCKDB_DATA_DIR
+              value: "/wiki/data"
+            - name: WIKI_DIR
+              value: "/wiki"
+            - name: GIT_REPO_DIR
+              value: "/wiki"
+            - name: GIT_REPO_URL
+              value: "https://git.moritzgraf.de/moritz/knowledge-wiki.git"
+            - name: GIT_REMOTE_URL
+              value: "https://git.moritzgraf.de/moritz/knowledge-wiki.git"
+            - name: GIT_SYNC_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: git-sync-username
+            - name: GIT_SYNC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: wiki-sync-token
+            - name: GIT_SYNC_INTERVAL_SECONDS
+              value: "60"
+            - name: PORT
+              value: "8000"
+            - name: HOST
+              value: "0.0.0.0"
+          ports:
+            - name: http
+              containerPort: 8000
+          volumeMounts:
+            - name: duckdb-wiki-data
+              mountPath: /wiki
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 1000m
+              memory: 1Gi
+          readinessProbe:
+            httpGet:
+              path: /duckdb-api/v1/healthz
+              port: 8000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /duckdb-api/v1/healthz
+              port: 8000
+            initialDelaySeconds: 15
+            periodSeconds: 20
+      volumes:
+        - name: duckdb-wiki-data
+          persistentVolumeClaim:
+            claimName: duckdb-wiki-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: duckdb-backend
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 8000
+      targetPort: 8000
+      name: http
+  selector:
+    app.kubernetes.io/name: duckdb-backend

k8s/family/ingress.yaml

@@ -0,0 +1,55 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: family-portal
+  namespace: family
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - family.haumdaucher.de
+      secretName: family-haumdaucher-de-tls
+  rules:
+    - host: family.haumdaucher.de
+      http:
+        paths:
+          - path: /duckdb-api
+            pathType: Prefix
+            backend:
+              service:
+                name: duckdb-backend
+                port:
+                  number: 8000
+          - path: /tracking/mila
+            pathType: Prefix
+            backend:
+              service:
+                name: mila
+                port:
+                  number: 80
+          - path: /tracking/moritz
+            pathType: Prefix
+            backend:
+              service:
+                name: moritz
+                port:
+                  number: 80
+          - path: /outpost.goauthentik.io
+            pathType: Prefix
+            backend:
+              service:
+                name: duckdb-backend
+                port:
+                  number: 8000
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: mila
+                port:
+                  number: 80

k8s/family/mila.yaml

@@ -0,0 +1,64 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mila
+  namespace: family
+  labels:
+    app.kubernetes.io/name: mila
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mila
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mila
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: mila
+          image: registry.haumdaucher.de/family/mila:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 80
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /tracking/mila/
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /tracking/mila/
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: mila
+  namespace: family
+  labels:
+    app.kubernetes.io/name: mila
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app.kubernetes.io/name: mila

k8s/family/moritz.yaml

@@ -0,0 +1,64 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: moritz
+  namespace: family
+  labels:
+    app.kubernetes.io/name: moritz
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: moritz
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: moritz
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: moritz
+          image: registry.haumdaucher.de/family/moritz:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 80
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /tracking/moritz/
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /tracking/moritz/
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: moritz
+  namespace: family
+  labels:
+    app.kubernetes.io/name: moritz
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app.kubernetes.io/name: moritz

k8s/family/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: family
+  labels:
+    name: family

k8s/family/pvc.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: duckdb-wiki-data
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: openebs-hostpath
+  resources:
+    requests:
+      storage: 5Gi

k8s/grabengasse/backend.yaml

@@ -0,0 +1,65 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grabengasse-backend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-backend
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grabengasse-backend
+  template:
+    metadata:
+      labels:
+        app: grabengasse-backend
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: backend
+          image: registry.haumdaucher.de/website/grabengasse_backend:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 8000
+              name: http
+          env:
+            - name: PROMETHEUS_URL
+              value: "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090"
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: 8000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8000
+            initialDelaySeconds: 10
+            periodSeconds: 15
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grabengasse-backend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-backend
+spec:
+  type: ClusterIP
+  ports:
+    - port: 8000
+      targetPort: 8000
+      name: http
+  selector:
+    app: grabengasse-backend

k8s/grabengasse/frontend.yaml

@@ -0,0 +1,62 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grabengasse-frontend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-frontend
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grabengasse-frontend
+  template:
+    metadata:
+      labels:
+        app: grabengasse-frontend
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: frontend
+          image: registry.haumdaucher.de/website/grabengasse_frontend:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 80
+              name: http
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grabengasse-frontend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-frontend
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app: grabengasse-frontend

k8s/grabengasse/ingress.yaml

@@ -0,0 +1,35 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grabengasse7
+  namespace: grabengasse
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - grabengasse7.de
+      secretName: grabengasse7-de
+  rules:
+    - host: grabengasse7.de
+      http:
+        paths:
+          - path: /api
+            pathType: Prefix
+            backend:
+              service:
+                name: grabengasse-backend
+                port:
+                  number: 8000
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: grabengasse-frontend
+                port:
+                  number: 80

k8s/grabengasse/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: grabengasse
+  labels:
+    name: grabengasse

k8s/haumdaucher/haumdaucher.yaml

@@ -0,0 +1,71 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: haumdaucher-website
+  namespace: haumdaucher
+  labels:
+    app: haumdaucher
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: haumdaucher
+  template:
+    metadata:
+      labels:
+        app: haumdaucher
+    spec:
+      imagePullSecrets:
+      - name: registry-haumdaucher-de
+      containers:
+      - name: haumdaucher
+        image: registry.haumdaucher.de/haumdaucher-website:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 80
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: haumdaucher-service
+  namespace: haumdaucher
+spec:
+  selector:
+    app: haumdaucher
+  ports:
+    - protocol: TCP
+      port: 80
+      targetPort: 80
+  type: ClusterIP
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: haumdaucher-ingress
+  namespace: haumdaucher
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    kubernetes.io/ingress.class: nginx
+    kubernetes.io/tls-acme: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-buffering: "off"
+    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: haumdaucher.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: haumdaucher-service
+            port:
+              number: 80
+  tls:
+  - hosts:
+    - haumdaucher.de
+    secretName: haumdaucher-de-tls

k8s/kuard/ingress.yaml

@@ -1,12 +1,11 @@
-apiVersion: extensions/v1beta1
+apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   name: kuard
   namespace: kuard
   annotations:
-    kubernetes.io/ingress.class: "nginx"    
+    kubernetes.io/ingress.class: "traefik"    
     cert-manager.io/cluster-issuer: "letsencrypt-prod"
-
 spec:
   tls:
    - hosts:
@@ -17,6 +16,10 @@ spec:
     http:
       paths:
       - path: /
+        pathType: Prefix
         backend:
-          serviceName: kuard
-          servicePort: 80
+          service:
+            name: kuard
+            port:
+              number: 80
+

k8s/mailu/backup.secret.yml

@@ -1,45 +0,0 @@
-# kind: ConfigMap
-# metadata:
-#   creationTimestamp: null
-#   name: rclone-dropbox-config
-#   namespace: mailu
-# apiVersion: v1
-# data:
-#   rclone.conf: |
-#     [dropbox]
-#     type = dropbox
-#     token = {"access_token":"17hvEArIh3cAAAAAAAGTXUfwkGdz8EC_rDcEseLmSRltQoUz_AVceuUewx0sGMQ1","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}
-# ---
-# apiVersion: batch/v1beta1
-# kind: CronJob
-# metadata:
-#   name: backup-mailu
-#   namespace: mailu
-# spec:
-#   schedule: "30 4 * * *"
-#   jobTemplate:
-#     spec:
-#       template:
-#         spec:
-#           containers:
-#           - name: backup-mailu
-#             image: iptizer/swiss
-#             volumeMounts:
-#             - name: rclone-dropbox-config
-#               mountPath: /rclone.conf
-#               subPath: "rclone.conf"
-#             - mountPath: /data/
-#               name: data
-#             command:
-#               - "/bin/bash"
-#               - "-c"
-#             args: 
-#               - "cd / && tar zcvf - /data/ | /usr/bin/rclone --config /rclone.conf -v rcat dropbox:server_backup/$( /bin/expr $( date +%m ) % 3 )_$( /bin/expr $( date +%j ) % 3 )_mailu.tar.gz"
-#           volumes:
-#           - name: rclone-dropbox-config
-#             configMap:
-#               name: rclone-dropbox-config
-#           - name: data
-#             persistentVolumeClaim:
-#               claimName: mailu-storage
-#           restartPolicy: Never

k8s/mailu/mailu.secret.yml

@@ -1,82 +0,0 @@
-# DO NOT USE MASTER!!!!
-#mailuVersion: master
-# DO NOT USE MASTER!!!
-
-hostnames: 
-    - mail.moritzgraf.de
-domain: moritzgraf.de 
-secretKey: fa5faeD9aegietaesahbiequ5Pe9au
-subnet: 10.233.0.0/16
-
-ingress:
-  annotations: 
-    kubernetes.io/ingress.class: nginx
-
-clamav:
-  enabled: false
-front:
-  resources:
-    requests:
-      memory: 10Mi
-      cpu: 10m
-    limits:
-      memory: 200Mi
-      cpu: 200m
-admin:
-  resources:
-    requests:
-      memory: 20Mi
-      cpu: 50m
-    limits:
-      memory: 500Mi
-      cpu: 500m
-
-dovecot:
-  resources:
-    requests:
-      memory: 50Mi
-      cpu: 10m
-    limits:
-      memory: 500Mi
-      cpu: 500m
-postfix:
-  resources:
-    requests:
-      memory: 1Gi
-      cpu: 10m
-    limits:
-      memory: 2Gi
-      cpu: 500m
-
-redis:
-  resources:
-    requests:
-      memory: 20Mi
-      cpu: 10m
-    limits:
-      memory: 300Mi
-      cpu: 200m
-roundcube:
-  enabled: true
-  logLevel: INFO
-  resources:
-    requests:
-      memory: 10Mi
-      cpu: 10m
-    limits:
-      memory: 200Mi
-      cpu: 200m
-rspamd:
-  resources:
-    requests:
-      memory: 10Mi
-      cpu: 10m
-    limits:
-      memory: 2000Mi
-      cpu: 600m
-certmanager:
-  issuerType: ClusterIssuer
-  issuerName: "letsencrypt-prod"
-  apiVersion: cert-manager.io/v1
-persistence:
-  storageClass: openebs-hostpath

k8s/mailu/restore.yaml

@@ -1,25 +0,0 @@
-apiVersion: v1
-kind: Pod
-metadata:
-  creationTimestamp: null
-  labels:
-    run: restore
-  name: restore
-spec:
-  containers:
-  - image: iptizer/swiss
-    name: restore
-    resources: {}
-    volumeMounts:
-    - mountPath: /data/
-      name: data
-    command:
-      - "sleep"
-      - "3600"
-  dnsPolicy: ClusterFirst
-  restartPolicy: Never
-  volumes:
-  - name: data
-    persistentVolumeClaim:
-      claimName: mailu-storage
-status: {}

k8s/minio/hosting-ingress.yaml

@@ -0,0 +1,108 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: hosting-add-prefix
+  namespace: minio
+spec:
+  addPrefix:
+    prefix: /hosting
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: hosting-redirect-mila
+  namespace: minio
+spec:
+  redirectRegex:
+    regex: "^https?://([^/]+)/(?:family/)?(mila|sleep)\\.html.*"
+    replacement: "https://family.haumdaucher.de/tracking/mila/"
+    permanent: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: hosting-redirect-moritz
+  namespace: minio
+spec:
+  redirectRegex:
+    regex: "^https?://([^/]+)/(?:family/)?(moritz|nutrition)\\.html.*"
+    replacement: "https://family.haumdaucher.de/tracking/moritz/"
+    permanent: true
+---
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: hosting-redirect-family-root
+  namespace: minio
+spec:
+  redirectRegex:
+    regex: "^https?://([^/]+)/family/?$"
+    replacement: "https://family.haumdaucher.de/"
+    permanent: true
+---
+# Public Ingress for hosting.haumdaucher.de (Recipes, Public Assets, Legacy Redirects)
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: minio-hosting
+  namespace: minio
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.priority: "10"
+    traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-redirect-mila@kubernetescrd,minio-hosting-redirect-moritz@kubernetescrd,minio-hosting-redirect-family-root@kubernetescrd,minio-hosting-add-prefix@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+  - host: hosting.haumdaucher.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: minio
+            port:
+              number: 9000
+  tls:
+  - hosts:
+    - hosting.haumdaucher.de
+    secretName: hosting-haumdaucher-de-tls
+---
+# Protected Ingress for /family (Redirects and Authentik Outpost)
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: minio-hosting-family
+  namespace: minio
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.priority: "100"
+    traefik.ingress.kubernetes.io/router.middlewares: minio-hosting-redirect-mila@kubernetescrd,minio-hosting-redirect-moritz@kubernetescrd,minio-hosting-redirect-family-root@kubernetescrd,auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd
+spec:
+  ingressClassName: traefik
+  rules:
+  - host: hosting.haumdaucher.de
+    http:
+      paths:
+      - path: /family
+        pathType: Prefix
+        backend:
+          service:
+            name: minio
+            port:
+              number: 9000
+      - path: /outpost.goauthentik.io
+        pathType: Prefix
+        backend:
+          service:
+            name: minio
+            port:
+              number: 9000
+  tls:
+  - hosts:
+    - hosting.haumdaucher.de
+    secretName: hosting-haumdaucher-de-tls

k8s/monitoring/AGENTS.md

@@ -0,0 +1,116 @@
+# AGENTS.md — Monitoring Folder Guide for AI Agents
+
+## Overview
+
+This folder manages the **kube-prometheus-stack** deployment and **Grafana dashboards as code** for the `haumdaucher` Kubernetes cluster. The monitoring stack runs in the `monitoring` namespace.
+
+## Prerequisites — Grafana Skills
+
+Before creating or modifying dashboards, **read these skills first** — they cover general Grafana topics (JSON schema, panel types, PromQL, provisioning) that this document does not repeat:
+
+| Skill | Covers |
+|---|---|
+| **`dashboarding`** | Dashboard JSON schema, panel types, units, template variables, transformations, annotations |
+| **`grafana-oss`** | Dashboard/datasource provisioning, RBAC, plugin config, health checks |
+| **`promql`** | PromQL query writing, rate/irate, histograms, recording rules, cardinality |
+
+For project-specific dashboard creation workflows (ConfigMap template, InfluxDB Flux queries, deploy/verify steps), see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.
+
+## Folder Structure
+
+```
+monitoring/
+├── AGENTS.md                          ← You are here
+├── README.md                          ← Human-oriented deployment guide
+├── DASHBOARD_CREATION.md              ← Project-specific dashboard creation guide
+├── prometheus-operator.secret.yml     ← Helm values for kube-prometheus-stack
+├── alertmanagerconfig.secret.yaml     ← Alertmanager config (Telegram alerts)
+├── servicemonitor.secret.yml          ← ServiceMonitor examples (commented out)
+├── taupi-fan.yml                      ← Cellar fan exporter (Service, Endpoints, ServiceMonitor, Alerts)
+├── tankerkoenig.yml                   ← Tankerkoenig fuel price exporter
+├── dashboards/                        ← Grafana dashboards as ConfigMaps
+│   ├── grafana-dashboard-home-climate.yaml
+│   ├── grafana-dashboard-energy-monitor.yaml
+│   ├── grafana-dashboard-heating-gas.yaml
+│   ├── grafana-dashboard-outdoor-weather.yaml
+│   └── grafana-dashboard-taupi-fan.yaml
+└── scripts/                           ← Discovery & helper scripts
+    ├── discover-influxdb.sh           ← Query InfluxDB schema
+    └── discover-home-assistant.sh     ← Query Home Assistant entities
+```
+
+## Key Infrastructure Context
+
+### Namespaces & Services
+
+| Namespace | Service | Purpose |
+|-----------|---------|---------|
+| `monitoring` | kube-prometheus-stack | Prometheus, Alertmanager, Grafana |
+| `influxdb` | InfluxDB2 | Time-series storage for Home Assistant sensor data |
+| `home-assistant` | Home Assistant | Smart home hub, writes sensor data to InfluxDB |
+
+### Data Flow
+
+```
+Home Assistant sensors → InfluxDB (bucket: default, org: influxdata)
+                                    ↓
+                         Grafana (Flux queries via datasource)
+
+Custom exporters (taupi-fan, etc.) → Prometheus (scraped via ServiceMonitor)
+                                              ↓
+                                   Grafana (PromQL queries)
+```
+
+### Grafana Datasources
+
+| Name | UID | Type | URL |
+|------|-----|------|-----|
+| InfluxDB Home Assistant | `P2AB959DC95E5519F` | influxdb (Flux) | `http://influxdb-influxdb2.influxdb.svc.cluster.local:80` |
+| InfluxDB Home Assistant InfluxQL | `P86455D1023ECC470` | influxdb (InfluxQL) | same |
+| Prometheus | `prometheus` | prometheus | internal |
+
+> **⚠️ IMPORTANT**: The datasource UIDs above are the **current** values. If the Grafana deployment is recreated, these UIDs will change. Always re-discover UIDs via `kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- curl -s -u "admin:<password>" http://localhost:3000/api/datasources` before creating new dashboards.
+
+### Grafana Admin Credentials
+
+- URL: `https://grafana.haumdaucher.de`
+- User: `admin`
+- Password: stored in `prometheus-operator.secret.yml` under `grafana.adminPassword`
+
+### Home Assistant API
+
+- URL: `https://hass.moritzgraf.de`
+- Token: stored in `../home-assistant/tmp_long_lived_token.secret.yml`
+
+### InfluxDB API
+
+- Internal URL: `http://influxdb-influxdb2.influxdb.svc.cluster.local:80`
+- Org: `influxdata`
+- Bucket: `default`
+- Token: stored in `prometheus-operator.secret.yml` under `grafana.additionalDataSources[0].secureJsonData.token`
+
+## Dashboard Architecture
+
+Dashboards are managed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The kube-prometheus-stack's **Grafana Dashboard Sidecar** (`grafana-sc-dashboard` container, singular!) watches for these ConfigMaps and auto-loads them into Grafana.
+
+### Key Design Decisions
+
+1. **One file per dashboard** — Each ConfigMap YAML contains the dashboard JSON embedded directly in `data`.
+2. **Strict GitOps** — Dashboards are `editable: false` in Grafana UI. All changes go through YAML files.
+3. **Namespace-scoped** — Sidecar only watches `monitoring` namespace (`searchNamespace: monitoring`).
+4. **Folder annotation** — Use `grafana_dashboard_folder` annotation to organize dashboards.
+5. **No `"dashboard"` wrapper** — File provisioning requires top-level JSON properties. The `{"dashboard": {...}}` wrapper is only for the HTTP API.
+6. **Never add `uid` to helm datasource config** — Let Grafana auto-generate UIDs. Adding `uid` to `additionalDataSources` crashes provisioning.
+
+For the full creation workflow, Flux query templates, and InfluxDB conventions, see **[DASHBOARD_CREATION.md](DASHBOARD_CREATION.md)**.
+
+## Related Files Outside This Folder
+
+- `../home-assistant/home-assistant.secret.yaml` — HA helm values (InfluxDB integration config)
+- `../home-assistant/tmp_long_lived_token.secret.yml` — HA API token for discovery
+- `../influxdb/influxdb2.secret.yml` — InfluxDB helm values
+
+## Alerting Policy
+
+We prioritize a high threshold for alerts to prevent alarm fatigue in this private home lab environment:
+* **Alert Duration Threshold:** Most alerts (including container downtime, offline bridges, and missing device metrics) are configured with a `for: 1h` duration. They are only treated as critical and trigger active alerts if they remain unresolved for longer than 1 hour.

k8s/monitoring/DASHBOARD_CREATION.md

@@ -0,0 +1,226 @@
+# DASHBOARD_CREATION.md — How to Create & Update Grafana Dashboards
+
+> **Linked from**: [AGENTS.md](AGENTS.md)
+
+This document captures the project-specific process for creating Grafana dashboards in this folder. For general Grafana dashboard JSON schema, panel types, units, transformations, and API workflows, use the **Grafana skills** (`dashboarding`, `grafana-oss`, `promql`) — they are the authoritative reference for Grafana-native topics.
+
+> [!IMPORTANT]
+> **Always read the skills first** before creating a dashboard:
+> - **`dashboarding`** — JSON schema, panel types, units, template variables, transformations, links, annotations
+> - **`grafana-oss`** — Dashboard provisioning, datasource config, RBAC, plugin provisioning
+> - **`promql`** — PromQL query patterns, rate/irate, histograms, recording rules
+
+---
+
+## Step 1: Identify the Data Source
+
+This project has **two** types of dashboard data sources:
+
+| Data Source | Type | When to Use | Query Language |
+|---|---|---|---|
+| Prometheus | `prometheus` | Kubernetes metrics, custom exporters (taupi-fan, tankerkoenig) | PromQL |
+| InfluxDB Home Assistant | `influxdb` (Flux) | Home Assistant sensor data (temperature, humidity, energy, etc.) | Flux |
+
+**Prometheus dashboards** are simpler — just write PromQL expressions in `targets[].expr`. See the `promql` skill and existing examples like [grafana-dashboard-taupi-fan.yaml](dashboards/grafana-dashboard-taupi-fan.yaml).
+
+**InfluxDB dashboards** require Flux queries with project-specific conventions (see [§ InfluxDB Flux Queries](#influxdb-flux-queries) below).
+
+### Datasource UIDs
+
+Datasource references in panels require the **actual UID**, not the display name. UIDs can change when Grafana is redeployed. Current known UIDs are documented in [AGENTS.md](AGENTS.md) — always verify before creating a new dashboard:
+
+```bash
+kubectl -n monitoring exec deploy/prometheus-operator-grafana -c grafana -- \
+  curl -s -u "admin:<GRAFANA_PASSWORD>" \
+  http://localhost:3000/api/datasources | jq '.[] | {name, uid, type}'
+```
+
+The Grafana admin password is in [`prometheus-operator.secret.yml`](prometheus-operator.secret.yml) under `grafana.adminPassword`.
+
+---
+
+## Step 2: Discover Available Metrics
+
+### For Prometheus Dashboards
+
+Query Prometheus directly or use the Grafana Explore UI:
+
+```bash
+# Port-forward to Prometheus
+kubectl -n monitoring port-forward svc/prometheus-operated 9090
+
+# Browse available metrics
+curl -s http://localhost:9090/api/v1/label/__name__/values | jq '.data[]' | grep '<prefix>'
+```
+
+### For InfluxDB Dashboards (Home Assistant Data)
+
+Use the helper scripts to discover what data exists:
+
+```bash
+# Query Home Assistant for entity metadata
+./scripts/discover-home-assistant.sh
+
+# Query InfluxDB for measurements and schema
+# (requires port-forward first: kubectl -n influxdb port-forward svc/influxdb-influxdb2 8086:80)
+./scripts/discover-influxdb.sh
+```
+
+**Sensor selection tip**: Use **Wandthermostat** entities for temperature/humidity (they provide calibrated readings). Filter by `wandthermostat` or `thermostat` in the entity_id.
+
+---
+
+## Step 3: Create the Dashboard ConfigMap
+
+Dashboards are deployed as **Kubernetes ConfigMaps** with the label `grafana_dashboard: "1"`. The Grafana sidecar (`grafana-sc-dashboard` container) watches for these and auto-loads them.
+
+### ConfigMap Template
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-<name>
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+  annotations:
+    grafana_dashboard_folder: "🏠 Home"
+data:
+  <name>.json: |
+    {
+      "uid": "<unique-uid>",
+      "title": "<Dashboard Title>",
+      "tags": ["<category>"],
+      "timezone": "browser",
+      "schemaVersion": 39,
+      "version": 1,
+      "refresh": "30s",
+      "editable": false,
+      "graphTooltip": 2,
+      "time": { "from": "now-24h", "to": "now" },
+      "panels": [ ... ]
+    }
+```
+
+> [!CAUTION]
+> **No `"dashboard"` wrapper.** The JSON must have `uid`, `title`, `panels` at the top level. The `{"dashboard": {...}}` wrapper is only for the Grafana HTTP API — file provisioning via the sidecar will fail with *"Dashboard title cannot be empty"* if you use it.
+
+For the full panel JSON schema, units, thresholds, overrides, and transformations, see the **`dashboarding`** skill's `references/json-schema.md`.
+
+### Project-Specific Dashboard Conventions
+
+- `"editable": false` — strict GitOps; never edit in the Grafana UI
+- `"graphTooltip": 2` — shared crosshair + tooltip across all panels
+- Use emoji in titles for visual appeal (🏠 ⚡ 🔥 🌡️ 💧 💨)
+- Include `calcs` in legend: `["lastNotNull", "min", "max", "mean"]`
+- Use `"spanNulls": true` to handle gaps in sensor data
+- Use semantic threshold colors: blue=cold, green=comfort, orange=warm, red=hot
+
+---
+
+## Step 4: Deploy & Verify
+
+```bash
+# Apply
+kubectl apply -f dashboards/grafana-dashboard-<name>.yaml
+
+# Verify ConfigMap labeling
+kubectl get configmaps -n monitoring -l grafana_dashboard=1
+
+# Check sidecar pickup (container name is singular: grafana-sc-dashboard)
+kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
+  -c grafana-sc-dashboard --tail=20
+
+# Check for Grafana errors
+kubectl -n monitoring logs -l "app.kubernetes.io/name=grafana" \
+  -c grafana --tail=30 | grep -i "error\|failed"
+```
+
+### Updating an Existing Dashboard
+
+1. Edit the YAML file in `dashboards/`
+2. Increment the `version` field in the dashboard JSON
+3. `kubectl apply -f dashboards/grafana-dashboard-<name>.yaml`
+4. Sidecar detects the change within ~60 seconds
+
+### Removing a Dashboard
+
+```bash
+kubectl delete configmap -n monitoring grafana-dashboard-<name>
+```
+
+---
+
+## InfluxDB Flux Queries
+
+This section covers the **project-specific Flux conventions** for querying Home Assistant data in InfluxDB. These patterns are unique to this project and not covered by the Grafana skills.
+
+### Data Schema
+
+Home Assistant writes to InfluxDB with this structure:
+
+- **Measurement name** = unit of measurement (e.g., `°C`, `%`, `W`, `kWh`)
+- **Tags**: `entity_id` (e.g., `home_wohnzimmer_temperature`), `domain` (e.g., `sensor`)
+- **Fields**: `value` (numeric), plus metadata strings (`friendly_name_str`, etc.)
+
+### Critical Flux Query Rules
+
+> [!WARNING]
+> These rules come from hard-won debugging. Violating any of them will result in "no data" panels.
+
+1. **`entity_id` has NO `sensor.` prefix** — HA stores the domain in a separate `domain` tag. Use `home_wohnzimmer_temperature`, NOT `sensor.home_wohnzimmer_temperature`.
+2. **Always use bracket notation** for underscore-prefixed fields: `r["_measurement"]`, `r["_field"]` — dot notation (`r._measurement`) may silently fail.
+3. **Always use Grafana time variables** — `v.timeRangeStart`, `v.timeRangeStop`, `v.windowPeriod`. Never hardcode `range(start: -24h)`.
+4. **Always end with `|> yield(name: "mean")`** — required for Grafana to parse Flux results.
+5. **Always filter `r["_field"] == "value"`** — InfluxDB stores metadata strings in other fields.
+6. **Use regex for multi-entity panels** — `r.entity_id =~ /^home_/` instead of `or` chains (Flux `or` inside a single filter is unreliable).
+
+### Flux Query Templates
+
+**Time series panel (aggregated):**
+```flux
+from(bucket: "default")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
+  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
+  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
+  |> yield(name: "mean")
+```
+
+**Stat panel (current value):**
+```flux
+from(bucket: "default")
+  |> range(start: -15m)
+  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
+  |> filter(fn: (r) => r.entity_id == "<ENTITY_ID>")
+  |> last()
+  |> yield(name: "mean")
+```
+
+**Multi-entity panel (regex):**
+```flux
+from(bucket: "default")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r["_measurement"] == "<UNIT>" and r["_field"] == "value")
+  |> filter(fn: (r) => r.entity_id =~ /^home_/)
+  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
+  |> yield(name: "mean")
+```
+
+---
+
+## InfluxDB Measurement Reference
+
+| Measurement | Grafana Unit | Typical Entities |
+|---|---|---|
+| `°C` | `celsius` | Wandthermostat temperatures, outdoor temp, battery temp |
+| `%` | `percent` | Humidity, battery levels, heating valve position, cloud cover |
+| `W` | `watt` | Solar production, self-consumption, grid import/export |
+| `kWh` | `kwatth` | Daily energy totals (production, consumption, import, export) |
+| `m³` | `m3` | Gas volume counter |
+| `m³/h` | `m3/h` | Current gas flow rate |
+| `hPa` | `pressurehpa` | Barometric pressure |
+| `km/h` | `velocitykmh` | Wind speed |
+| `mm/h` | `lengthmm` | Precipitation rate |
+| `EUR` | `currencyEUR` | Gas cost, electricity cost |

k8s/monitoring/README.md

@@ -0,0 +1,90 @@
+# Grafana Dashboards as Code
+
+This directory contains Grafana dashboards managed as Kubernetes ConfigMaps. Dashboards are automatically loaded into Grafana via the dashboard sidecar.
+
+## Architecture
+
+```
+dashboards/*.yaml  (ConfigMap with embedded JSON)
+       │
+       │  kubectl apply
+       ▼
+Kubernetes ConfigMap (label: grafana_dashboard=1)
+       │
+       │  Sidecar watches for ConfigMaps
+       ▼
+Grafana Dashboard (read-only, GitOps managed)
+```
+
+## Quick Deploy
+
+```bash
+# Apply all dashboards
+kubectl apply -f dashboards/
+
+# Apply a single dashboard
+kubectl apply -f dashboards/grafana-dashboard-home-climate.yaml
+```
+
+## Verify
+
+```bash
+# Check ConfigMaps are created and labeled correctly
+kubectl get configmaps -n monitoring -l grafana_dashboard=1
+
+# Check sidecar logs to confirm dashboards were loaded
+kubectl logs -n monitoring deployment/kube-prometheus-stack-grafana -c grafana-sc-dashboards --tail=50
+
+# List dashboards via Grafana API
+kubectl -n monitoring exec deploy/kube-prometheus-stack-grafana -- \
+  curl -s -u admin:$(kubectl -n monitoring get secret kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d) \
+  http://localhost:3000/api/search?type=dash-db | jq '.[].title'
+```
+
+## Discovery
+
+Before generating new dashboards, discover what metrics are available:
+
+```bash
+# Discover InfluxDB measurements, fields, and tags
+./scripts/discover-influxdb.sh
+
+# Discover Home Assistant entities and their metadata
+./scripts/discover-home-assistant.sh
+```
+
+## Dashboard Structure
+
+Each dashboard is a single YAML file containing a Kubernetes ConfigMap with the dashboard JSON embedded:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-<name>
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+  annotations:
+    grafana_dashboard_folder: "<Folder Name>"
+data:
+  <name>.json: |
+    {
+      "dashboard": { ... }
+    }
+```
+
+## Adding a New Dashboard
+
+1. Run discovery scripts to understand available metrics
+2. Create a new ConfigMap YAML in `dashboards/`
+3. Apply: `kubectl apply -f dashboards/grafana-dashboard-<name>.yaml`
+4. Verify the dashboard appears in Grafana at https://grafana.haumdaucher.de
+
+## Removing a Dashboard
+
+```bash
+kubectl delete configmap -n monitoring grafana-dashboard-<name>
+```
+
+The sidecar will automatically remove the dashboard from Grafana within ~60 seconds.

k8s/monitoring/README_Z2M.md

@@ -0,0 +1,94 @@
+# Zigbee2MQTT & Shelly Weather Station Integration
+
+This guide describes how to deploy Zigbee2MQTT in the `monitoring` namespace and pair your **Shelly Weather Station (Ecowitt WS90)** using the **SLZB-MR1U Zigbee multiradio** (`192.168.10.10`).
+
+> [!NOTE]
+> These resources have already been successfully applied and verified running in the cluster.
+
+---
+
+## 1. Deploy Zigbee2MQTT
+
+The Kubernetes secret and workload manifests are located in `k8s/monitoring/`:
+
+```bash
+# 1. Apply the encrypted secrets (MQTT credentials & Traefik basic auth credentials)
+# Note: Basic auth credentials are split into 'zigbee2mqtt-auth-secret' to satisfy 
+# Traefik's constraint requiring exactly one key in the basic auth secret.
+kubectl apply -f k8s/monitoring/zigbee2mqtt.secret.yaml
+
+# 2. Apply the main deployment, PVC, ConfigMap, Service, Middleware, and Ingress
+kubectl apply -f k8s/monitoring/zigbee2mqtt.yaml
+```
+
+---
+
+## 2. Verify the Deployment
+
+1. Check that the Persistent Volume is provisioned and the pod starts:
+   ```bash
+   kubectl get pods -n monitoring -l app=zigbee2mqtt
+   ```
+
+2. Inspect the logs to ensure Zigbee2MQTT connects successfully to the SLZB-MR1U coordinator (`192.168.10.10:7638` — port `7638` is the TI CC2652P7 coordinator chip on the MR1U) and the Mosquitto MQTT broker (`mosquitto.datalab.svc.cluster.local`):
+   ```bash
+   kubectl logs -n monitoring deploy/zigbee2mqtt
+   ```
+
+   **Expected healthy output:**
+   ```
+   info:  z2m: Starting Zigbee2MQTT version 1.40.1 (commit #403d3c0)
+   info:  z2m: Starting zigbee-herdsman (0.57.3)
+   info:  zh:zstack:znp: Opening TCP socket with 192.168.10.10:7638
+   info:  zh:zstack:znp: Socket connected
+   info:  z2m: zigbee-herdsman started (resumed)
+   info:  z2m: Connecting to MQTT server at mqtt://mosquitto.datalab.svc.cluster.local:1883
+   info:  z2m: Connected to MQTT server
+   info:  z2m: Started frontend on port 0.0.0.0:8080
+   info:  z2m: Zigbee2MQTT started!
+   ```
+
+---
+
+## 3. Access the Dashboard
+
+The dashboard is exposed on your custom domain with TLS and basic authentication:
+* **URL:** [https://zigbee2mqtt.moritzgraf.de](https://zigbee2mqtt.moritzgraf.de)
+* **Username:** `admin`
+* **Password:** `moritz-z2m-2026`
+
+*Note: You can change the basic auth password by running the `scripts/htpasswd.py` script on your machine to generate a new bcrypt hash, then updating the `users` value in `k8s/monitoring/zigbee2mqtt.secret.yaml`.*
+
+---
+
+## 4. Pair the Weather Station (WS90)
+
+1. Open the Zigbee2MQTT dashboard and click **Permit join (All)** at the top right.
+2. Go to your physical Ecowitt WS90 weather station and **double-press the physical CAL button** on the bottom of the sensor array.
+3. The LED on the weather station should flash, indicating it is in pairing mode (active for 3 minutes).
+4. Within a few seconds, the weather station will appear in the Zigbee2MQTT dashboard. You can rename it there (e.g. to `Shelly WS90`).
+
+---
+
+## 5. Heartbeat LED Blinking (How to Disable)
+
+By default, the weather station's LED flashes briefly every ~8.8 seconds to indicate it is transmitting data.
+* **To disable the blinking:** Press the physical **CAL button** on the bottom of the sensor package **exactly 3 times** in quick succession. This toggles the heartbeat LED mode.
+* **To enable the blinking again:** Press the **CAL button exactly 3 times** again.
+
+---
+
+## 6. Verified Metrics Flow
+
+The weather station successfully reports metrics. You can query them in Prometheus or display them in Grafana:
+
+| Metric Name | Description | Verified Value Example |
+| :--- | :--- | :--- |
+| `homeassistant_sensor_temperature_celsius` | Outdoor Temperature | `30.8` °C |
+| `homeassistant_sensor_humidity_percent` | Outdoor Humidity | `29.0` % |
+| `homeassistant_sensor_pressure_kpa` | Atmospheric Pressure | `97.8` kPa |
+| `homeassistant_sensor_illuminance_lux_lx` | Ambient Light Level | `80.0` lx |
+| `homeassistant_sensor_battery_percent` | Sensor Battery Level | `100.0` % |
+
+> [!NOTE]
+> **Wind/Rain/UV Data:** The Shelly WS90 custom edition transmits advanced meteorological data (wind speed, wind direction, rain, and UV index) over manufacturer-specific Zigbee clusters. Depending on your Zigbee2MQTT version and device configuration, these may not be fully parsed out-of-the-box. If you require advanced wind/rain/UV metrics, integrating the device via **Bluetooth (BTHome protocol)** using a BLE proxy (e.g. an ESP32 or the SLZB-MR1U's Bluetooth proxy capability) is the recommended alternative.