feat(shelly): optimize dew point ventilation setpoints, duty cycle, and night-window gating

- Raise target humidity setpoints from 40% to 55% with a 2% hysteresis shut-off
- Eliminate 9999-min unlimited runtime bypass and strictly cap bursts at 35 min
- Rebalance duty cycle to 1:1 ratio (35 min run, 35 min cooldown rest)
- Implement Shelly internal RTC time-of-day check to restrict routine drying to 02:00-07:00
- Add delta-T guard requiring spread >= 3.5°C and max 20 min bursts when outside is >8°C colder
- Update AGENTS.md documentation to reflect optimized gates and thresholds

.agents/skills/dashboarding/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: dashboarding
+license: Apache-2.0
+description: Build, modify, and ship Grafana dashboards as JSON via the HTTP API — panel types (timeseries / stat / gauge / table / heatmap / logs / traces / node-graph), `gridPos` 24-column layout, units, thresholds, template + datasource + chained variables, transformations (`organize` / `calculateField` / `filterByValue`), panel + dashboard links with `${__field.labels.x}` / `${__from}`, and Loki/Prometheus annotations. Use when scripting dashboard creation, writing the dashboard JSON for a new service, adding a `$job` dropdown variable, computing an "Error %" column with a transformation, overlaying deploys as annotations, or pushing a dashboard via `POST /api/dashboards/db` — even when the user says "create a dashboard for this metric", "add a service dropdown", "show errors as percentage", "overlay our deploys", or "export the dashboard JSON" without naming the API or schema. After every API push, verify with the returned `version` plus a GET on the dashboard UID.
+---
+
+# Grafana Dashboard Authoring
+
+> **Docs**: https://grafana.com/docs/grafana/latest/dashboards/
+
+Dashboards are JSON. Author once, push via API, share by `uid`.
+
+## Prerequisites
+
+- Grafana stack (OSS, Enterprise, or Cloud) reachable from your machine
+- API token with `dashboards:write` (`Authorization: Bearer <token>`)
+- `jq` for inspecting responses
+- The JSON-schema cheat sheet in [`references/json-schema.md`](references/json-schema.md)
+
+## Common Workflows
+
+### 1. Push a new dashboard via the API + verify
+
+```bash
+# 1. Build the payload — wrap the dashboard JSON, set folder, mark overwrite
+cat > /tmp/dash.json <<'JSON'
+{
+  "dashboard": {
+    "uid": "demo-svc-v1",
+    "title": "Demo Service",
+    "schemaVersion": 41,
+    "tags": ["demo"],
+    "time": { "from": "now-1h", "to": "now" },
+    "templating": { "list": [] },
+    "panels": [{
+      "id": 1, "type": "timeseries", "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 24, "h": 8 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{
+        "expr": "sum(rate(http_requests_total[5m])) by (status_code)",
+        "legendFormat": "{{status_code}}", "refId": "A"
+      }],
+      "fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] }
+    }]
+  },
+  "folderUid": "",
+  "overwrite": true,
+  "message": "initial push"
+}
+JSON
+
+# 2. Validate the JSON BEFORE you send it (catches trailing-comma typos)
+jq empty /tmp/dash.json && echo "json ok"
+
+# 3. POST
+RESP=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  "$GRAFANA/api/dashboards/db" -d @/tmp/dash.json)
+echo "$RESP" | jq '{status, uid, url, version}'
+# Expect: status="success", url="/d/demo-svc-v1/...", version=1 (incremented on each push)
+
+# 4. Verify the round-trip — read it back and confirm one panel + the expected title
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" \
+  | jq '{title: .dashboard.title, panels: (.dashboard.panels | length)}'
+# Expect: {"title":"Demo Service","panels":1}
+
+# 5. Open the dashboard in a browser — confirm the panel renders with data.
+```
+
+### 2. Add a `$job` template variable to an existing dashboard
+
+```bash
+# 1. Fetch existing dashboard
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/demo-svc-v1" > /tmp/dash.json
+
+# 2. Edit templating.list — append:
+#   { "name":"job", "type":"query",
+#     "datasource":{"type":"prometheus","uid":"prometheus"},
+#     "query":{"query":"label_values(up, job)","refId":"A"},
+#     "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+#  (Use jq, an editor, or the Grafana UI — schema in references/json-schema.md.)
+
+# 3. Update the panel expr to use the variable: rate(http_requests_total{job=~"$job"}[5m])
+
+# 4. POST it back with overwrite: true. Verify the variable appears in the UI dropdown.
+```
+
+### 3. Compute an "Error %" column with a transformation
+
+```json
+{
+  "id": "calculateField",
+  "options": {
+    "alias": "Error %", "mode": "reduceRow",
+    "reduce": { "reducer": "last" },
+    "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }
+}
+```
+
+Add this to the panel's `transformations: []`. Verify in the UI panel inspector — the new field should appear and update with the variable selection.
+
+Full schema (panels, units, all transformations, annotations, links): [`references/json-schema.md`](references/json-schema.md).
+
+## API reference
+
+```bash
+# Get
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/dashboards/uid/<uid>" | jq '.dashboard'
+
+# Search
+curl -s -H "Authorization: Bearer $TOKEN" \
+  "$GRAFANA/api/search?query=kubernetes&type=dash-db" | jq '.[] | {uid,title,folderTitle}'
+
+# Create folder
+curl -s -X POST -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" "$GRAFANA/api/folders" \
+  -d '{"uid":"platform-team","title":"Platform Team"}'
+```
+
+For dashboards embedded in app plugins, use `@grafana/scenes` (skill `grafana-o11y:grafana-scenes`).
+
+## Resources
+
+- [Dashboard JSON model](https://grafana.com/docs/grafana/latest/dashboards/build-dashboards/view-dashboard-json-model/)
+- [HTTP API — dashboards](https://grafana.com/docs/grafana/latest/developers/http_api/dashboard/)
+- [Panel types](https://grafana.com/docs/grafana/latest/panels-visualizations/)
+- [Variables](https://grafana.com/docs/grafana/latest/dashboards/variables/)
+- [Transformations](https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/)

.agents/skills/dashboarding/references/json-schema.md

@@ -0,0 +1,157 @@
+# Dashboard + panel JSON schema
+
+## Dashboard root
+
+```json
+{
+  "title": "My Dashboard",
+  "uid": "my-dashboard-v1",
+  "tags": ["service", "production"],
+  "time":     { "from": "now-1h", "to": "now" },
+  "refresh":  "30s",
+  "timezone": "browser",
+  "schemaVersion": 41,
+  "templating":  { "list": [] },
+  "annotations": { "list": [] },
+  "panels": []
+}
+```
+
+- `uid` — stable identifier; keep short
+- `schemaVersion` — `41` for Grafana 11+
+- `time.from` / `to` — relative (`now-1h`) or absolute ISO
+- `refresh` — `"30s"`, `"1m"`, `"5m"`, `""` (off)
+
+## Panel
+
+```json
+{
+  "id": 1,
+  "type": "timeseries",
+  "title": "Request Rate",
+  "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+  "datasource": { "type": "prometheus", "uid": "${datasource}" },
+  "targets": [{
+    "expr": "sum(rate(http_requests_total{job=\"$job\"}[5m])) by (status_code)",
+    "legendFormat": "{{status_code}}",
+    "refId": "A"
+  }],
+  "fieldConfig": {
+    "defaults": {
+      "unit": "reqps",
+      "thresholds": { "mode": "absolute", "steps": [
+        { "color": "green",  "value": null },
+        { "color": "yellow", "value": 1000 },
+        { "color": "red",    "value": 5000 }
+      ]}
+    },
+    "overrides": []
+  },
+  "options": {
+    "legend":  { "calcs": ["mean","max","last"], "displayMode": "table", "placement": "bottom" },
+    "tooltip": { "mode": "multi", "sort": "desc" }
+  }
+}
+```
+
+`gridPos`: 24-column grid. Widths: full=24, half=12, third=8, quarter=6. Height 1 unit ≈ 30 px.
+
+## Panel types
+
+| Panel | Use case |
+|---|---|
+| **Time series** | Any metric over time |
+| **Stat** | Single value + sparkline |
+| **Gauge** | % or value against min/max |
+| **Bar gauge** | Side-by-side comparison |
+| **Table** | Multi-column data |
+| **Heatmap** | Distribution over time |
+| **Logs** | Loki log streams |
+| **Traces** | Tempo trace search |
+| **Text** | Markdown docs |
+| **Candlestick** | OHLC / min-max-avg |
+| **Node graph** | Service dependency graph |
+
+## Useful units
+
+```
+reqps        requests/sec
+ops          ops/sec
+Bps          bytes/sec
+percentunit  0.0-1.0 as %
+bytes        bytes (auto-scales)
+decbytes     decimal bytes (1 KB = 1000 B)
+ms / s       milliseconds / seconds
+dtdurationms 1h 2m 3s
+short        compact (1.2k, 3.4M)
+none         raw number
+```
+
+## Template variables
+
+```json
+{ "name":"job", "type":"query", "datasource":{"type":"prometheus","uid":"prometheus"},
+  "query":{"query":"label_values(up, job)","refId":"A"},
+  "refresh":2, "includeAll":true, "multi":true, "label":"Service" }
+
+{ "name":"cluster", "type":"constant", "query":"production", "label":"Cluster" }
+
+{ "name":"datasource", "type":"datasource", "pluginId":"prometheus",
+  "includeAll":false, "label":"Prometheus" }
+```
+
+Multi-value variables expand to a regex OR: `$job=["api","worker"]` → `job=~"api|worker"`.
+Chained: `label_values(kube_pod_info{namespace=\"$namespace\"}, pod)`.
+
+## Transformations
+
+```json
+"transformations": [
+  { "id": "merge", "options": {} },
+  { "id": "organize", "options": {
+      "renameByName":  { "Value #A": "Request Rate", "Value #B": "Error Rate" },
+      "excludeByName": { "Time": true }
+  }},
+  { "id": "calculateField", "options": {
+      "alias": "Error %", "mode": "reduceRow",
+      "reduce": { "reducer": "last" },
+      "binary": { "left": "errors", "right": "total", "operator": "/" }
+  }},
+  { "id": "filterByValue", "options": {
+      "filters": [{ "fieldName":"Error %", "config":{ "id":"greater", "options":{ "value":0.01 }}}],
+      "type": "include", "match": "any"
+  }}
+]
+```
+
+Common IDs: `merge`, `organize`, `rename`, `calculateField`, `filterByValue`, `groupBy`, `sortBy`, `limit`, `labelsToFields`, `seriesToRows`, `partitionByValues`.
+
+## Links & annotations
+
+```json
+"links": [
+  { "title":"Go to details", "url":"/d/details?var-service=${__field.labels.service}", "targetBlank":false },
+  { "title":"Runbook",        "url":"https://wiki.example.com/runbook/${job}", "icon":"external link",
+    "targetBlank":true, "type":"link" }
+]
+```
+
+Built-in vars: `${__value.raw}`, `${__field.labels.job}`, `${__url.params}`, `${__from}` / `${__to}` (Unix ms).
+
+Loki annotation:
+
+```json
+{ "datasource":{"type":"loki","uid":"loki"},
+  "expr":"{job=\"deployments\"} |= \"deployed\"",
+  "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"{{service}} deployed", "textFormat":"{{version}} by {{author}}" }
+```
+
+Prometheus annotation:
+
+```json
+{ "datasource":{"type":"prometheus","uid":"prometheus"},
+  "expr":"changes(kube_deployment_status_observed_generation{namespace=\"production\"}[5m]) > 0",
+  "step":"60s", "name":"Deployments", "iconColor":"blue",
+  "titleFormat":"Deploy: {{deployment}}" }
+```

.agents/skills/grafana-oss/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: grafana-oss
+license: Apache-2.0
+description: Configure Grafana OSS — provisions dashboards from YAML, sets up data sources (Prometheus / Loki / Tempo / Pyroscope), writes dashboard JSON with template variables, builds panel queries, assigns built-in roles (Viewer / Editor / Admin / GrafanaAdmin), mints service-account tokens, edits grafana.ini server config, creates annotations, installs plugins via provisioning, and validates each step with a health-check curl. Use when building dashboards, configuring data sources, setting up provisioning YAML, picking a panel type, writing template variables, managing users and roles, configuring SMTP/OAuth in grafana.ini, creating annotations via API, troubleshooting why a provisioned dashboard isn't showing up, or running Grafana OSS locally — even when the user says "set up a Prometheus data source", "provision dashboards from git", "make a service account", or "configure SSO in OSS" without saying "Grafana OSS".
+---
+
+# Grafana OSS
+
+> **Docs**: https://grafana.com/docs/grafana/latest.md
+
+## Common Workflows
+
+### Provisioning dashboards from disk
+
+1. Drop dashboard JSON file(s) under `/var/lib/grafana/dashboards/`
+2. Add a provider in `provisioning/dashboards/default.yaml` (see [§ Dashboard provisioning](#dashboard-provisioning) below)
+3. Restart Grafana so the provider config is loaded
+4. **Verify the dashboard landed**:
+   ```bash
+   curl https://grafana.example.com/api/dashboards/uid/<uid> \
+     -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+   ```
+   Returns the title → success. 404 → provisioning didn't pick it up; check Grafana server logs (`journalctl -u grafana-server | grep -i provisioning`) for parse errors.
+
+### Provisioning data sources
+
+1. Write `provisioning/datasources/datasources.yaml` (see [§ Data source provisioning](#data-source-provisioning) below)
+2. Restart Grafana
+3. **Health-check the data source via API**:
+   ```bash
+   curl https://grafana.example.com/api/datasources/uid/<uid>/health \
+     -H "Authorization: Bearer <token>"
+   # { "status": "OK", "message": "..." } → working
+   # { "status": "ERROR", ... } → URL unreachable or auth misconfigured
+   ```
+
+### Creating a service account + token
+
+1. Provision via YAML or `POST /api/serviceaccounts` (full API in [references/api.md § Users + service accounts](references/api.md#users--service-accounts))
+2. Mint a token via `POST /api/serviceaccounts/{id}/tokens`
+3. **Verify the token works**:
+   ```bash
+   curl https://grafana.example.com/api/org \
+     -H "Authorization: Bearer <new-token>"
+   # 200 + org JSON → token + role assignment work
+   # 401 → token wrong; 403 → role wrong
+   ```
+
+## Dashboard provisioning
+
+```yaml
+# provisioning/dashboards/default.yaml
+apiVersion: 1
+providers:
+  - name: default
+    folder: MyFolder
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 30
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+For the dashboard JSON shape itself (panels, queries, template variables), see [references/dashboard-json.md](references/dashboard-json.md).
+
+## Data source provisioning
+
+```yaml
+# provisioning/datasources/datasources.yaml
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    jsonData:
+      timeInterval: 15s
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+
+  - name: Tempo
+    type: tempo
+    access: proxy
+    url: http://tempo:3200
+    jsonData:
+      tracesToLogsV2:
+        datasourceUid: loki_uid
+        tags: [{ key: "service.name", value: "app" }]
+      serviceMap:
+        datasourceUid: prometheus_uid
+      nodeGraph:
+        enabled: true
+
+  - name: Pyroscope
+    type: grafana-pyroscope-datasource
+    url: http://pyroscope:4040
+```
+
+## RBAC (built-in roles)
+
+| Role | Permissions |
+|------|-------------|
+| **Viewer** | Read dashboards, alerts |
+| **Editor** | Create/edit dashboards, alerts |
+| **Admin** | Manage data sources, users, plugins |
+| **GrafanaAdmin** | Server-wide admin (superuser) |
+
+Service-account provisioning:
+
+```yaml
+# provisioning/access-control/service_accounts.yaml
+apiVersion: 1
+serviceAccounts:
+  - name: ci-reader
+    orgId: 1
+    role: Viewer
+    tokens:
+      - name: ci-token
+        # expires: optional ISO 8601 timestamp; omit for no-expiry tokens
+```
+
+(Custom RBAC roles with fine-grained permissions are Enterprise / Cloud only — see the `grafana-cloud/admin` skill if you need those.)
+
+## Plugin provisioning
+
+```yaml
+# provisioning/plugins/plugins.yaml
+apiVersion: 1
+apps:
+  - type: grafana-pyroscope-app
+    disabled: false
+    jsonData:
+      backendUrl: http://pyroscope:4040
+```
+
+After restart, verify via `GET /api/plugins/<plugin-id>/health`.
+
+## References
+
+- [`references/dashboard-json.md`](references/dashboard-json.md) — full dashboard JSON model + template variables + common problems (uid uniqueness, gridPos arithmetic, datasource uid matching)
+- [`references/dashboards.md`](references/dashboards.md) — dashboard workflows, settings, variables, annotations, sharing, versions, playlists, and provisioning-as-code
+- [`references/datasources.md`](references/datasources.md) — data source setup and query examples for Prometheus, Loki, Tempo, SQL, CloudWatch, and plugins
+- [`references/panel-types.md`](references/panel-types.md) — panel-type table + decision guide for picking the right one
+- [`references/panels.md`](references/panels.md) — panel editor, visualization options, field config, transformations, query options, inspection, and performance tips
+- [`references/alerting.md`](references/alerting.md) — alerting concepts, contact points, notification policies, templates, silences, and common rule examples
+- [`references/api.md`](references/api.md) — full Grafana OSS API reference (dashboards, data sources, users, service accounts, annotations) with verification curls and common failure modes
+- [`references/config.md`](references/config.md) — `grafana.ini` server / database / SMTP / auth / security / feature-toggle config + restart-required issues

.agents/skills/grafana-oss/references/alerting.md

@@ -0,0 +1,425 @@
+# Grafana Alerting - Detailed Reference
+
+## Overview
+
+Grafana Alerting is a unified alerting system for monitoring metrics and logs across multiple data sources. It fires notifications when conditions are breached.
+
+Key capabilities:
+- Query multiple data sources in a single alert rule
+- Multi-dimensional alerts (one rule creates many alert instances)
+- Flexible notification routing via policies
+- Silences and mute timings for planned maintenance
+- Alert history and state tracking
+
+---
+
+## Core Concepts
+
+### Alert Rule
+Defines what to monitor and when to fire. Contains: queries, a condition (threshold), an evaluation interval, and a pending period.
+
+### Alert Instance
+When a rule produces multi-dimensional data, it creates one alert instance per unique label combination. Example: an alert on `cpu_usage{host=~".*"}` creates one instance per host.
+
+### Alert States
+
+| State | Description |
+|-------|-------------|
+| **Normal** | Query running; condition not met |
+| **Pending** | Condition met but pending period not yet elapsed |
+| **Firing** | Condition met + pending period elapsed; notifications sent |
+| **Resolved** | Previously firing alert returned to normal |
+| **No Data** | Query returned no data (configurable behavior) |
+| **Error** | Query failed with an error (configurable behavior) |
+
+### Evaluation Group
+Alert rules are organized into evaluation groups. All rules in a group share the same evaluation interval and are evaluated sequentially.
+
+### Pending Period
+How long the condition must be continuously met before firing.
+- `0s` = fire immediately
+- `5m` = must be in breach for 5 continuous minutes
+
+### Keep Firing For
+How long an alert continues firing after the condition resolves (prevents brief recovery from clearing the alert).
+
+---
+
+## Alert Rule Types
+
+### Grafana-Managed Rules (Recommended)
+- Stored in Grafana's database
+- Can query any data source
+- Support multi-dimensional alerting
+- Support expressions (math, reduce, threshold)
+
+### Data Source-Managed Rules (Prometheus/Mimir/Loki)
+- Rules stored in the external system
+- Evaluated by the external system
+- Grafana provides UI to manage them
+- Useful when migrating from Prometheus alerting
+
+---
+
+## Creating Grafana-Managed Alert Rules
+
+Navigate to: **Alerting > Alert rules > New alert rule**
+
+### Step 1: Define query and condition
+
+Write one or more queries (labeled A, B, C...).
+
+Example with Prometheus:
+```promql
+# Query A: CPU usage per host
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Expression types you can add after queries:
+- **Math**: `$A > 80` or `($A + $B) / 2`
+- **Reduce**: Collapse time series to single value (Last, Mean, Sum, Max, Min)
+- **Resample**: Change time resolution
+- **Classic conditions**: Multiple threshold conditions with AND/OR
+- **Threshold**: Set the firing threshold
+
+### Step 2: Set evaluation behavior
+
+| Setting | Description |
+|---------|-------------|
+| Folder | Organize rules; folder is the RBAC boundary |
+| Evaluation group | Group name (all rules share this group's interval) |
+| Evaluation interval | How often the rule is evaluated (e.g., 1m) |
+| Pending period | How long condition must hold before firing (e.g., 5m) |
+| Keep firing for | How long alert stays firing after recovery |
+
+### Step 3: Configure labels and notifications
+
+**Labels** - Key-value pairs attached to alert instances. Used for routing and grouping:
+```
+severity=critical
+team=infrastructure
+service=database
+environment=production
+```
+
+**Annotations** - Context included in notification messages:
+```
+Summary: CPU usage above 80% on {{ $labels.instance }}
+Description: CPU usage is {{ $values.A.Value | humanize }}% on {{ $labels.instance }}
+Runbook URL: https://runbooks.company.com/cpu-high
+```
+
+**Notification settings**:
+- Set a **Contact point** to send directly, bypassing the routing policy tree
+- Or leave empty to use the notification policy routing tree
+
+### Step 4: No data and error handling
+
+| Situation | Options |
+|-----------|---------|
+| No data | Alerting, OK, No Data state, Keep last state |
+| Query error | Alerting, OK, Error state, Keep last state |
+
+---
+
+## Contact Points
+
+Contact points are notification destinations.
+
+Navigate to: **Alerting > Contact points**
+
+### Supported integrations
+
+| Integration | Notes |
+|-------------|-------|
+| Email | Requires SMTP config in grafana.ini |
+| Slack | Webhook URL or API token + channel |
+| PagerDuty | Integration key |
+| OpsGenie | API key |
+| VictorOps (Splunk) | API key |
+| Microsoft Teams | Incoming webhook URL |
+| Discord | Webhook URL |
+| Telegram | Bot token + chat ID |
+| Webhook | HTTP POST to custom URL |
+| Alertmanager | Forward to external Alertmanager |
+| Pushover | User key + API token |
+| LINE | LINE Notify token |
+
+### Email configuration in grafana.ini
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@company.com
+password = app-password
+from_address = alerts@company.com
+from_name = Grafana Alerts
+```
+
+### Webhook payload format
+Grafana POSTs a JSON payload to webhook endpoints:
+```json
+{
+  "receiver": "webhook-receiver",
+  "status": "firing",
+  "alerts": [
+    {
+      "status": "firing",
+      "labels": { "alertname": "HighCPU", "instance": "server1" },
+      "annotations": { "summary": "CPU above 80%" },
+      "startsAt": "2024-01-15T10:00:00Z",
+      "generatorURL": "http://grafana/alerting/..."
+    }
+  ],
+  "groupLabels": { "alertname": "HighCPU" },
+  "externalURL": "http://grafana"
+}
+```
+
+---
+
+## Notification Policies
+
+Notification policies route alerts to contact points based on label matchers.
+
+Navigate to: **Alerting > Notification policies**
+
+### Default policy
+The root policy - all alerts reach here if no specific policy matches.
+
+### Child policies
+Add policies that match specific labels:
+```
+Match labels:
+  severity = critical    -> contact: pagerduty
+  team = infrastructure  -> contact: slack-infra
+  environment = staging  -> contact: email-dev
+```
+
+### Policy settings
+
+| Setting | Description |
+|---------|-------------|
+| Contact point | Where to send matching alerts |
+| Continue matching | If true, also evaluate subsequent sibling policies |
+| Group by | Labels used for batching alerts into single notifications |
+| Group wait | Wait before sending first notification for a new group (default: 30s) |
+| Group interval | Wait before sending updates for an existing group (default: 5m) |
+| Repeat interval | Wait before re-sending for still-firing alerts (default: 4h) |
+
+### Grouping
+When multiple alerts have the same "group by" labels, they are batched into a single notification. This prevents notification storms.
+
+Example:
+- 50 hosts alert on HighCPU simultaneously
+- Group by: `[alertname, datacenter]`
+- Result: 1 notification per datacenter containing all affected hosts
+
+---
+
+## Notification Templates
+
+Customize notification message format using Go templating.
+
+Navigate to: **Alerting > Contact points > Notification templates**
+
+### Built-in variables
+```
+{{ $labels }}          # Alert labels as map
+{{ $values }}          # Query values map
+{{ $labels.instance }} # Specific label value
+{{ $values.A.Value }}  # Specific query value
+{{ $status }}          # firing or resolved
+{{ $startsAt }}        # When alert started firing
+```
+
+### Example Slack template
+```
+{{ define "slack_message" }}
+{{ if eq .Status "firing" }}:red_circle:{{ else }}:large_green_circle:{{ end }} *{{ .Labels.alertname }}*
+
+*Status:* {{ .Status }}
+*Severity:* {{ .Labels.severity }}
+
+{{ range .Alerts }}
+*Instance:* {{ .Labels.instance }}
+*Value:* {{ .Values.A.Value | humanize }}
+*Summary:* {{ .Annotations.summary }}
+{{ end }}
+{{ end }}
+```
+
+### Humanize functions
+```
+{{ $value | humanize }}           # "12.3k"
+{{ $value | humanize1024 }}       # "12.3Ki"
+{{ $value | humanizeBytes }}      # "12.3 kB"
+{{ $value | humanizeDuration }}   # "3h 2m 1s"
+{{ $value | humanizePercentage }} # "12.3%"
+```
+
+---
+
+## Silences
+
+Silences temporarily suppress alert notifications without stopping alert evaluation.
+
+Navigate to: **Alerting > Silences**
+
+### Create a silence
+1. Click **Add silence**
+2. Set start time and end time (or duration)
+3. Add label matchers:
+   ```
+   alertname = HighCPU
+   instance =~ ".*staging.*"   # Regex match
+   severity != critical         # Negative match
+   ```
+4. Add a comment explaining why
+5. Save
+
+### Use cases
+- Planned maintenance windows
+- Known issues being investigated
+- Silencing noisy alerts during deploys
+
+From a firing alert detail view: click **Silence** to pre-populate label matchers.
+
+---
+
+## Mute Timings
+
+Recurring schedules when notifications are suppressed (unlike silences which are one-time).
+
+Navigate to: **Alerting > Mute timings**
+
+### Example mute timings
+```
+Name: no-alerts-weekends
+  Weekdays: Saturday, Sunday
+
+Name: business-hours-only
+  Weekdays: Monday-Friday
+  Times: 09:00-17:00
+```
+
+Attach to notification policies in the policy settings.
+
+---
+
+## RBAC for Alerting
+
+Default permissions by role:
+
+| Action | Viewer | Editor | Admin |
+|--------|--------|--------|-------|
+| View alert rules | Yes | Yes | Yes |
+| Create/edit alert rules | No | Yes | Yes |
+| Delete alert rules | No | No | Yes |
+| Manage contact points | No | No | Yes |
+| Manage notification policies | No | No | Yes |
+| Create silences | No | Yes | Yes |
+
+---
+
+## grafana.ini Alerting Configuration
+
+```ini
+[unified_alerting]
+# Enable unified alerting (default: true since Grafana 9)
+enabled = true
+
+# Maximum alert instances a single rule can produce
+max_annotations_to_keep = 100
+
+# Evaluation timeout
+evaluation_timeout = 30s
+
+# Minimum evaluation interval (prevent too-frequent evaluation)
+min_interval = 10s
+
+[smtp]
+# Required for email contact points
+enabled = true
+host = smtp.example.com:587
+```
+
+---
+
+## Common Alert Rule Examples
+
+### CPU usage above threshold
+```promql
+# Query A
+100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+# Threshold: A > 80
+# Pending: 5m
+# Labels: severity=warning
+```
+
+### Memory usage
+```promql
+# Query A
+(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100
+# Threshold: A > 90
+```
+
+### HTTP error rate
+```promql
+# Query A: error requests
+sum(rate(http_requests_total{status=~"5.."}[5m]))
+# Query B: total requests
+sum(rate(http_requests_total[5m]))
+# Expression C (Math): $A / $B * 100
+# Threshold: C > 5
+```
+
+### Disk space
+```promql
+# Query A
+(1 - (node_filesystem_free_bytes{fstype!="tmpfs"} / node_filesystem_size_bytes{fstype!="tmpfs"})) * 100
+# Threshold: A > 85
+```
+
+### Service down (no data = firing)
+```promql
+# Query A
+up{job="my-service"}
+# Threshold: A < 1
+# No data handling: Alerting (treat absence as firing)
+```
+
+### Loki log error rate
+```logql
+# Query A
+sum(rate({job="api"} |= "ERROR" [5m]))
+# Threshold: A > 10
+```
+
+---
+
+## Connecting Alerts to Dashboards
+
+### Create alert from panel
+1. Open panel editor
+2. Click **Alert** tab
+3. Click **Create alert rule from this panel**
+4. Pre-populates query from the panel
+
+### Link alert to dashboard panel
+In the alert rule definition, set **Dashboard** and **Panel** to link to a specific visualization. The alert state badge appears on the panel and clicking it goes to the alert rule.
+
+---
+
+## High Availability (HA) Alerting
+
+```ini
+[unified_alerting]
+ha_peers = grafana-1:9094,grafana-2:9094,grafana-3:9094
+ha_advertise_address = ${POD_IP}:9094
+ha_peer_timeout = 15s
+ha_gossip_interval = 200ms
+ha_push_pull_interval = 60s
+```
+
+Multiple Grafana instances share state to avoid duplicate notifications.

.agents/skills/grafana-oss/references/api.md

@@ -0,0 +1,107 @@
+# Grafana OSS API
+
+Base URL: `https://your-grafana.example.com/api/`. Auth: service account token (`Authorization: Bearer <token>`).
+
+## Contents
+
+- [Dashboards](#dashboards)
+- [Data sources](#data-sources)
+- [Users + service accounts](#users--service-accounts)
+- [Annotations](#annotations)
+
+## Dashboards
+
+```bash
+# Search
+GET /api/search?query=service&type=dash-db&folderIds=1
+
+# Get by UID
+GET /api/dashboards/uid/{uid}
+
+# Create / update (overwrite: true replaces existing)
+POST /api/dashboards/db
+Body: { "dashboard": {...}, "folderUID": "...", "overwrite": true }
+
+# Delete
+DELETE /api/dashboards/uid/{uid}
+```
+
+After provisioning a dashboard via YAML, verify it landed:
+```bash
+curl https://grafana.example.com/api/dashboards/uid/<uid> \
+  -H "Authorization: Bearer <token>" | jq '.dashboard.title'
+# Should print the dashboard's title. 404 = not provisioned correctly.
+```
+
+## Data sources
+
+```bash
+# List
+GET /api/datasources
+
+# Get by UID
+GET /api/datasources/uid/{uid}
+
+# Create
+POST /api/datasources
+Body: { "name": "...", "type": "...", "url": "...", "access": "proxy" }
+
+# Health-check (good post-provision validation)
+GET /api/datasources/uid/{uid}/health
+# Returns { "status": "OK" | "ERROR", "message": "..." }
+```
+
+## Users + service accounts
+
+```bash
+# List org users
+GET /api/org/users
+
+# List service accounts
+GET /api/serviceaccounts/search?perpage=100&page=1
+
+# Create service account
+POST /api/serviceaccounts
+Body: { "name": "ci-reader", "role": "Viewer", "isDisabled": false }
+
+# Mint a token
+POST /api/serviceaccounts/{id}/tokens
+Body: { "name": "ci-token", "secondsToLive": 0 }   # 0 = no expiry
+
+# Verify a token works
+curl https://grafana.example.com/api/org \
+  -H "Authorization: Bearer <new-token>"
+# 200 + org JSON = token + role assignment work.
+```
+
+## Annotations
+
+```bash
+# Create
+curl -X POST https://grafana.example.com/api/annotations \
+  -H 'Authorization: Bearer <token>' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "dashboardUID": "service-overview",
+    "panelId": 1,
+    "time": 1706745600000,
+    "timeEnd": 1706749200000,
+    "tags": ["deploy", "v2.0"],
+    "text": "Deployed v2.0"
+  }'
+
+# Find (by tag)
+GET /api/annotations?tags=deploy&from=1706745600000&to=1706832000000
+
+# Delete
+DELETE /api/annotations/{id}
+```
+
+## Common failure modes
+
+| Symptom | Likely cause |
+|---|---|
+| `401 Unauthorized` | Token expired or wrong stack URL — check the Authorization header |
+| `403 Forbidden` on dashboard create | Service account lacks Editor role on the target folder |
+| `412 Precondition Failed` on POST `/dashboards/db` | UID exists and you didn't set `overwrite: true` |
+| Data source health check `ERROR` | Network unreachable (check `url` field) or credentials wrong |

.agents/skills/grafana-oss/references/config.md

@@ -0,0 +1,100 @@
+# grafana.ini configuration
+
+Reference for `grafana.ini` server-side settings.
+
+## Contents
+
+- [Server + database](#server--database)
+- [Alerting](#alerting)
+- [SMTP](#smtp)
+- [Auth (OAuth example)](#auth-oauth-example)
+- [Security](#security)
+- [Feature toggles](#feature-toggles)
+
+## Server + database
+
+```ini
+[server]
+http_port = 3000
+domain = grafana.example.com
+root_url = https://grafana.example.com/
+
+[database]
+type = postgres
+host = postgres:5432
+name = grafana
+user = grafana
+password = secret
+```
+
+After changing these, restart Grafana and verify:
+- `curl http://localhost:3000/api/health` returns `{"database": "ok", ...}`
+- `curl http://localhost:3000/api/admin/settings | jq '.database'` shows the new config
+
+## Alerting
+
+```ini
+[alerting]
+enabled = true
+
+[unified_alerting]
+enabled = true
+```
+
+## SMTP
+
+```ini
+[smtp]
+enabled = true
+host = smtp.gmail.com:587
+user = alerts@example.com
+password = yourpassword
+from_address = alerts@example.com
+```
+
+Verify by sending a test email via the Alerting → Contact points UI.
+
+## Auth (OAuth example)
+
+```ini
+[auth.generic_oauth]
+enabled = true
+name = Okta
+client_id = your_client_id
+client_secret = your_secret
+auth_url = https://your-org.okta.com/oauth2/v1/authorize
+token_url = https://your-org.okta.com/oauth2/v1/token
+api_url = https://your-org.okta.com/oauth2/v1/userinfo
+scopes = openid profile email groups
+```
+
+For SAML and GitHub OAuth, see the [grafana-cloud/admin skill](../../../grafana-cloud/admin/references/sso.md). The configs are the same in OSS.
+
+## Security
+
+```ini
+[security]
+admin_user = admin
+admin_password = secret
+allow_embedding = true       # required for embedding dashboards in iframes
+```
+
+`admin_password` is only consulted on first startup. To change later, use `grafana-cli admin reset-admin-password <new-password>`.
+
+## Feature toggles
+
+```ini
+[feature_toggles]
+enable = publicDashboards
+```
+
+Multiple toggles are space-separated:
+```ini
+enable = publicDashboards correlations grafanaApiServer
+```
+
+## Common problems
+
+- **`grafana.ini` changes need a restart** — config is read at startup, not live-reloaded
+- **`root_url` matters for OAuth callbacks** — if you set `domain` but not `root_url`, OAuth redirect URIs may not match
+- **Sections are global** — `[server]`, `[database]`, etc. apply at the process level, not per-org

.agents/skills/grafana-oss/references/dashboard-json.md

@@ -0,0 +1,81 @@
+# Dashboard JSON model + template variables
+
+## Minimal dashboard JSON
+
+```json
+{
+  "title": "Service Overview",
+  "uid": "service-overview",
+  "time": { "from": "now-1h", "to": "now" },
+  "refresh": "30s",
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Request Rate",
+      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus" },
+          "expr": "rate(http_requests_total{job=\"$job\"}[5m])",
+          "legendFormat": "{{method}} {{status}}"
+        }
+      ],
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 1000 }
+            ]
+          }
+        }
+      }
+    }
+  ]
+}
+```
+
+## Template variables
+
+```json
+{
+  "templating": {
+    "list": [
+      {
+        "name": "namespace",
+        "type": "query",
+        "datasource": { "type": "prometheus", "uid": "prom" },
+        "definition": "label_values(kube_pod_info, namespace)",
+        "includeAll": true,
+        "multi": true
+      },
+      {
+        "name": "env",
+        "type": "custom",
+        "query": "production,staging,dev",
+        "current": { "value": "production" }
+      },
+      {
+        "name": "interval",
+        "type": "interval",
+        "query": "1m,5m,15m,1h",
+        "auto": true
+      }
+    ]
+  }
+}
+```
+
+Reference variables in queries with `$variable`:
+```promql
+rate(http_requests_total{namespace="$namespace"}[$interval])
+```
+
+## Common problems
+
+- **`uid` must be unique across the org** — if you POST a dashboard with an existing UID and `overwrite: false`, Grafana returns 412 Precondition Failed
+- **`gridPos`** uses a 24-column grid; `w + x` must be ≤ 24
+- **`datasource.uid` in `targets`** must match an existing data source UID; misspell it and panels render as "Datasource not found"
+- **Template variable `query` field** is data-source-specific syntax (PromQL `label_values(...)`, LogQL `{label="…"}`, SQL, etc.)

.agents/skills/grafana-oss/references/dashboards.md

@@ -0,0 +1,215 @@
+# Grafana Dashboards - Detailed Reference
+
+## What is a Dashboard?
+
+A Grafana dashboard is a set of one or more panels organized into rows, providing an at-a-glance view of related information. Dashboards connect to data sources, display data through panels, and support interactive filtering via variables.
+
+## Core Components
+
+- **Panels**: Containers that display visualizations; each combines a query + a visualization type
+- **Rows**: Horizontal groupings of panels; can be collapsed for organization
+- **Data Sources**: Connections to databases, APIs, or services that panels query
+- **Variables**: Dropdown selectors at the top of a dashboard for dynamic filtering
+- **Annotations**: Markers overlaid on graphs to indicate events
+
+---
+
+## Creating Dashboards
+
+### From scratch
+1. Click **Dashboards** in the left sidebar
+2. Click **New > New dashboard**
+3. Click **Add visualization** to add your first panel
+4. Select a data source, write a query, choose visualization type
+5. Click **Apply** to save the panel to the dashboard
+6. Click the save icon (or Ctrl+S) to save the dashboard
+
+### From a template / Import
+- Click **Dashboards > New > Import**
+- Paste a dashboard JSON, enter a Grafana.com dashboard ID, or upload a JSON file
+- Grafana.com hosts thousands of community dashboards (grafana.com/grafana/dashboards)
+
+---
+
+## Dashboard Settings
+
+Access via the gear icon at the top-right of any dashboard.
+
+| Setting | Description |
+|---------|-------------|
+| General | Name, description, tags, folder, editable flag |
+| Annotations | Configure annotation queries that overlay events on panels |
+| Variables | Add/edit template variables (dropdowns at top) |
+| Links | Add links to other dashboards or external URLs |
+| JSON Model | View/edit raw JSON for the entire dashboard |
+| Versions | Browse and restore prior versions |
+| Time options | Default time range, auto-refresh intervals, timezone |
+
+---
+
+## Time Range Controls
+
+- **Time picker** (top-right): Select absolute or relative ranges (Last 6 hours, Last 7 days, etc.)
+- **Auto-refresh**: Set to Off, 5s, 10s, 30s, 1m, 5m, etc.
+- **Zoom**: Click-drag on any time series to zoom in
+
+### Common relative time shortcuts
+```
+now-5m    last 5 minutes
+now-1h    last 1 hour
+now-24h   last 24 hours
+now-7d    last 7 days
+now/d     today so far
+now-1d/d  yesterday
+```
+
+---
+
+## Annotations
+
+Annotations are event markers overlaid on time series panels.
+
+### Types
+- **Native annotations**: Manually add a note to a specific time directly in the dashboard
+- **Query annotations**: Pull events from a data source and display as markers
+
+### Adding a manual annotation
+- Hold Ctrl (or Cmd on Mac) and click a time series panel
+- Type a description; optionally set a time range
+
+---
+
+## Library Panels
+
+Library panels are reusable panel definitions shared across multiple dashboards.
+
+- **Create**: Panel menu (3-dot) > "Create library panel"
+- **Use**: Add a panel > "Add from panel library"
+- **Update**: Edit the source panel; all dashboards using it reflect the change
+- **Unlink**: Detach to make it independent in a specific dashboard
+
+---
+
+## Panel Layout
+
+- Drag panel corners to resize
+- Drag panel header to move
+- **Add > Row** to insert collapsible rows
+- Panel menu > **Duplicate** to clone within the same dashboard
+
+---
+
+## Dashboard Versions
+
+- Access via Dashboard Settings > Versions
+- See who saved what and when
+- Compare two versions (diff view)
+- Restore any previous version
+
+---
+
+## JSON Model
+
+Every dashboard is stored as a JSON document.
+
+Key top-level JSON fields:
+```json
+{
+  "title": "My Dashboard",
+  "uid": "abc123",
+  "tags": ["production", "infrastructure"],
+  "time": { "from": "now-6h", "to": "now" },
+  "refresh": "30s",
+  "panels": [],
+  "templating": { "list": [] },
+  "annotations": { "list": [] }
+}
+```
+
+---
+
+## Sharing Dashboards
+
+### Share link
+- Click **Share** icon > **Link** tab
+- Toggle "Lock time range" to embed the current time window
+- Toggle "Include template variable values"
+
+### Snapshot
+- **Share > Snapshot**: Creates a read-only, public snapshot
+- Contains rendered data at share time - no live data source access needed
+- Can expire (1 hour, 1 day, 7 days, or never)
+
+### Export / Import JSON
+- **Share > Export**: Download the dashboard JSON
+- Import at **Dashboards > New > Import**
+- Useful for version control, migration, sharing with the community
+
+### Embed
+- **Share > Embed**: Generates an iframe HTML snippet
+- Requires anonymous access in Grafana config (OSS/Enterprise only)
+
+### Public dashboards (Grafana 10+)
+- Make a dashboard publicly accessible with no login
+- Enable per-dashboard in Share > Public dashboard
+
+---
+
+## Playlists
+
+Playlists cycle through dashboards automatically at a configurable interval.
+
+- Create at **Dashboards > Playlists > New playlist**
+- Add dashboards by name or tag
+- Set interval (e.g., 5 minutes)
+- Append `?kiosk=1` to URL for kiosk/TV mode
+
+---
+
+## Best Practices
+
+1. **Organize with folders**: Use folders as RBAC permission boundaries
+2. **Use variables**: Make dashboards reusable across environments/services
+3. **Limit panels per dashboard**: Aim for <20 panels for performance
+4. **Use library panels**: Standardize common panels across teams
+5. **Tag dashboards**: Consistent tags for searchability
+6. **Version control**: Export JSON and store in Git
+7. **Use rows**: Collapse related panels to organize complex dashboards
+8. **Template your queries**: Use variables so one dashboard covers many targets
+
+---
+
+## Keyboard Shortcuts
+
+| Key | Action |
+|-----|--------|
+| `Ctrl+S` | Save dashboard |
+| `e` | Open panel editor (when panel focused) |
+| `v` | Toggle panel fullscreen |
+| `d r` | Refresh all panels |
+| `d s` | Dashboard settings |
+| `d k` | Toggle kiosk mode |
+| `?` | Show all shortcuts |
+
+---
+
+## Provisioning Dashboards (as Code)
+
+```yaml
+# /etc/grafana/provisioning/dashboards/default.yaml
+apiVersion: 1
+
+providers:
+  - name: default
+    orgId: 1
+    folder: Infrastructure
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    allowUiUpdates: false
+    options:
+      path: /var/lib/grafana/dashboards
+      foldersFromFilesStructure: true
+```
+
+Place dashboard JSON files in `/var/lib/grafana/dashboards/`. Subdirectories become folders when `foldersFromFilesStructure: true`.

.agents/skills/grafana-oss/references/datasources.md

@@ -0,0 +1,462 @@
+# Grafana Data Sources - Detailed Reference
+
+## Overview
+
+Data sources are connections between Grafana and the systems storing your data. Grafana ships with built-in support for many popular data sources and supports additional sources via plugins.
+
+- Only **Organization Admins** can add or remove data sources
+- Each data source has its own query editor
+- Data sources can be used in: Dashboard panels, Explore, Alert rules, Annotations
+
+---
+
+## Managing Data Sources
+
+### Add a data source
+1. Go to **Connections > Data sources** (or **Configuration > Data sources** in older versions)
+2. Click **Add new data source**
+3. Search for and select the data source type
+4. Fill in connection details (URL, credentials, etc.)
+5. Click **Save & Test** to verify the connection
+
+### Data source settings (common to all types)
+| Setting | Description |
+|---------|-------------|
+| Name | Display name used in dashboards |
+| Default | If checked, pre-selected in new panels |
+| HTTP URL | The endpoint for the data source server |
+| Auth | Basic auth, TLS, bearer token, API key options |
+
+---
+
+## Prometheus
+
+Native support - no plugin required.
+
+### Configuration
+```ini
+URL: http://prometheus:9090
+HTTP method: POST (recommended, supports longer queries)
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Scrape interval | Default: 15s - should match your Prometheus scrape interval |
+| Query timeout | Default: 60s |
+| Exemplars | Enable to link metrics to traces |
+| Ruler URL | For Prometheus-managed alert rules |
+
+### Query editor
+- **Metrics browser**: Browse available metrics with autocomplete
+- **Query type**: Range query (time series) or Instant query (single value)
+- **Legend**: Customize series labels using `{{label_name}}` syntax
+
+### Template variables with Prometheus
+```
+# Variable type: Query
+# Query examples:
+label_values(metric_name, label_name)    # All values of a label for a metric
+label_values(label_name)                  # All values across all metrics
+metrics(prefix)                           # All metric names matching prefix
+query_result(promql_expression)           # PromQL result as variable values
+```
+
+### Exemplars
+When enabled, Prometheus exemplars link high-cardinality trace IDs to metric data points. Requires a Tempo data source for trace drill-through.
+
+---
+
+## Loki (Log Aggregation)
+
+### Configuration
+```ini
+URL: http://loki:3100
+Maximum lines: 1000
+```
+
+### Derived fields
+Extract values from log lines and link to other systems.
+
+Example - extract trace ID and link to Tempo:
+```
+Name: TraceID
+Regex: traceID=(\w+)
+URL: (internal link to Tempo data source)
+```
+
+### Query editor (LogQL)
+```logql
+# Basic log stream selector
+{job="nginx", namespace="production"}
+
+# Filter by text
+{job="nginx"} |= "error"
+{job="nginx"} != "debug"
+
+# Regex filter
+{job="nginx"} |~ "status=5\d\d"
+
+# Parse and filter structured logs
+{job="api"} | json | level="error"
+{job="api"} | logfmt | duration > 1s
+
+# Metrics from logs
+rate({job="nginx"} |= "error" [5m])
+sum(rate({job="nginx"}[5m])) by (status_code)
+```
+
+### Template variables with Loki
+```
+label_names()                          # All label names
+label_values(label_name)               # All values for a label
+label_values({job="nginx"}, pod)       # Label values filtered by stream selector
+```
+
+---
+
+## Tempo (Distributed Tracing)
+
+### Configuration
+```ini
+URL: http://tempo:3200
+```
+
+### Key options
+| Option | Description |
+|--------|-------------|
+| Trace to logs | Link traces to Loki logs via trace ID |
+| Trace to metrics | Link trace spans to Prometheus metrics |
+| Service graph | Enable service dependency graph |
+| Node graph | Show trace as node graph |
+
+### TraceQL (query language)
+```traceql
+# Find traces with error spans
+{status=error}
+
+# Filter by service and duration
+{.service.name="frontend" && duration > 1s}
+
+# Structural queries
+{.http.url=~"/api/.*"} >> {status=error}
+```
+
+---
+
+## Alertmanager
+
+For connecting to an external Alertmanager.
+
+```ini
+URL: http://alertmanager:9093
+# Implementation: Prometheus, Mimir, or Cortex
+```
+
+---
+
+## Elasticsearch / OpenSearch
+
+### Configuration
+```ini
+URL: http://elasticsearch:9200
+Index name: logs-*
+Time field name: @timestamp
+Elasticsearch version: 8.x
+```
+
+---
+
+## MySQL
+
+Built-in support for MySQL 5.7+ and compatible databases (MariaDB, Percona, Amazon Aurora MySQL, Azure Database for MySQL, Google Cloud SQL MySQL).
+
+### Configuration
+```ini
+Host: mysql:3306
+Database: mydb
+User: grafana
+Password: secret
+Max open connections: 100
+Max idle connections: 100
+Connection max lifetime: 14400
+```
+
+### Time series queries
+```sql
+SELECT
+  UNIX_TIMESTAMP(time_col) as time_sec,
+  value_col as value,
+  name_col as metric
+FROM my_table
+WHERE $__timeFilter(time_col)
+ORDER BY time_col ASC
+```
+
+### Macros
+| Macro | Description |
+|-------|-------------|
+| `$__time(column)` | Converts column to Unix timestamp |
+| `$__timeFilter(column)` | Adds WHERE clause for dashboard time range |
+| `$__timeFrom()` | Start of dashboard time range as Unix timestamp |
+| `$__timeTo()` | End of dashboard time range as Unix timestamp |
+| `$__timeGroup(column, interval)` | Groups by time interval |
+| `$__timeGroupAlias(column, interval)` | As above, aliases as "time" |
+| `$__unixEpochFilter(column)` | Time filter for Unix epoch columns |
+| `$__interval` | Auto-calculated interval for current time range |
+
+### Example time series query
+```sql
+SELECT
+  $__timeGroup(created_at, $__interval) AS time,
+  status,
+  count(*) AS cnt
+FROM orders
+WHERE $__timeFilter(created_at)
+GROUP BY 1, 2
+ORDER BY 1
+```
+
+### Annotations
+```sql
+SELECT
+  UNIX_TIMESTAMP(time) AS time,
+  title AS text,
+  tags
+FROM events
+WHERE $__timeFilter(time)
+```
+
+---
+
+## PostgreSQL
+
+### Configuration
+```ini
+Host: postgres:5432
+Database: mydb
+User: grafana
+Password: secret
+SSL mode: disable / require / verify-ca / verify-full
+```
+
+### PostgreSQL time series
+```sql
+SELECT
+  time_bucket('$__interval', time) AS time,
+  avg(value)
+FROM metrics
+WHERE time BETWEEN $__timeFrom() AND $__timeTo()
+GROUP BY 1
+ORDER BY 1
+```
+
+---
+
+## Microsoft SQL Server
+
+### Configuration
+```ini
+Host: sqlserver:1433
+Database: mydb
+User: grafana
+Password: secret
+Encrypt: false / true / disable
+```
+
+---
+
+## InfluxDB
+
+Supports InfluxDB 1.x (InfluxQL) and InfluxDB 2.x / 3.x (Flux).
+
+### InfluxDB 1.x (InfluxQL)
+```ini
+URL: http://influxdb:8086
+Database: telegraf
+```
+
+Query example:
+```sql
+SELECT mean("value") FROM "measurement"
+WHERE $timeFilter
+GROUP BY time($interval), "host"
+```
+
+### InfluxDB 2.x (Flux)
+```ini
+URL: http://influxdb:8086
+Organization: myorg
+Token: <api-token>
+Default bucket: mydata
+```
+
+Flux query example:
+```flux
+from(bucket: "mydata")
+  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
+  |> filter(fn: (r) => r._measurement == "cpu")
+  |> filter(fn: (r) => r._field == "usage_idle")
+  |> aggregateWindow(every: v.windowPeriod, fn: mean)
+```
+
+---
+
+## AWS CloudWatch
+
+### Configuration
+```ini
+# Auth options:
+# - AWS SDK Default (instance role, ~/.aws/credentials, env vars)
+# - Access & secret key (explicit credentials)
+# - Assume role ARN
+
+Default region: us-east-1
+```
+
+Required IAM permissions:
+- `cloudwatch:GetMetricData`
+- `cloudwatch:ListMetrics`
+- `logs:*` (for CloudWatch Logs)
+
+### CloudWatch Logs Insights
+```
+fields @timestamp, @message
+| filter level = "ERROR"
+| sort @timestamp desc
+| limit 100
+```
+
+---
+
+## Azure Monitor
+
+Requires Azure App Registration with:
+- Tenant ID, Client ID, Client Secret
+- `Monitoring Reader` role on target subscriptions/resource groups
+
+---
+
+## Google Cloud Monitoring
+
+Uses Google Cloud service account credentials (JSON key file or workload identity).
+
+---
+
+## TestData (built-in)
+
+A built-in data source for generating test/demo data without a real backend.
+
+Use cases: Testing visualizations, demo dashboards, development without a real data source.
+Scenarios: Random walk, CSV metric values, streaming data, etc.
+
+---
+
+## Graphite
+
+```ini
+URL: http://graphite:8080
+```
+
+Query examples:
+```
+target(servers.*.cpu)
+averageSeries(servers.*.cpu)
+groupByNode(servers.*.cpu, 1, 'averageSeries')
+```
+
+---
+
+## Jaeger (Tracing)
+
+```ini
+URL: http://jaeger:16686
+```
+
+---
+
+## Zipkin (Tracing)
+
+```ini
+URL: http://zipkin:9411
+```
+
+---
+
+## Pyroscope (Continuous Profiling)
+
+```ini
+URL: http://pyroscope:4040
+```
+
+---
+
+## Data Source Permissions (Enterprise)
+
+By default, all org users can query any data source. With RBAC:
+- Assign specific users/teams to specific data sources
+- Configure at **Data source settings > Permissions**
+
+---
+
+## Provisioning Data Sources (as Code)
+
+```yaml
+# /etc/grafana/provisioning/datasources/prometheus.yaml
+apiVersion: 1
+
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    version: 1
+    editable: false
+    jsonData:
+      timeInterval: "15s"
+      queryTimeout: "60s"
+      httpMethod: POST
+
+  - name: Loki
+    type: loki
+    access: proxy
+    url: http://loki:3100
+    jsonData:
+      maxLines: 1000
+      derivedFields:
+        - datasourceUid: tempo
+          matcherRegex: "traceID=(\\w+)"
+          name: TraceID
+          url: "${__value.raw}"
+
+  - name: MySQL Production
+    type: mysql
+    url: mysql:3306
+    database: mydb
+    user: grafana
+    secureJsonData:
+      password: "$GRAFANA_MYSQL_PASSWORD"
+    jsonData:
+      maxOpenConns: 100
+      maxIdleConns: 100
+      connMaxLifetime: 14400
+```
+
+Place YAML files in the provisioning directory; Grafana loads on startup and watches for changes.
+
+---
+
+## Plugin Data Sources
+
+Install additional data sources via:
+- **Connections > Add new connection** (search the plugin catalog)
+- `grafana-cli plugins install <plugin-id>`
+- Docker: set `GF_INSTALL_PLUGINS` environment variable
+
+Popular plugin data sources:
+- `grafana-opensearch-datasource` - OpenSearch
+- `grafana-bigquery-datasource` - Google BigQuery
+- `grafana-mongodb-datasource` - MongoDB (Enterprise)
+- `grafana-splunk-datasource` - Splunk
+- `grafana-datadog-datasource` - Datadog

.agents/skills/grafana-oss/references/panel-types.md

@@ -0,0 +1,29 @@
+# Panel types
+
+| Panel | Use Case |
+|-------|----------|
+| **Time series** | Line/bar charts over time (default for metrics) |
+| **Stat** | Single value with color thresholds |
+| **Gauge** | Radial gauge for current value |
+| **Bar gauge** | Horizontal bars for comparisons |
+| **Table** | Tabular data, sortable columns |
+| **Logs** | Log stream viewer (Loki) |
+| **Traces** | Trace visualization (Tempo) |
+| **Heatmap** | Distribution over time |
+| **Histogram** | Value distribution |
+| **Pie chart** | Part-to-whole ratios |
+| **Geomap** | Geographic data |
+| **Canvas** | Custom SVG-based layouts |
+| **Node graph** | Service/topology graphs |
+| **Flame graph** | CPU/memory profiling |
+| **Text** | Markdown/HTML content |
+| **Alert list** | Show firing alerts |
+
+## Picking a panel type
+
+Default is **Time series** for almost any metric-over-time visualization. Switch from the default only when:
+
+- The user wants **one number, not a series** → Stat / Gauge / Bar gauge
+- The data is **categorical, not temporal** → Pie chart / Bar gauge / Table
+- The data is **logs/traces/profiles** → Logs / Traces / Flame graph (per data type)
+- The data is **2D distribution** (e.g. histogram-over-time) → Heatmap

.agents/skills/grafana-oss/references/panels.md

@@ -0,0 +1,325 @@
+# Grafana Panels and Visualizations - Detailed Reference
+
+## What is a Panel?
+
+A panel is the basic building block of a Grafana dashboard. Each panel combines:
+- A **query** (or multiple queries) to a data source
+- A **visualization type** to display the data
+- **Field configuration** (units, thresholds, color mappings)
+- Optional **transformations** to reshape the data before display
+
+---
+
+## Panel Editor
+
+Open the panel editor by clicking a panel's title > Edit, or clicking **Add visualization** for a new panel.
+
+### Panel Editor Layout
+
+**Top bar**: Back to dashboard, Discard changes, Save dashboard
+
+**Center**: Visualization preview (live update as you configure)
+
+**Toggle**: "Table view" - shows raw query results as a table for debugging
+
+**Right sidebar tabs**:
+1. **Query** - configure data sources and write queries
+2. **Transform** - apply data transformations
+3. **Alert** - create alert rules from this panel
+4. Below tabs: visualization-specific options and field config
+
+### Query Tab
+
+- **Data source selector**: Choose which data source to query
+- **Query editor**: Data-source-specific interface (PromQL for Prometheus, LogQL for Loki, SQL for databases)
+- **Query options**:
+  - Max data points: Limit data points fetched
+  - Min interval: Minimum auto-calculated interval
+  - Interval: Override the auto-calculated interval
+  - Relative time: Override dashboard time range for this panel only
+  - Time shift: Shift the time range (e.g., to compare to last week)
+- **+ Query**: Add multiple queries (labeled A, B, C...)
+- **Expression**: Add server-side math expressions combining query results
+
+---
+
+## Visualization Types
+
+### Time Series (default)
+Best for: Metrics over time, continuous data
+- Renders as lines, points, or bars
+- Supports multiple series
+- Configurable line width, fill, point size
+- Supports thresholds as colored background regions
+- Supports annotations overlay
+- **Graph styles**: Lines, Bars, Points; can mix per series
+- **Stacking**: None, Normal, 100%
+- **Axis**: Left Y, Right Y, hidden
+- Default visualization - supports alerting
+
+### Stat
+Best for: Single important metric (KPI display)
+- Shows one or more values as large text
+- Supports sparkline background
+- Color modes: value, background, none
+- Can show last value, mean, sum, etc.
+- Great for dashboards that need quick status overviews
+
+### Bar Chart
+Best for: Comparing categorical data
+- Horizontal or vertical orientation
+- Grouped or stacked
+- Supports labels on bars
+
+### Gauge
+Best for: Showing a value relative to min/max range
+- Circular gauge with arc
+- Configurable thresholds set colors
+- Shows current value prominently
+
+### Bar Gauge
+Best for: Multiple metrics as horizontal/vertical bars
+- Useful for comparing many items
+- Supports thresholds for color coding
+- Modes: gradient, retro LCD, basic
+
+### Table
+Best for: Tabular data, multi-column metrics
+- Supports sorting by column
+- Column width customization
+- Cell display modes: color text, color background, gradient gauge
+- Pagination for large datasets
+- Can embed sparklines in cells
+
+### Heatmap
+Best for: Distribution over time, histogram-over-time
+- X axis: time; Y axis: buckets; Color: density/value
+- Supports pre-bucketed data (Prometheus histogram) and raw values
+- Tooltip shows exact bucket counts
+
+### Histogram
+Best for: Value distribution analysis
+- Groups values into buckets
+- Can combine multiple series
+- Configurable bucket size
+
+### Pie Chart
+Best for: Proportional data, parts-of-whole
+- Pie or donut style
+- Labels: name, value, percentage
+
+### Logs
+Best for: Log data from Loki, Elasticsearch, etc.
+- Displays raw log lines with timestamp
+- Log level coloring (info/warn/error)
+- Search/filter within results
+- Deduplication and time wrapping
+- Prettify JSON option
+
+### Traces
+Best for: Distributed tracing visualization (Tempo)
+- Renders trace spans as a waterfall/Gantt chart
+- Shows service name, operation, duration
+- Click to drill into trace details
+
+### Flame Graph
+Best for: CPU profiling data (Pyroscope)
+- Visualizes call stacks by CPU time
+- Click to zoom into subtrees
+
+### Node Graph
+Best for: Service dependency maps, network topology
+- Renders nodes and edges
+- Node color/size configurable by metric
+
+### Geomap
+Best for: Geographic data visualization
+- Layers: markers, heatmap, route
+- Multiple base map options (OpenStreetMap, CARTO, etc.)
+- Supports GeoJSON data
+
+### Canvas
+Best for: Custom layouts, process diagrams, status boards
+- Drag-and-drop element placement
+- Elements: text, metric value, rectangle, ellipse, icon, image, connections
+- Dynamic data binding per element
+
+### State Timeline
+Best for: State changes over time (on/off, OK/warn/crit)
+- Horizontal bands showing state duration
+- Each series = one row
+- Color per state value
+
+### Status History
+Best for: Periodic state checks over time
+- Grid: Y=services, X=time buckets
+- Color per state
+
+### XY Chart
+Best for: Correlation between two metrics
+- Scatter plot
+- X and Y axis from different fields
+- Bubble size from a third field
+
+### Candlestick
+Best for: Financial OHLC data
+- Open/High/Low/Close representation
+- Volume bars
+
+### Text
+Best for: Documentation panels, headers
+- Renders Markdown or HTML
+
+### Alert List
+Best for: Dashboard overview of current alert states
+
+### Dashboard List
+Best for: Navigation panels linking to other dashboards
+
+---
+
+## Field Configuration (Standard Options)
+
+Available for most visualizations under the visualization options panel.
+
+| Option | Description |
+|--------|-------------|
+| Unit | Display unit (bytes, seconds, %, requests/sec, etc.) |
+| Min / Max | Override auto-detected min/max for scales |
+| Decimals | Number of decimal places |
+| Display name | Override the series/field name |
+| Color scheme | Fixed, thresholds-based, palette, etc. |
+| No value | Text to display when value is null |
+
+### Thresholds
+Define color-coded boundaries:
+- **Absolute**: Fixed numeric values (e.g., >90 = red, >70 = yellow, else green)
+- **Percentage**: Relative to min/max
+
+Example threshold config:
+```
+Base (default): Green
+70: Yellow (warn)
+90: Red (crit)
+```
+
+### Value Mappings
+Transform raw values into human-readable labels or colors:
+- **Value to text**: e.g., `1` -> "OK", `0` -> "Down"
+- **Range to text**: e.g., `0-50` -> "Low"
+- **Regex to text**: Match patterns
+
+### Data Links
+Create clickable links from panel values:
+- Link to other dashboards with variable values interpolated
+- Link to external systems (e.g., Kibana, PagerDuty)
+- Use `${__value.raw}` and `${__field.name}` in URLs
+
+---
+
+## Field Overrides
+
+Apply specific field options to individual series/columns rather than all data:
+
+1. Click **+ Add field override** in the Overrides section
+2. Choose override target:
+   - Fields with name (exact match)
+   - Fields with name matching regex
+   - Fields with type (number, string, time)
+   - Fields returned by query (A, B, C...)
+3. Add properties to override (unit, color, alias, thresholds, etc.)
+
+Example: In a table with columns `cpu_idle` and `cpu_used`, set `cpu_used` to show as percentage and color by threshold while leaving `cpu_idle` with default styling.
+
+---
+
+## Transformations
+
+Transformations reshape query data before rendering. Apply multiple in sequence.
+
+Access: Panel editor > Transform tab > Add transformation
+
+### Most-Used Transformations
+
+| Transformation | Description |
+|----------------|-------------|
+| **Reduce** | Collapse a time series to a single value (last, mean, sum, max, min) |
+| **Filter by name** | Keep only specific fields/columns |
+| **Filter by value** | Filter rows where a field matches a condition |
+| **Organize fields** | Rename, reorder, or hide fields |
+| **Merge** | Combine multiple query results into one table |
+| **Join by field** | SQL-style join on a common field (e.g., time) |
+| **Group by** | Group rows and aggregate (count, sum, mean, etc.) |
+| **Sort by** | Sort rows by a field |
+| **Limit** | Keep only first N rows |
+| **Add field from calculation** | Add a new column computed from existing columns |
+| **Convert field type** | Change a field's data type |
+| **Rename by regex** | Batch rename fields using regex |
+| **Extract fields** | Parse JSON or regex from a string field |
+| **Labels to fields** | Convert label key/values into separate columns |
+| **Rows to fields** | Pivot: turn row values into column headers |
+| **Prepare time series** | Normalize time series format |
+| **Time series to table** | Convert time series format to table format |
+
+Enable "Debug" toggle on any transformation to see input/output for troubleshooting.
+
+---
+
+## Query Options
+
+### Multiple queries
+Add multiple queries (A, B, C...) to one panel. They are overlaid in the visualization. Use this to compare metrics or show multiple services on one graph.
+
+### Expressions (server-side)
+Server-side expressions allow math across query results:
+- **Math**: `$A + $B`, `$A / $B * 100`
+- **Reduce**: Collapse a series to scalar
+- **Resample**: Change time resolution of a series
+- **Classic conditions**: Threshold logic (used in alerting)
+
+Example - calculate error rate as percentage:
+```
+Query A: total_requests{job="api"}
+Query B: error_requests{job="api"}
+Expression C (Math): $B / $A * 100
+Display C as percentage
+```
+
+### Important query variables
+These variables are available in queries:
+
+| Variable | Description |
+|----------|-------------|
+| `$__interval` | Auto-calculated interval based on time range and resolution |
+| `$__rate_interval` | Interval suitable for rate() functions (at least 4x scrape interval) |
+| `$__from` | Start of the current time range (ms epoch) |
+| `$__to` | End of the current time range (ms epoch) |
+| `$__range` | Duration of current time range (e.g., "6h") |
+| `$__range_s` | Duration in seconds |
+| `$__range_ms` | Duration in milliseconds |
+
+Example PromQL using interval variable:
+```promql
+rate(http_requests_total[${__rate_interval}])
+```
+
+---
+
+## Panel Inspect
+
+Click panel menu (3-dot) > **Inspect** to access:
+- **Data**: Raw table view of the data powering the panel
+- **Stats**: Query performance (time, row count)
+- **JSON**: Panel JSON model
+- **Query**: Equivalent of query inspector showing raw query and response
+
+---
+
+## Performance Tips
+
+1. Limit max data points to avoid over-fetching
+2. Use recording rules in Prometheus for expensive queries
+3. Set longer min intervals for historical dashboards
+4. Use `$__interval` variable in queries to align with time resolution
+5. Use `$__rate_interval` instead of hardcoded intervals for rate() queries
+6. Avoid too many panels on one dashboard (aim for <20)

.agents/skills/marktguru/SKILL.md

@@ -0,0 +1,142 @@
+---
+name: marktguru
+license: Apache-2.0
+description: Query weekly supermarket offers, flyer discounts, and price comparisons across German supermarkets (ALDI Süd, ALDI Nord, Lidl, EDEKA, REWE, Kaufland, PENNY, Netto, dm, ROSSMANN, etc.) via the Marktguru API and local MCP server. Use when the user asks to check supermarket discounts, find deals in brochures/prospekte, compare grocery prices for a postal code (PLZ), or create shopping lists from current promotions without needing fragile PDF OCR.
+---
+
+# Marktguru Supermarket Offers & MCP Server
+
+> Query live promotional flyers and circular offers from German supermarkets and drugstores without brittle PDF scraping or layout OCR.
+
+This skill is inspired by and expands upon the Home Assistant integration pattern (`thorschtn/ha-marktguru`), leveraging Marktguru's backend API (`api.marktguru.de`) to retrieve structured deal data for any German postal code.
+
+---
+
+## Supported Supermarkets & Retailers
+
+Marktguru covers **over 450 retail chains** across Germany. The primary supported chains are:
+
+*   **Discounter**:
+    *   **ALDI SÜD** (`127153`) & **ALDI NORD** (`127139`)
+    *   **Lidl** (`126679`)
+    *   **PENNY** (`126765`)
+    *   **Netto Marken-Discount** (`126735`) & **Netto (mit Hund)** (`126734`)
+    *   **NORMA** (`126745`)
+*   **Supermärkte & Vollsortimenter**:
+    *   **EDEKA** (`127093`) & **E center** (`126699`) + regionale Kaufleute (Simmel, Zurheide, etc.)
+    *   **REWE** (`126802`) & **REWE Center** (`126804`)
+    *   **nahkauf** (`137320`), **HIT** (`126598`), **tegut** (`126866`)
+*   **Großfläche / Hypermarket**:
+    *   **Kaufland** (`126654`)
+    *   **Globus** (`127177`)
+*   **Drogerien & Reform**:
+    *   **dm-drogerie markt** (`127057`)
+    *   **ROSSMANN** (`126811`)
+    *   **Müller Drogeriemarkt** (`126720`)
+
+Full details and ID mappings are documented in [`references/supermarkets.md`](references/supermarkets.md).
+
+---
+
+## Components
+
+The skill provides two primary components:
+
+1.  **CLI & Python Client (`scripts/marktguru_client.py`)**: Zero external dependencies (uses standard library `urllib` and `json`).
+2.  **MCP Server (`scripts/marktguru_mcp.py`)**: Standard JSON-RPC 2.0 stdio server providing AI tools for search, comparison, and catalog retrieval.
+
+---
+
+## 1. CLI Usage
+
+Run directly with standard `python3` (uses default postal code **`93059` Regensburg** if `--zip` is omitted):
+
+### A. Search for Deals
+```bash
+# Search for butter in Regensburg (93059 default) across Aldi, Lidl, and Edeka
+python3 scripts/marktguru_client.py search "Butter" --supermarket "Aldi,Lidl,Edeka"
+
+# Search for coffee with raw JSON output
+python3 scripts/marktguru_client.py search "Kaffee" --json
+```
+
+### B. Price Comparison (Sorted Ascending)
+```bash
+# Compare chocolate prices across all supermarkets in 93059
+python3 scripts/marktguru_client.py compare "Schokolade"
+
+# Compare beer prices between Lidl and Rewe in another postal code (e.g. 80331)
+python3 scripts/marktguru_client.py compare "Bier" --zip 80331 --supermarket "Lidl,Rewe"
+```
+
+### C. List Supported Chains
+```bash
+python3 scripts/marktguru_client.py supermarkets
+```
+
+---
+
+## 2. Model Context Protocol (MCP) Integration
+
+The MCP server exposes three tools to any MCP-compatible AI agent:
+
+*   `search_supermarket_offers(query, zip_code="93059", supermarkets=None, limit=20)`
+*   `compare_supermarket_prices(query, zip_code="93059", supermarkets=None)`
+*   `list_supported_supermarkets()`
+
+### MCP Server Configuration (`mcp_config.json`)
+
+To enable the server in Antigravity or other MCP hosts:
+
+```json
+{
+  "mcpServers": {
+    "marktguru": {
+      "command": "python3",
+      "args": [
+        "/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts/marktguru_mcp.py"
+      ]
+    }
+  }
+}
+```
+
+---
+
+## 3. Python Integration
+
+You can import the module directly in Python scripts or agent sidecars:
+
+```python
+import sys
+sys.path.append("/Users/moritz/src/infrapuzzle/.agents/skills/marktguru/scripts")
+from marktguru_client import search_offers, compare_prices, DEFAULT_ZIP_CODE
+
+# Search deals (defaults to 93059)
+results = search_offers("Kaffee", supermarkets=["Aldi", "Lidl"])
+for offer in results["offers"]:
+    print(f"[{offer['supermarket']}] {offer['product_name'] or offer['description']}: {offer['price']} €")
+
+# Compare prices
+comparison = compare_prices("Milch")
+cheapest = comparison["cheapest_offer"]
+print(f"Cheapest milk: {cheapest['price']} € at {cheapest['supermarket']}")
+```
+
+---
+
+## Data Extracted Per Offer
+
+Each item returned from the Marktguru API includes:
+*   `id`: Internal Marktguru offer ID.
+*   `supermarket`: Retailer name (e.g. "ALDI SÜD", "Lidl", "EDEKA").
+*   `product_name`: Generic product name (e.g. "Butter", "Filterkaffee").
+*   `brand`: Brand name (e.g. "Kerrygold", "Melitta", "Milka").
+*   `description`: Promotional description / pack size (e.g. "je 250g Packung").
+*   `price`: Promotional sale price in Euros (€).
+*   `old_price`: Regular price or crossed-out price (UVP).
+*   `discount_percent`: Calculated percentage savings (e.g. 28%).
+*   `reference_price`: Unit price (e.g. 5.16 € / kg).
+*   `loyalty_app_required`: Boolean flag indicating if a digital store app (Lidl Plus, Edeka App, Penny App) is required to receive the discount.
+*   `valid_from` & `valid_to`: Start and end dates of the promotion.
+*   `image_url`: High-resolution product image hosted on CDN.

.agents/skills/marktguru/references/supermarkets.md

@@ -0,0 +1,94 @@
+# Unterstützte Supermärkte & Händler (Marktguru)
+
+Über die Marktguru-Schnittstelle können Werbeprospekte, Wochenangebote und Rabattaktionen von über 450 Handelsketten und Einzelhändlern in Deutschland abgefragt werden.
+
+---
+
+## 1. Discounter
+
+Discounter machen einen Großteil der wöchentlich wechselnden Prospektaktionen aus. Marktguru unterscheidet sauber zwischen den Ketten:
+
+| Kette | Marktguru ID | Typischer Aktionsrhythmus | App-Vorteile erfasst? |
+| :--- | :--- | :--- | :--- |
+| **ALDI SÜD** | `127153` | Mo. & Do. / Fr. & Sa. | Ja (ALDI App Angebote) |
+| **ALDI NORD** | `127139` | Mo. & Do. / Fr. & Sa. | Ja |
+| **Lidl** | `126679` | Mo.–Sa. / Do.–Sa. / Fr.–Sa. | Ja (`loyalty_app_required` für Lidl Plus) |
+| **PENNY** | `126765` | Mo.–Sa. / Do.–Sa. | Ja (Penny App / Payback) |
+| **Netto Marken-Discount** | `126735` | Mo.–Sa. / Do.–Sa. | Ja (Netto App / DeutschlandCard) |
+| **Netto (mit Hund)** | `126734` | Mo.–Sa. (primär Ost-/Norddeutschland) | Ja |
+| **NORMA** | `126745` | Mo. & Mi. / Fr. & Sa. | Ja |
+
+---
+
+## 2. Vollsortimenter & Supermärkte
+
+Hier gibt es oft filialgenaue oder regionale Prospekte. Die API filtert automatisch anhand der übergebenen Postleitzahl (PLZ):
+
+| Kette | Marktguru ID | Besonderheiten |
+| :--- | :--- | :--- |
+| **EDEKA** | `127093` | Nationale & regionale Angebote; Marktguru aggregiert auch selbstständige Kaufleute (z.B. EDEKA Simmel, Zurheide, Niemerszein). |
+| **E center** | `126699` | Großformatige EDEKA-Märkte mit erweitertem Non-Food- und Frischeangebot. |
+| **REWE** | `126802` | Bundesweite Angebote sowie regionale Aktionen. |
+| **REWE Center** | `126804` | Großflächenmärkte von REWE. |
+| **nahkauf** | `137320` | Nachbarschaftsmärkte der REWE Group. |
+| **HIT** | `126598` | Große Frische-Supermärkte (Dohle Handelsgruppe). |
+| **tegut** | `126866` | Fokus auf Bio, Nachhaltigkeit und Regionales (Hessen, Bayern, Thüringen). |
+
+---
+
+## 3. Großfläche & SB-Warenhäuser (Hypermarkets)
+
+| Kette | Marktguru ID | Besonderheiten |
+| :--- | :--- | :--- |
+| **Kaufland** | `126654` | Riesiges wöchentliches Prospektangebot (Lebensmittel + Non-Food, Kaufland Card). |
+| **Globus** | `127177` | SB-Warenhäuser mit eigener Metzgerei und Bäckerei. |
+
+---
+
+## 4. Drogeriemärkte
+
+Drogerieprospekte enthalten neben Pflege- und Haushaltsartikeln auch viele Bio-Lebensmittel, Snacks, Kaffee und Babynahrung:
+
+| Kette | Marktguru ID | Typische Angebote |
+| :--- | :--- | :--- |
+| **dm-drogerie markt** | `127057` | dmBio, Drogeriewaren, Tiernahrung. |
+| **ROSSMANN** | `126811` | EnerBio, Kaffeeangebote, wöchentliche 10%-Coupons. |
+| **Müller Drogeriemarkt** | `126720` | Drogerie, Parfümerie, Spielwaren, Naturshop. |
+| **budni** | `126944` | Primär im Norden (Metropolregion Hamburg). |
+
+---
+
+## 5. Getränkemärkte & Bio-Fachmärkte
+
+| Kette | Marktguru ID | Sortiment |
+| :--- | :--- | :--- |
+| **Netto Getränke-Discount** | `134009` | Bier, Mineralwasser, Softdrinks. |
+| **Denns BioMarkt / Alnatura** | Diverse | Bio-Vollsortiment, Wochenaktionen. |
+
+---
+
+## 6. Unterstützte Aliase im Client & MCP Server
+
+Wenn im Tool-Aufruf der Parameter `supermarkets` übergeben wird, werden folgende umgangssprachliche Bezeichnungen automatisch auf die korrekten Retailer-IDs gemappt:
+
+```json
+{
+  "aldi": [127153, 127139],       // Findet automatisch ALDI SÜD und ALDI NORD
+  "aldi süd" / "aldi sued": [127153],
+  "aldi nord": [127139],
+  "lidl": [126679],
+  "edeka": [127093, 126699],      // EDEKA und E center
+  "rewe": [126802, 126804],       // REWE und REWE Center
+  "kaufland": [126654],
+  "penny": [126765],
+  "netto": [126735, 126734],      // Netto Marken-Discount und Netto (mit Hund)
+  "norma": [126745],
+  "globus": [127177],
+  "dm": [127057],
+  "rossmann": [126811],
+  "müller" / "mueller": [126720],
+  "hit": [126598],
+  "tegut": [126866],
+  "nahkauf": [137320]
+}
+```

.agents/skills/marktguru/scripts/marktguru_client.py

@@ -0,0 +1,577 @@
+#!/usr/bin/env python3
+"""
+Marktguru API Client (Production Grade)
+Fetches promotional offers and discounts from German supermarkets via the Marktguru API.
+
+Key Design Principles:
+- Zero external dependencies (uses standard library only: urllib, json, os, time, datetime).
+- Resilient: Retries with exponential backoff on transient errors.
+- Low Maintenance: Configurable keys and postal codes via environment variables.
+- Fail-Safe: Caching with stale fallback if upstream API is temporarily unreachable.
+- Defensive: Robust parsing protecting against upstream schema shifts and malformed payloads.
+"""
+
+import os
+import sys
+import time
+import json
+import logging
+import hashlib
+import tempfile
+import urllib.request
+import urllib.parse
+import urllib.error
+from datetime import datetime, timezone
+from typing import List, Dict, Any, Optional, Set, Tuple
+
+# Logger writing to stderr to keep stdout pure for JSON / CLI
+logger = logging.getLogger("marktguru_client")
+if not logger.handlers:
+    handler = logging.StreamHandler(sys.stderr)
+    handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
+    logger.addHandler(handler)
+    logger.setLevel(logging.INFO)
+
+API_BASE = "https://api.marktguru.de/api/v1"
+
+# Environment variable overrides for zero-code configuration changes
+DEFAULT_ZIP_CODE = os.environ.get("MARKTGURU_ZIP_CODE", "93059")
+CLIENT_KEY = os.environ.get("MARKTGURU_CLIENT_KEY", "WU/RH+PMGDi+gkZer3WbMelt6zcYHSTytNB7VpTia90=")
+API_KEY = os.environ.get("MARKTGURU_API_KEY", "8Kk+pmbf7TgJ9nVj2cXeA7P5zBGv8iuutVVMRfOfvNE=")
+CACHE_TTL = int(os.environ.get("MARKTGURU_CACHE_TTL", "1800"))  # 30 minutes
+CACHE_ENABLED = os.environ.get("MARKTGURU_CACHE_ENABLED", "true").lower() in ("true", "1", "yes")
+
+HEADERS = {
+    "x-clientkey": CLIENT_KEY,
+    "x-apikey": API_KEY,
+    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)",
+    "Accept": "application/json",
+}
+
+# Major supermarket chains and their Marktguru retailer IDs
+MAJOR_RETAILERS: Dict[str, Dict[str, Any]] = {
+    "ALDI SÜD": {"id": 127153, "type": "Discounter", "major": True},
+    "ALDI NORD": {"id": 127139, "type": "Discounter", "major": True},
+    "Lidl": {"id": 126679, "type": "Discounter", "major": True},
+    "EDEKA": {"id": 127093, "type": "Supermarkt", "major": True},
+    "E center": {"id": 126699, "type": "Supermarkt", "major": True},
+    "REWE": {"id": 126802, "type": "Supermarkt", "major": True},
+    "REWE Center": {"id": 126804, "type": "Supermarkt", "major": True},
+    "Kaufland": {"id": 126654, "type": "Großfläche/Hypermarkt", "major": True},
+    "PENNY": {"id": 126765, "type": "Discounter", "major": True},
+    "Netto Marken-Discount": {"id": 126735, "type": "Discounter", "major": True},
+    "Netto (mit Hund)": {"id": 126734, "type": "Discounter", "major": False},
+    "NORMA": {"id": 126745, "type": "Discounter", "major": True},
+    "Globus": {"id": 127177, "type": "Großfläche/Hypermarkt", "major": True},
+    "HIT": {"id": 126598, "type": "Supermarkt", "major": False},
+    "tegut": {"id": 126866, "type": "Supermarkt", "major": False},
+    "nahkauf": {"id": 137320, "type": "Supermarkt", "major": False},
+    "dm-drogerie markt": {"id": 127057, "type": "Drogerie", "major": True},
+    "ROSSMANN": {"id": 126811, "type": "Drogerie", "major": True},
+    "Müller Drogeriemarkt": {"id": 126720, "type": "Drogerie", "major": True},
+}
+
+# Aliases mapping user query strings to Marktguru retailer IDs
+RETAILER_ALIASES: Dict[str, List[int]] = {
+    "aldi": [127153, 127139],
+    "aldi süd": [127153],
+    "aldi sued": [127153],
+    "aldi nord": [127139],
+    "lidl": [126679],
+    "edeka": [127093, 126699],
+    "e center": [126699],
+    "ecenter": [126699],
+    "rewe": [126802, 126804],
+    "kaufland": [126654],
+    "penny": [126765],
+    "netto": [126735, 126734],
+    "norma": [126745],
+    "globus": [127177],
+    "hit": [126598],
+    "tegut": [126866],
+    "nahkauf": [137320],
+    "dm": [127057],
+    "rossmann": [126811],
+    "müller": [126720],
+    "mueller": [126720],
+}
+
+
+def _get_cache_dir() -> str:
+    """Returns directory path for local cached responses."""
+    cache_dir = os.path.join(tempfile.gettempdir(), "marktguru_cache")
+    try:
+        os.makedirs(cache_dir, exist_ok=True)
+    except Exception:
+        pass
+    return cache_dir
+
+
+def _read_cache(cache_key: str, max_age_seconds: int = CACHE_TTL) -> Tuple[Optional[Dict[str, Any]], bool]:
+    """
+    Reads cache file. Returns (data, is_fresh).
+    If cache is expired, returns (data, False) so caller can use it as emergency fallback.
+    """
+    if not CACHE_ENABLED:
+        return None, False
+
+    cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
+    if not os.path.isfile(cache_file):
+        return None, False
+
+    try:
+        with open(cache_file, "r", encoding="utf-8") as f:
+            payload = json.load(f)
+        cached_time = payload.get("_cached_at", 0)
+        age = time.time() - cached_time
+        data = payload.get("data")
+        is_fresh = age < max_age_seconds
+        return data, is_fresh
+    except Exception as e:
+        logger.debug(f"Failed to read cache {cache_key}: {e}")
+        return None, False
+
+
+def _write_cache(cache_key: str, data: Dict[str, Any]) -> None:
+    """Writes data to cache file atomically."""
+    if not CACHE_ENABLED:
+        return
+
+    cache_file = os.path.join(_get_cache_dir(), f"{cache_key}.json")
+    temp_file = f"{cache_file}.tmp.{os.getpid()}"
+    try:
+        payload = {
+            "_cached_at": time.time(),
+            "data": data,
+        }
+        with open(temp_file, "w", encoding="utf-8") as f:
+            json.dump(payload, f, ensure_ascii=False)
+        os.replace(temp_file, cache_file)
+    except Exception as e:
+        logger.debug(f"Failed to write cache {cache_key}: {e}")
+        if os.path.exists(temp_file):
+            try:
+                os.remove(temp_file)
+            except Exception:
+                pass
+
+
+def resolve_supermarket_ids(supermarkets: Optional[List[str]]) -> Optional[str]:
+    """Resolves human supermarket names/aliases to comma-separated Marktguru retailer IDs."""
+    if not supermarkets:
+        return None
+
+    # Handle string passed instead of list (e.g. "Aldi, Lidl")
+    if isinstance(supermarkets, str):
+        supermarkets = [s.strip() for s in supermarkets.split(",") if s.strip()]
+
+    resolved_ids: Set[int] = set()
+    for name in supermarkets:
+        if not isinstance(name, str):
+            continue
+        cleaned = name.strip().lower()
+        if not cleaned:
+            continue
+
+        if cleaned in RETAILER_ALIASES:
+            resolved_ids.update(RETAILER_ALIASES[cleaned])
+        elif cleaned.isdigit():
+            resolved_ids.add(int(cleaned))
+        else:
+            found = False
+            for r_name, r_info in MAJOR_RETAILERS.items():
+                if cleaned in r_name.lower():
+                    resolved_ids.add(r_info["id"])
+                    found = True
+            if not found:
+                logger.warning(f"Unknown supermarket name '{name}' skipped.")
+
+    return ",".join(str(i) for i in sorted(resolved_ids)) if resolved_ids else None
+
+
+def format_iso_date(iso_str: Optional[str]) -> str:
+    """Formats an ISO-8601 date string to human-readable DD.MM.YYYY format."""
+    if not iso_str or not isinstance(iso_str, str):
+        return "-"
+    try:
+        dt = datetime.fromisoformat(iso_str.replace("Z", "+00:00"))
+        return dt.strftime("%d.%m.%Y")
+    except Exception:
+        return iso_str[:10] if len(iso_str) >= 10 else iso_str
+
+
+def parse_offer(raw: Any, zip_code: str) -> Optional[Dict[str, Any]]:
+    """
+    Defensively parses and standardizes an offer payload from Marktguru.
+    Protects against null fields, malformed dates, and missing keys.
+    """
+    if not isinstance(raw, dict):
+        return None
+
+    try:
+        offer_id = raw.get("id")
+        price = raw.get("price")
+        if isinstance(price, (int, float)):
+            price = round(float(price), 2)
+        else:
+            price = None
+
+        old_price = raw.get("oldPrice")
+        if isinstance(old_price, (int, float)):
+            old_price = round(float(old_price), 2)
+        else:
+            old_price = None
+
+        ref_price = raw.get("referencePrice")
+        if isinstance(ref_price, (int, float)):
+            ref_price = round(float(ref_price), 2)
+        else:
+            ref_price = None
+
+        discount_pct = None
+        if old_price and price and old_price > price:
+            discount_pct = round(((old_price - price) / old_price) * 100)
+
+        # Validity dates
+        validity = raw.get("validityDates")
+        valid_from_raw = None
+        valid_to_raw = None
+        is_valid_now = True
+        days_remaining = None
+
+        if isinstance(validity, list) and len(validity) > 0 and isinstance(validity[0], dict):
+            valid_from_raw = validity[0].get("from")
+            valid_to_raw = validity[0].get("to")
+
+            try:
+                now = datetime.now(timezone.utc)
+                if valid_to_raw and isinstance(valid_to_raw, str):
+                    dt_to = datetime.fromisoformat(valid_to_raw.replace("Z", "+00:00"))
+                    diff = (dt_to - now).days
+                    days_remaining = max(0, diff)
+                    if dt_to < now:
+                        is_valid_now = False
+
+                if valid_from_raw and isinstance(valid_from_raw, str):
+                    dt_from = datetime.fromisoformat(valid_from_raw.replace("Z", "+00:00"))
+                    if dt_from > now:
+                        is_valid_now = False
+            except Exception:
+                pass
+
+        # Supermarket / advertiser extraction
+        supermarket = "Unbekannt"
+        supermarket_id = None
+        advertisers = raw.get("advertisers")
+        if isinstance(advertisers, list) and len(advertisers) > 0 and isinstance(advertisers[0], dict):
+            supermarket = advertisers[0].get("name") or "Unbekannt"
+            supermarket_id = advertisers[0].get("id")
+
+        # High resolution CDN image
+        image_url = None
+        if offer_id:
+            image_url = f"https://mg2de.b-cdn.net/api/v1/offers/{offer_id}/images/default/0/medium.webp"
+
+        # Unit & volume
+        unit_info = raw.get("unit")
+        unit_name = None
+        if isinstance(unit_info, dict):
+            unit_name = unit_info.get("shortName") or unit_info.get("name")
+
+        product_info = raw.get("product")
+        product_name = None
+        if isinstance(product_info, dict):
+            product_name = product_info.get("name")
+
+        brand_info = raw.get("brand")
+        brand_name = None
+        if isinstance(brand_info, dict):
+            brand_name = brand_info.get("name")
+
+        return {
+            "id": offer_id,
+            "supermarket": supermarket,
+            "supermarket_id": supermarket_id,
+            "product_name": product_name,
+            "brand": brand_name,
+            "description": raw.get("description"),
+            "price": price,
+            "old_price": old_price,
+            "reference_price": ref_price,
+            "discount_percent": discount_pct,
+            "unit": unit_name,
+            "volume": raw.get("volume"),
+            "loyalty_app_required": bool(raw.get("requiresLoyalityMembership", False)),
+            "is_valid_now": is_valid_now,
+            "days_remaining": days_remaining,
+            "valid_from": format_iso_date(valid_from_raw),
+            "valid_to": format_iso_date(valid_to_raw),
+            "image_url": image_url,
+            "zip_code": str(zip_code),
+        }
+    except Exception as e:
+        logger.debug(f"Error parsing single offer {raw.get('id') if isinstance(raw, dict) else raw}: {e}")
+        return None
+
+
+def _fetch_with_retry(url: str, max_retries: int = 2) -> Dict[str, Any]:
+    """
+    Executes HTTP request with exponential backoff on transient errors (5xx / network blips).
+    """
+    last_err: Optional[Exception] = None
+
+    for attempt in range(max_retries + 1):
+        req = urllib.request.Request(url, headers=HEADERS)
+        try:
+            with urllib.request.urlopen(req, timeout=10) as resp:
+                raw_bytes = resp.read()
+                return json.loads(raw_bytes.decode("utf-8"))
+        except urllib.error.HTTPError as e:
+            last_err = e
+            # Retry only on transient server errors (500, 502, 503, 504) or rate limits (429)
+            if e.code in (429, 500, 502, 503, 504) and attempt < max_retries:
+                backoff = 0.5 * (2 ** attempt)
+                logger.warning(f"Marktguru HTTP {e.code}. Retrying in {backoff:.1f}s (attempt {attempt+1}/{max_retries})...")
+                time.sleep(backoff)
+                continue
+            raise
+        except (urllib.error.URLError, TimeoutError, OSError) as e:
+            last_err = e
+            if attempt < max_retries:
+                backoff = 0.5 * (2 ** attempt)
+                logger.warning(f"Connection issue ({e}). Retrying in {backoff:.1f}s...")
+                time.sleep(backoff)
+                continue
+            raise
+
+    raise last_err or RuntimeError("Max retries exceeded")
+
+
+def search_offers(
+    query: str,
+    zip_code: str = DEFAULT_ZIP_CODE,
+    supermarkets: Optional[List[str]] = None,
+    limit: int = 24,
+    offset: int = 0,
+    force_refresh: bool = False,
+) -> Dict[str, Any]:
+    """
+    Searches current supermarket offers in Germany via the Marktguru API.
+    
+    Args:
+        query: Search term (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch')
+        zip_code: German postal code (default: '93059' Regensburg)
+        supermarkets: List of supermarket names or aliases (e.g. ['Aldi', 'Lidl', 'Edeka'])
+        limit: Max number of items to return (default: 24, max: 50)
+        offset: Result pagination offset
+        force_refresh: If True, bypasses local TTL cache
+    """
+    cleaned_query = (query or "").strip()
+    cleaned_zip = str(zip_code or DEFAULT_ZIP_CODE).strip()
+    limit = max(1, min(int(limit), 50))
+    offset = max(0, int(offset))
+
+    retailer_ids = resolve_supermarket_ids(supermarkets)
+
+    # Compute deterministic cache key
+    cache_raw = f"{cleaned_query.lower()}:{cleaned_zip}:{retailer_ids or ''}:{limit}:{offset}"
+    cache_key = hashlib.sha256(cache_raw.encode("utf-8")).hexdigest()[:16]
+
+    if not force_refresh:
+        cached_data, is_fresh = _read_cache(cache_key)
+        if is_fresh and cached_data:
+            return cached_data
+
+    params: Dict[str, Any] = {
+        "as": "web",
+        "limit": limit,
+        "offset": offset,
+        "q": cleaned_query,
+        "zipCode": cleaned_zip,
+    }
+    if retailer_ids:
+        params["retailers"] = retailer_ids
+
+    url = f"{API_BASE}/offers/search?{urllib.parse.urlencode(params)}"
+
+    try:
+        data = _fetch_with_retry(url)
+    except Exception as e:
+        logger.error(f"Marktguru API query failed: {e}")
+        # Emergency fail-safe: Check if stale cache is available
+        cached_data, _ = _read_cache(cache_key, max_age_seconds=86400 * 7)
+        if cached_data:
+            logger.warning("Serving stale cached data due to API failure.")
+            cached_data["warning"] = "Stale cache returned due to temporary upstream API issue."
+            return cached_data
+
+        return {
+            "error": f"API request failed: {str(e)}",
+            "query": cleaned_query,
+            "zip_code": cleaned_zip,
+            "total_results": 0,
+            "returned_count": 0,
+            "offers": [],
+            "available_retailers": [],
+        }
+
+    raw_results = data.get("results") or []
+    parsed_offers: List[Dict[str, Any]] = []
+    for item in raw_results:
+        parsed = parse_offer(item, cleaned_zip)
+        if parsed is not None:
+            parsed_offers.append(parsed)
+
+    filters = data.get("filters") or {}
+    raw_retailers = filters.get("retailers") or []
+    available_retailers = [
+        {"name": r.get("name"), "id": r.get("id"), "count": r.get("resultsCount")}
+        for r in raw_retailers if isinstance(r, dict) and r.get("name")
+    ]
+
+    result = {
+        "query": cleaned_query,
+        "zip_code": cleaned_zip,
+        "total_results": data.get("totalResults", len(parsed_offers)),
+        "returned_count": len(parsed_offers),
+        "available_retailers": available_retailers,
+        "offers": parsed_offers,
+    }
+
+    _write_cache(cache_key, result)
+    return result
+
+
+def compare_prices(
+    query: str,
+    zip_code: str = DEFAULT_ZIP_CODE,
+    supermarkets: Optional[List[str]] = None,
+    force_refresh: bool = False,
+) -> Dict[str, Any]:
+    """
+    Compares prices for a given product query across supermarkets, sorted from cheapest to most expensive.
+    """
+    data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=50, force_refresh=force_refresh)
+    offers = data.get("offers", [])
+
+    # Filter out items without price and sort ascending
+    priced_offers = [o for o in offers if o.get("price") is not None]
+    priced_offers.sort(key=lambda x: x["price"])
+
+    return {
+        "query": query,
+        "zip_code": str(zip_code),
+        "total_compared": len(priced_offers),
+        "cheapest_offer": priced_offers[0] if priced_offers else None,
+        "all_offers_sorted": priced_offers,
+    }
+
+
+def list_supported_supermarkets() -> Dict[str, Any]:
+    """Returns a structured catalog of all major supermarket chains supported by Marktguru."""
+    categories: Dict[str, List[Dict[str, Any]]] = {
+        "Discounter": [],
+        "Supermarkt": [],
+        "Großfläche / Hypermarkt": [],
+        "Drogerie": [],
+    }
+
+    for name, info in MAJOR_RETAILERS.items():
+        cat = info.get("type", "Sonstige")
+        if cat not in categories:
+            categories[cat] = []
+        categories[cat].append({
+            "name": name,
+            "id": info["id"],
+            "major": info.get("major", False)
+        })
+
+    return {
+        "default_zip_code": DEFAULT_ZIP_CODE,
+        "cache_ttl_seconds": CACHE_TTL,
+        "supported_chains_count": len(MAJOR_RETAILERS),
+        "categories": categories,
+        "supported_aliases": list(RETAILER_ALIASES.keys()),
+    }
+
+
+def main():
+    import argparse
+    parser = argparse.ArgumentParser(description="Query German supermarket flyers/discounts via Marktguru")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    # Search
+    search_p = subparsers.add_parser("search", help="Search offers for a product")
+    search_p.add_argument("query", help="Product to search (e.g. Butter, Kaffee)")
+    search_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
+    search_p.add_argument("--supermarket", help="Comma-separated supermarkets (e.g. 'Aldi,Lidl,Edeka')")
+    search_p.add_argument("--limit", type=int, default=15, help="Max results (default 15)")
+    search_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
+    search_p.add_argument("--json", action="store_true", help="Output raw JSON")
+
+    # Compare
+    comp_p = subparsers.add_parser("compare", help="Compare prices across supermarkets")
+    comp_p.add_argument("query", help="Product to compare")
+    comp_p.add_argument("--zip", default=DEFAULT_ZIP_CODE, help=f"Postal code (default: {DEFAULT_ZIP_CODE})")
+    comp_p.add_argument("--supermarket", help="Comma-separated supermarkets")
+    comp_p.add_argument("--refresh", action="store_true", help="Bypass local cache")
+    comp_p.add_argument("--json", action="store_true", help="Output raw JSON")
+
+    # Supermarkets
+    subparsers.add_parser("supermarkets", help="List all supported supermarkets and chains")
+
+    args = parser.parse_args()
+
+    if args.command == "supermarkets":
+        data = list_supported_supermarkets()
+        print(json.dumps(data, indent=2, ensure_ascii=False))
+        return
+
+    sm_list = [s.strip() for s in args.supermarket.split(",")] if getattr(args, "supermarket", None) else None
+
+    if args.command == "search":
+        result = search_offers(
+            args.query,
+            zip_code=args.zip,
+            supermarkets=sm_list,
+            limit=args.limit,
+            force_refresh=args.refresh
+        )
+        if args.json:
+            print(json.dumps(result, indent=2, ensure_ascii=False))
+        else:
+            if "error" in result and not result.get("offers"):
+                print(f"❌ Fehler: {result['error']}")
+                return
+
+            print(f"\n🔍 Angebote für '{args.query}' in PLZ {args.zip} (Gefunden: {result['total_results']}):\n")
+            for o in result["offers"]:
+                disc_str = f" (-{o['discount_percent']}%)" if o.get("discount_percent") else ""
+                loyalty_str = " 📱 [App Coupon]" if o.get("loyalty_app_required") else ""
+                timing_str = f"noch {o['days_remaining']} Tage" if o.get("days_remaining") is not None else f"bis {o['valid_to']}"
+                print(f" • [{o['supermarket']}] {o['description'] or o['product_name'] or 'Angebot'}")
+                print(f"   Preis: {o['price']:.2f} €{disc_str}{loyalty_str} | Gültig: {timing_str}")
+                if o.get("reference_price"):
+                    print(f"   Grundpreis: {o['reference_price']:.2f} € / {o.get('unit') or 'Einheit'}")
+                print()
+
+    elif args.command == "compare":
+        result = compare_prices(
+            args.query,
+            zip_code=args.zip,
+            supermarkets=sm_list,
+            force_refresh=args.refresh
+        )
+        if args.json:
+            print(json.dumps(result, indent=2, ensure_ascii=False))
+        else:
+            print(f"\n🏷️ Preisvergleich für '{args.query}' in PLZ {args.zip}:\n")
+            for idx, o in enumerate(result["all_offers_sorted"], 1):
+                disc_str = f" (UVP {o['old_price']:.2f}€)" if o.get("old_price") else ""
+                loyalty_str = " [App erforderlich]" if o.get("loyalty_app_required") else ""
+                print(f" {idx:2d}. {o['price']:.2f} € - [{o['supermarket']}] {o['description'] or o['product_name']}{disc_str}{loyalty_str}")
+            print()
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/marktguru/scripts/marktguru_mcp.py

@@ -0,0 +1,319 @@
+#!/usr/bin/env python3
+"""
+Marktguru MCP (Model Context Protocol) Server (Production Grade)
+Exposes German supermarket flyers, discounts, and price comparison tools over standard JSON-RPC 2.0 stdio.
+
+Key Design Principles:
+- Zero external dependencies: Uses standard library only.
+- Strict Protocol Compliance: Conforms to MCP (2024-11-05) and JSON-RPC 2.0 specifications.
+- Defensive Argument Coercion: Tolerates string/number variations commonly output by LLMs.
+- Graceful Signal Handling: Clean exit on SIGINT, SIGTERM, and EOF.
+- Stream Isolation: All logs and diagnostics route to stderr, ensuring stdout is 100% clean JSON-RPC.
+"""
+
+import sys
+import json
+import signal
+import logging
+from typing import Dict, Any, List, Optional
+
+# Import business logic from hardened client script
+from marktguru_client import (
+    search_offers,
+    compare_prices,
+    list_supported_supermarkets,
+    DEFAULT_ZIP_CODE,
+    MAJOR_RETAILERS,
+    RETAILER_ALIASES,
+)
+
+# Configure logging strictly to stderr
+logging.basicConfig(
+    stream=sys.stderr,
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
+)
+logger = logging.getLogger("marktguru-mcp")
+
+PROTOCOL_VERSION = "2024-11-05"
+SERVER_NAME = "marktguru-mcp-server"
+SERVER_VERSION = "1.1.0"
+
+TOOLS = [
+    {
+        "name": "search_supermarket_offers",
+        "description": (
+            "Search current weekly supermarket offers and promotional discounts in Germany via the Marktguru API. "
+            "Supports filtering by postal code (PLZ) and specific supermarket chains (e.g. Aldi Süd, Aldi Nord, Lidl, Edeka, Rewe, Kaufland, Penny, Netto)."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "The product or keyword to search for (e.g. 'Butter', 'Kaffee', 'Bier', 'Milch', 'Nutella', 'Schokolade')."
+                },
+                "zip_code": {
+                    "type": "string",
+                    "description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
+                },
+                "supermarkets": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Optional list of supermarket names to filter by (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe', 'Kaufland', 'Penny', 'Netto', 'dm', 'Rossmann']). If omitted, all available retailers are included."
+                },
+                "limit": {
+                    "type": "integer",
+                    "description": "Maximum number of offers to return (default: 20, max: 50)."
+                }
+            },
+            "required": ["query"]
+        }
+    },
+    {
+        "name": "compare_supermarket_prices",
+        "description": (
+            "Compare current prices for a product across German supermarkets in a given postal code. "
+            "Returns results sorted from lowest to highest price, highlighting the cheapest deal, regular prices (UVP), and required loyalty app memberships."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "query": {
+                    "type": "string",
+                    "description": "The product or grocery item to compare (e.g. 'Kaffee', 'Butter', 'Bier', 'Olivenöl')."
+                },
+                "zip_code": {
+                    "type": "string",
+                    "description": f"5-digit German postal code (PLZ). Defaults to '{DEFAULT_ZIP_CODE}' (Regensburg)."
+                },
+                "supermarkets": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "description": "Optional list of supermarket names to compare (e.g. ['Aldi', 'Lidl', 'Edeka', 'Rewe'])."
+                }
+            },
+            "required": ["query"]
+        }
+    },
+    {
+        "name": "list_supported_supermarkets",
+        "description": (
+            "List all supermarket, discounter, hypermarket, and drugstore chains supported by the Marktguru service in Germany, "
+            "including their categories (Discounter, Supermarkt, Großfläche, Drogerie) and accepted name aliases."
+        ),
+        "inputSchema": {
+            "type": "object",
+            "properties": {}
+        }
+    }
+]
+
+
+def _sanitize_supermarkets(sm_input: Any) -> Optional[List[str]]:
+    """Coerces various LLM input formats (list, comma-string, single string) into a clean list."""
+    if not sm_input:
+        return None
+    if isinstance(sm_input, list):
+        return [str(s).strip() for s in sm_input if str(s).strip()]
+    if isinstance(sm_input, str):
+        return [s.strip() for s in sm_input.split(",") if s.strip()]
+    return None
+
+
+def handle_initialize(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
+    return {
+        "jsonrpc": "2.0",
+        "id": req_id,
+        "result": {
+            "protocolVersion": PROTOCOL_VERSION,
+            "capabilities": {
+                "tools": {
+                    "listChanged": False
+                }
+            },
+            "serverInfo": {
+                "name": SERVER_NAME,
+                "version": SERVER_VERSION
+            }
+        }
+    }
+
+
+def handle_tools_list(req_id: Any) -> Dict[str, Any]:
+    return {
+        "jsonrpc": "2.0",
+        "id": req_id,
+        "result": {
+            "tools": TOOLS
+        }
+    }
+
+
+def handle_tools_call(req_id: Any, params: Dict[str, Any]) -> Dict[str, Any]:
+    tool_name = params.get("name")
+    args = params.get("arguments", {})
+
+    logger.info(f"Tool call requested: {tool_name} with args={args}")
+
+    try:
+        if tool_name == "search_supermarket_offers":
+            query = str(args.get("query") or "").strip()
+            if not query:
+                return {
+                    "jsonrpc": "2.0",
+                    "id": req_id,
+                    "result": {
+                        "content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
+                        "isError": True
+                    }
+                }
+
+            zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
+            supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
+            try:
+                limit = int(args.get("limit") or 20)
+            except (ValueError, TypeError):
+                limit = 20
+
+            data = search_offers(query, zip_code=zip_code, supermarkets=supermarkets, limit=limit)
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        elif tool_name == "compare_supermarket_prices":
+            query = str(args.get("query") or "").strip()
+            if not query:
+                return {
+                    "jsonrpc": "2.0",
+                    "id": req_id,
+                    "result": {
+                        "content": [{"type": "text", "text": "Error: 'query' argument is required and cannot be empty."}],
+                        "isError": True
+                    }
+                }
+
+            zip_code = str(args.get("zip_code") or DEFAULT_ZIP_CODE).strip()
+            supermarkets = _sanitize_supermarkets(args.get("supermarkets"))
+
+            data = compare_prices(query, zip_code=zip_code, supermarkets=supermarkets)
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        elif tool_name == "list_supported_supermarkets":
+            data = list_supported_supermarkets()
+            text_result = json.dumps(data, indent=2, ensure_ascii=False)
+
+        else:
+            return {
+                "jsonrpc": "2.0",
+                "id": req_id,
+                "error": {
+                    "code": -32601,
+                    "message": f"Method or tool '{tool_name}' not found."
+                }
+            }
+
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "result": {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": text_result
+                    }
+                ],
+                "isError": False
+            }
+        }
+
+    except Exception as e:
+        logger.exception(f"Exception during tool execution '{tool_name}': {e}")
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "result": {
+                "content": [
+                    {
+                        "type": "text",
+                        "text": f"Tool execution failed: {type(e).__name__}: {str(e)}"
+                    }
+                ],
+                "isError": True
+            }
+        }
+
+
+def process_message(line: str) -> Optional[Dict[str, Any]]:
+    if not line.strip():
+        return None
+
+    try:
+        msg = json.loads(line)
+    except json.JSONDecodeError as e:
+        logger.error(f"Malformed JSON-RPC received: {e}")
+        return {
+            "jsonrpc": "2.0",
+            "id": None,
+            "error": {"code": -32700, "message": f"Parse error: {str(e)}"}
+        }
+
+    method = msg.get("method")
+    req_id = msg.get("id")
+    params = msg.get("params", {})
+
+    # Notifications do not return a response
+    if req_id is None:
+        if method == "notifications/initialized":
+            logger.info("Client acknowledged initialized notification.")
+        elif method == "notifications/cancelled":
+            logger.info(f"Client cancelled operation: {params}")
+        return None
+
+    # Standard Requests
+    if method == "initialize":
+        return handle_initialize(req_id, params)
+    elif method == "tools/list":
+        return handle_tools_list(req_id)
+    elif method == "tools/call":
+        return handle_tools_call(req_id, params)
+    elif method == "ping":
+        return {"jsonrpc": "2.0", "id": req_id, "result": {}}
+    else:
+        return {
+            "jsonrpc": "2.0",
+            "id": req_id,
+            "error": {
+                "code": -32601,
+                "message": f"Method '{method}' is not supported."
+            }
+        }
+
+
+def _sig_handler(sig, frame):
+    logger.info(f"Received termination signal {sig}. Exiting gracefully.")
+    sys.exit(0)
+
+
+def main():
+    signal.signal(signal.SIGINT, _sig_handler)
+    signal.signal(signal.SIGTERM, _sig_handler)
+
+    logger.info(f"Starting {SERVER_NAME} v{SERVER_VERSION} (stdio mode)...")
+
+    try:
+        for raw_line in sys.stdin:
+            response = process_message(raw_line)
+            if response is not None:
+                output = json.dumps(response, ensure_ascii=False) + "\n"
+                sys.stdout.write(output)
+                sys.stdout.flush()
+    except (KeyboardInterrupt, BrokenPipeError):
+        pass
+    except Exception as e:
+        logger.exception(f"Fatal error in stdio loop: {e}")
+        sys.exit(1)
+
+    logger.info("Marktguru MCP server shutdown complete.")
+
+
+if __name__ == "__main__":
+    main()

.agents/skills/marktguru/tests/test_marktguru.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+"""
+Unit tests for the Marktguru skill & MCP Server.
+Runs completely offline using standard library unittest.
+"""
+
+import sys
+import os
+import json
+import unittest
+from datetime import datetime, timezone, timedelta
+
+# Add scripts directory to path
+SCRIPTS_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "scripts"))
+if SCRIPTS_DIR not in sys.path:
+    sys.path.insert(0, SCRIPTS_DIR)
+
+from marktguru_client import (
+    resolve_supermarket_ids,
+    format_iso_date,
+    parse_offer,
+    list_supported_supermarkets,
+    DEFAULT_ZIP_CODE,
+    _read_cache,
+    _write_cache,
+    _get_cache_dir,
+)
+from marktguru_mcp import process_message, SERVER_NAME, PROTOCOL_VERSION
+
+
+class TestMarktguruClient(unittest.TestCase):
+    def test_default_zip_code(self):
+        self.assertEqual(DEFAULT_ZIP_CODE, "93059")
+
+    def test_resolve_supermarket_ids_aliases(self):
+        # Aldi maps to Aldi Nord and Aldi Süd
+        aldi_ids = resolve_supermarket_ids(["aldi"])
+        self.assertIn("127153", aldi_ids)
+        self.assertIn("127139", aldi_ids)
+
+        # Edeka maps to Edeka and E center
+        edeka_ids = resolve_supermarket_ids(["edeka"])
+        self.assertIn("127093", edeka_ids)
+        self.assertIn("126699", edeka_ids)
+
+        # Lidl
+        lidl_ids = resolve_supermarket_ids(["lidl"])
+        self.assertEqual(lidl_ids, "126679")
+
+        # Comma-separated string support
+        combined = resolve_supermarket_ids("Lidl, Kaufland")
+        self.assertIn("126679", combined)
+        self.assertIn("126654", combined)
+
+        # Unknown supermarket is ignored gracefully
+        unknown = resolve_supermarket_ids(["NonExistentMarket123"])
+        self.assertIsNone(unknown)
+
+    def test_format_iso_date(self):
+        self.assertEqual(format_iso_date("2026-10-15T12:00:00Z"), "15.10.2026")
+        self.assertEqual(format_iso_date(None), "-")
+        self.assertEqual(format_iso_date(""), "-")
+        self.assertEqual(format_iso_date("invalid-date-format"), "invalid-da")
+
+    def test_parse_offer_defensive(self):
+        # 1. Complete payload
+        now = datetime.now(timezone.utc)
+        from_date = (now - timedelta(days=1)).isoformat()
+        to_date = (now + timedelta(days=4)).isoformat()
+
+        raw_complete = {
+            "id": 99999,
+            "price": 1.49,
+            "oldPrice": 2.29,
+            "referencePrice": 5.96,
+            "requiresLoyalityMembership": True,
+            "description": "250g Packung",
+            "product": {"name": "Süßrahmbutter"},
+            "brand": {"name": "Landliebe"},
+            "advertisers": [{"name": "EDEKA", "id": "retailers/127093"}],
+            "unit": {"shortName": "kg"},
+            "volume": 0.25,
+            "validityDates": [{"from": from_date, "to": to_date}]
+        }
+
+        parsed = parse_offer(raw_complete, "93059")
+        self.assertIsNotNone(parsed)
+        self.assertEqual(parsed["id"], 99999)
+        self.assertEqual(parsed["supermarket"], "EDEKA")
+        self.assertEqual(parsed["price"], 1.49)
+        self.assertEqual(parsed["old_price"], 2.29)
+        self.assertEqual(parsed["discount_percent"], 35)  # (2.29 - 1.49) / 2.29 = 34.9% -> 35%
+        self.assertTrue(parsed["loyalty_app_required"])
+        self.assertTrue(parsed["is_valid_now"])
+        self.assertGreaterEqual(parsed["days_remaining"], 3)
+        self.assertIn("https://mg2de.b-cdn.net/api/v1/offers/99999", parsed["image_url"])
+
+        # 2. Minimal / empty payload
+        raw_empty = {}
+        parsed_min = parse_offer(raw_empty, "93059")
+        self.assertIsNotNone(parsed_min)
+        self.assertEqual(parsed_min["supermarket"], "Unbekannt")
+        self.assertIsNone(parsed_min["price"])
+        self.assertIsNone(parsed_min["discount_percent"])
+
+        # 3. Invalid types (not a dict)
+        self.assertIsNone(parse_offer("not a dict", "93059"))
+        self.assertIsNone(parse_offer(None, "93059"))
+
+    def test_cache_mechanism(self):
+        test_key = "test_unit_key_123"
+        test_data = {"hello": "world", "timestamp": 12345}
+
+        _write_cache(test_key, test_data)
+        data, is_fresh = _read_cache(test_key, max_age_seconds=60)
+        self.assertTrue(is_fresh)
+        self.assertEqual(data, test_data)
+
+        # Stale cache test
+        data_stale, is_fresh_stale = _read_cache(test_key, max_age_seconds=-1)
+        self.assertFalse(is_fresh_stale)
+        self.assertEqual(data_stale, test_data)  # still returned as fallback!
+
+    def test_list_supported_supermarkets(self):
+        catalog = list_supported_supermarkets()
+        self.assertEqual(catalog["default_zip_code"], "93059")
+        self.assertIn("Discounter", catalog["categories"])
+        self.assertIn("Supermarkt", catalog["categories"])
+        self.assertIn("aldi", catalog["supported_aliases"])
+        self.assertIn("edeka", catalog["supported_aliases"])
+
+
+class TestMarktguruMCP(unittest.TestCase):
+    def test_mcp_initialize(self):
+        req = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}})
+        resp = process_message(req)
+        self.assertEqual(resp["id"], 1)
+        self.assertEqual(resp["result"]["protocolVersion"], PROTOCOL_VERSION)
+        self.assertEqual(resp["result"]["serverInfo"]["name"], SERVER_NAME)
+
+    def test_mcp_tools_list(self):
+        req = json.dumps({"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}})
+        resp = process_message(req)
+        tools = [t["name"] for t in resp["result"]["tools"]]
+        self.assertIn("search_supermarket_offers", tools)
+        self.assertIn("compare_supermarket_prices", tools)
+        self.assertIn("list_supported_supermarkets", tools)
+
+    def test_mcp_tools_call_supermarket_catalog(self):
+        req = json.dumps({
+            "jsonrpc": "2.0",
+            "id": 3,
+            "method": "tools/call",
+            "params": {"name": "list_supported_supermarkets", "arguments": {}}
+        })
+        resp = process_message(req)
+        self.assertFalse(resp["result"]["isError"])
+        content_text = resp["result"]["content"][0]["text"]
+        data = json.loads(content_text)
+        self.assertEqual(data["default_zip_code"], "93059")
+
+    def test_mcp_tools_call_missing_query(self):
+        req = json.dumps({
+            "jsonrpc": "2.0",
+            "id": 4,
+            "method": "tools/call",
+            "params": {"name": "search_supermarket_offers", "arguments": {"query": ""}}
+        })
+        resp = process_message(req)
+        self.assertTrue(resp["result"]["isError"])
+        self.assertIn("required", resp["result"]["content"][0]["text"])
+
+    def test_mcp_notification_silence(self):
+        req = json.dumps({"jsonrpc": "2.0", "method": "notifications/initialized"})
+        resp = process_message(req)
+        self.assertIsNone(resp)  # Notifications must not produce responses
+
+
+if __name__ == "__main__":
+    unittest.main()

.agents/skills/openclaw-ops/SKILL.md

@@ -0,0 +1,231 @@
+---
+name: openclaw-ops
+license: Apache-2.0
+description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram.
+---
+
+# OpenClaw Operations & Troubleshooting Runbook
+
+> **Target Environment**: Kubernetes cluster `haumdaucher`, namespace `openclaw`, pod `openclaw-0`.  
+> **Key References**: [`k8s/openclaw/AGENTS.md`](../../../k8s/openclaw/AGENTS.md) and [`k8s/openclaw/TROUBLESHOOTING.md`](../../../k8s/openclaw/TROUBLESHOOTING.md).
+
+---
+
+## 0. Instant One-Shot Triage (< 2 Seconds)
+
+Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**.
+Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem.
+
+### Primary Command:
+```bash
+# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research)
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicNumber>
+
+# Or by agent name
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent <agentId>
+```
+
+The script automatically reports:
+1. Ground-truth session status (`done`, `processing`, `running`) and token size.
+2. Inbound user prompt (including speech-to-text audio transcripts).
+3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`❌`), and final text output.
+4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`.
+
+### 🚫 Anti-Patterns & Operational Guardrails:
+- ❌ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks.
+- ❌ **NEVER guess SQLite table or column names.**
+  - **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)`
+  - **`transcript_events`**: `(session_id, seq, event_json, created_at)`
+- 📍 **Known Path Anchors (Use Directly):**
+  - Knowledge Wiki: `/home/node/.openclaw/wiki/main`
+  - Agent SQLite DBs: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
+  - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
+  - Agent Workspace: `/home/openclaw/.openclaw/workspace-<agent>`
+  - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
+
+---
+
+## 1. Diagnostic Playbook: "Bot is Not Answering / Silent"
+
+When a user reports that a Telegram forum topic or direct chat is unresponsive:
+
+### ⚠️ Golden Rule: Do NOT Assume Container Crash
+Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (`5/5 Running`).
+
+### Step 1: Check Live Logs for In-Flight Turns
+```bash
+kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100
+```
+Look for:
+- `compact:start reason=auto-threshold`: Inline safeguard compaction in progress (takes 30–60s).
+- `embedded run tool start ... tool=write|exec`: Agent is actively executing tool steps.
+- `[model-fetch] start ... status=200`: Waiting on LLM token streaming.
+- `heartbeat: webhooks=... active=X`: Active turn count in gateway.
+
+### Step 2: Query Ground-Truth Session Status in SQLite
+Each agent maintains its own SQLite database at `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`.
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, glob
+for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
+    agent = db.split('/')[5]
+    con = sqlite3.connect(db)
+    rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall()
+    if rows:
+        print(f'[{agent}]', rows)
+"
+```
+- If `status == "processing"`: Turn is actively executing.
+- If `status == "done"`: Turn completed. Check outbound Telegram dispatch.
+- If `status == "running"` with old `updated_at`: Session is wedged (see Recovery below).
+
+### Step 3: Inspect Recent Transcript Events
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, json
+con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
+sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('<sessionKey>',)).fetchone()[0]
+for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall():
+    print(seq, ca, ej)
+"
+```
+
+---
+
+## 2. Session Recovery & Lock Clearing
+
+### Un-wedge a Stuck Session Node
+If a turn was killed mid-execution (e.g. OOMKill or node reboot):
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3
+con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
+con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('<sessionKey>',))
+con.commit()
+print('Session un-wedged successfully.')
+"
+```
+
+### Clean Stale State Locks
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock
+```
+
+### Check Foreign Key Violations & Clear Quarantine
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, glob
+for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'):
+    con = sqlite3.connect(db)
+    fk = con.execute('PRAGMA foreign_key_check;').fetchall()
+    if fk:
+        print(f'FK violations in {db}:', fk)
+"
+# Clear quarantine table if present
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import sqlite3, os
+p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite'
+if os.path.exists(p):
+    c = sqlite3.connect(p)
+    c.execute('DELETE FROM quarantined_databases')
+    c.commit()
+    print('Quarantine cleared.')
+"
+```
+
+---
+
+## 3. Multi-Agent Compaction & Transcript Bloat Control
+
+### Pitfall: Stale Sessions Report `totalTokens: null`
+`openclaw sessions --all-agents --json` returns `totalTokens: null` (`totalTokensFresh: false`) for sessions that haven't received a message in several days. **Always inspect SQLite directly**:
+```bash
+# Check size of active sessions across all agents
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import glob, sqlite3, json
+for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
+    agent = db.split('/')[5]
+    con = sqlite3.connect(db)
+    for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall():
+        cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone()
+        if bsz > 300000 or cnt > 40:
+            print(f'{agent:12} {bsz//1024:6} KB  {cnt:4} events  {skey}')
+"
+```
+
+### Manually Compact a Stale or Bloated Session
+```bash
+# LLM summarization with 3-minute timeout
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --timeout 180000
+
+# Truncation fallback (if LLM summarization times out)
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --max-lines 80 --timeout 180000
+```
+
+### Ensure User Compaction Notification is Enabled
+To prevent silent user-perceived freezes during daytime turns:
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json
+```
+
+---
+
+## 4. Mechanical HDD / Page Cache Pre-Warming
+
+Node `haumdaucher` uses spinning HDDs (`rotational: 1`). Cold database opens take up to 20–50 minutes without page cache warming.
+
+### Warm Page Cache Instantly (<1s)
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \
+  "cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true"
+```
+
+### WAL Checkpoints & SQLite Optimization
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
+import glob, sqlite3
+dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')
+for db in dbs:
+    con = sqlite3.connect(db)
+    con.execute('PRAGMA wal_checkpoint(TRUNCATE);')
+    con.execute('PRAGMA optimize;')
+    con.close()
+print('WAL checkpoints and optimizations complete.')
+"
+```
+
+---
+
+## 5. Emergency Telegram Direct Message Bypass
+
+If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn:
+```bash
+kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
+  --channel telegram \
+  --target "-1004326921510" \
+  --thread-id <topicNumber> \
+  --message "Dein Nachrichtentext hier..."
+```
+*(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).*
+
+---
+
+## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant
+
+At the end of **every single troubleshooting task, incident investigation, or bugfix**:
+
+### 🔄 Mandatory Dual-Reflection
+The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding:
+1. **Troubleshooting Process Optimization**:
+   - Did gathering information or isolating the root cause take too long?
+   - Did any query fail due to syntax, guessing, or slow filesystem sweeps?
+   - What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds?
+2. **System Maturity & Long-Term Hardening**:
+   - What fragile assumption or missing guardrail allowed this error to surface?
+   - How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs?
+
+### ⚖️ The Simplicity Invariant (The 98% Rule)
+- **Simplicity and clean, transparent architecture are the central design principles.**
+- Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%.
+- Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.
+

.agents/skills/promql/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: promql
+license: Apache-2.0
+description: Write, validate, and optimize PromQL for Prometheus / Grafana Mimir / Grafana Cloud Metrics. Covers `rate` vs `irate` vs `increase`, label matchers and regex, `sum / avg / topk / by / without` aggregation, classic + native `histogram_quantile`, ratios with divide-by-zero guards, `absent` / `changes` for staleness, time offsets and `predict_linear`, recording-rule naming, SLO + burn-rate math, and a cardinality-hunting playbook. Use when writing a metric query, fixing wrong p95s, building an error-budget alert, debugging "query is slow", finding the noisy label that blew up cardinality, or migrating a dashboard query to a recording rule — even when the user says "calculate the error rate", "p99 latency", "sum by service", "why is this query slow", or "what's filling Mimir" without naming PromQL.
+---
+
+# PromQL Query Patterns
+
+> **Docs**: https://prometheus.io/docs/prometheus/latest/querying/basics/
+
+PromQL returns either an **instant vector**, a **range vector**, or a **scalar**.
+
+**Golden rule:** `rate()` / `increase()` require a range vector ≥ 4× the scrape interval. 60s scrape → use `[5m]` minimum.
+
+## Prerequisites
+
+- A Prometheus / Mimir / Grafana Cloud endpoint to query (`/api/v1/query` or via Grafana Explore)
+- The PromQL pattern library in [`references/patterns.md`](references/patterns.md)
+
+## Common Workflows
+
+### 1. Write + validate a query
+
+```bash
+# 0. Point at your Prometheus/Mimir. For Grafana Cloud, use the metrics endpoint
+#    and add basic auth (-u "<metrics_user>:<token>") to each curl below.
+PROM=http://localhost:9090   # or https://prometheus-prod-XX.grafana.net/api/prom
+
+# 1. Sketch the query — for "5xx error rate per service":
+EXPR='sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (service)'
+
+# 2. Validate syntax + that the metric/labels exist
+curl -sG --data-urlencode "query=${EXPR}" \
+  "$PROM/api/v1/query" | jq '.status, (.data.result|length)'
+# Expect: "success" and result count > 0. If 0 — check label spelling and scrape activity:
+curl -sG --data-urlencode "match[]=http_requests_total" "$PROM/api/v1/series" | jq '.data | length'
+
+# 3. Sanity-check the magnitude — open Grafana Explore, paste the expr,
+#    confirm the values look right against a known ground truth (k6 run, log count, etc.)
+```
+
+### 2. Common patterns to copy
+
+**Per-status request rate** (aggregate AFTER rate):
+
+```promql
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+```
+
+**p95 latency** (must keep `le` in the inner aggregation):
+
+```promql
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+```
+
+**Error rate with divide-by-zero guard:**
+
+```promql
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / (sum(rate(http_requests_total[5m])) > 0)
+```
+
+Full library (recording rules, SLO burn-rate, offsets, cardinality hunt, native histograms): [`references/patterns.md`](references/patterns.md).
+
+### 3. Convert a slow dashboard query into a recording rule
+
+```yaml
+# 1. Pick the slow expression, give it a recording-rule name
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+```bash
+# 2. After rules load, verify the new metric exists
+curl -sG --data-urlencode "query=job:http_request_duration_p95:rate5m" \
+  "$PROM/api/v1/query" | jq '.data.result | length'   # → > 0
+
+# 3. Verify it matches the original expression for at least one sample window
+# (Both queries should produce the same value at the same timestamp.)
+
+# 4. Replace the dashboard panel expression with the recording-rule metric.
+```
+
+## Common bugs
+
+- `histogram_quantile` returns NaN → forgot `by (le)` in the inner aggregation
+- "No data" → check the metric exists (`/api/v1/series`) and the window ≥ 4× scrape interval
+- Wrong rate magnitude → counter was aggregated before `rate()` (always `rate()` first)
+- Query timeout → series count too high; use `topk(...)` + a recording rule + drop high-cardinality labels (see [`references/patterns.md`](references/patterns.md))
+
+## Resources
+
+- [PromQL basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
+- [Operators](https://prometheus.io/docs/prometheus/latest/querying/operators/)
+- [Functions](https://prometheus.io/docs/prometheus/latest/querying/functions/)
+- [Grafana Mimir](https://grafana.com/docs/mimir/latest/)

.agents/skills/promql/references/patterns.md

@@ -0,0 +1,174 @@
+# PromQL pattern library
+
+## Rate, counter, and aggregation
+
+```promql
+# rate() — per-second average
+rate(http_requests_total[5m])
+
+# CORRECT: rate first, then aggregate
+sum(rate(http_requests_total{job="api"}[5m])) by (status_code)
+
+# WRONG: never sum() a raw counter before rate()
+sum(http_requests_total) by (status_code)   # do NOT then rate()
+
+# increase() — total count over window
+increase(http_requests_total[1h])
+```
+
+`rate` vs `irate`:
+- `rate()` smooths the full window — use for dashboards + alerts
+- `irate()` uses the last two samples — for spike detection only, never alerting
+
+## Label matchers
+
+```promql
+http_requests_total{job="api", status_code="200"}
+http_requests_total{status_code=~"5.."}
+http_requests_total{status_code!~"2.."}
+http_requests_total{env=~"staging|production"}
+```
+
+## Aggregation
+
+```promql
+sum(rate(http_requests_total[5m])) by (service)
+avg(node_cpu_seconds_total{mode="idle"}) by (instance)
+
+# Top 5 by request rate
+topk(5, sum(rate(http_requests_total[5m])) by (service))
+
+# Count of distinct label values
+count(count(up) by (job)) by ()
+```
+
+`by` keeps only listed labels; `without` drops only listed labels.
+
+## Histogram quantiles
+
+```promql
+# Classic histogram
+histogram_quantile(0.99,
+  sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le))
+
+# Multi-service comparison — must include le AND service
+histogram_quantile(0.95,
+  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))
+
+# Native histograms (Prometheus 2.40+)
+histogram_quantile(0.95, sum(rate(http_request_duration_seconds[5m])))
+```
+
+Forgetting `by (le)` is the most common bug — `histogram_quantile` returns NaN or wrong values.
+
+## Ratios and error rates
+
+```promql
+# Error fraction
+sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+  / sum(rate(http_requests_total[5m]))
+
+# Success percentage
+(1 - sum(rate(http_requests_total{status_code=~"5.."}[5m]))
+     / sum(rate(http_requests_total[5m]))) * 100
+
+# Avoid divide-by-zero
+sum(rate(errors_total[5m]))
+  / (sum(rate(requests_total[5m])) > 0)
+```
+
+## Absence and staleness
+
+```promql
+absent(up{job="api"})                  # metric disappeared
+changes(up{job="api"}[5m]) == 0        # value not changing → stale exporter
+count_over_time(up{job="api"}[5m]) > 0 # at least one sample in window
+```
+
+## Time offsets
+
+```promql
+# Current vs 1h ago
+rate(http_requests_total[5m])
+  - rate(http_requests_total[5m] offset 1h)
+
+# Day-over-day
+rate(http_requests_total[5m])
+  / rate(http_requests_total[5m] offset 1d)
+
+# Predict 2h ahead from 1h trend (linear regression)
+predict_linear(node_filesystem_avail_bytes[1h], 2 * 3600)
+```
+
+## Recording rules
+
+Naming: `<aggregation_level>:<metric_name>:<operation_and_window>`
+
+```yaml
+groups:
+  - name: http_request_rates
+    interval: 1m
+    rules:
+      - record: job:http_requests_total:rate5m
+        expr:   sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_errors:ratio5m
+        expr: |
+          sum(rate(http_requests_total{status_code=~"5.."}[5m])) by (job)
+          / sum(rate(http_requests_total[5m])) by (job)
+
+      - record: job:http_request_duration_p95:rate5m
+        expr: |
+          histogram_quantile(0.95,
+            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
+```
+
+## SLO + burn-rate
+
+```promql
+# Availability SLO over 30 days
+1 - (sum(increase(http_requests_total{status_code=~"5.."}[30d]))
+     / sum(increase(http_requests_total[30d])))
+
+# 1h burn-rate vs target (replace 0.999 with your SLO target)
+( sum(rate(http_requests_total{status_code=~"5.."}[1h]))
+  / sum(rate(http_requests_total[1h])) )
+  / (1 - 0.999)
+```
+
+## Cardinality controls
+
+```promql
+# Top 10 metrics by series count
+topk(10, count by (__name__)({__name__=~".+"}))
+
+# Series count for one metric
+count(http_requests_total)
+
+# Cardinality of one label
+count(count by (user_id)(http_requests_total))
+```
+
+Drop a high-cardinality label at scrape time (Alloy):
+
+```alloy
+prometheus.scrape "api" {
+  targets = [...]
+  rule { source_labels = ["user_id"]
+         action        = "labeldrop" }
+}
+```
+
+## Other common patterns
+
+```promql
+# Service availability (alert)
+avg_over_time(up{job="api"}[5m]) < 0.9
+
+# Saturation — disk full in <4h
+predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 4 * 3600) < 0
+
+# Throughput spike — current rate > 3x 1h average
+rate(http_requests_total[5m])
+  > 3 * avg_over_time(rate(http_requests_total[5m])[1h:5m])
+```

.gitattributes

@@ -1,4 +1,8 @@
 *.secret filter=git-crypt diff=git-crypt
 *.secret.yaml filter=git-crypt diff=git-crypt
+*.secret.yml filter=git-crypt diff=git-crypt
 *.secret.values filter=git-crypt diff=git-crypt
 *.secret.sh filter=git-crypt diff=git-crypt
+*.secret.conf filter=git-crypt diff=git-crypt
+*.secret.auto.tfvars filter=git-crypt diff=git-crypt
+*.secret.tfvars filter=git-crypt diff=git-crypt

.gitignore

@@ -2,3 +2,10 @@ k8s/gitea-chart/
 .vagrant
 k8s/tekton-pipelines/auth
 .DS_Store
+node_modules/
+**/node_modules/
+dist/
+**/dist/
+*.tsbuildinfo
+.venv/
+**/__pycache__/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "knowledge-wiki"]
+	path = knowledge-wiki
+	url = https://git.moritzgraf.de/moritz/knowledge-wiki.git

AGENTS.md

@@ -0,0 +1,132 @@
+# AGENTS.md
+
+> [!NOTE]
+> **Infrapuzzle** is the monorepo for my personal infrastructure, encompassing Kubernetes cluster provisioning, workload deployment, DNS management, and IoT monitoring.
+
+## Repository Structure & Documentation
+This repository is divided into distinct domains, each with its own `AGENTS.md` or specialized documentation.
+
+### 1. [Bootstrap](./bootstrap/AGENTS.md) (`bootstrap/`)
+*   **Purpose**: Provisioning the single-node Kubernetes cluster (Haumdaucher).
+*   **Tool**: Kubespray (Ansible).
+*   **Key File**: `init.sh` (Source of Truth for versioning).
+*   **Read**: [bootstrap/AGENTS.md](./bootstrap/AGENTS.md)
+
+### 2. [Kubernetes Workloads](./k8s/AGENTS.md) (`k8s/`)
+*   **Purpose**: Deploying applications and services to the cluster.
+*   **Strict Namespace Rule**: **Every top-level subdirectory in `k8s/` strictly corresponds to a Kubernetes namespace.** Non-namespace folders, loose scripts, or documentation do not belong here.
+*   **Constraints**: Enforces `ingress-nginx` / `traefik`, `cert-manager` (Let's Encrypt), and `git-crypt` for secrets.
+*   **OpenClaw**: For OpenClaw AI agent deployment and troubleshooting, see [k8s/openclaw/AGENTS.md](./k8s/openclaw/AGENTS.md) and [k8s/openclaw/TROUBLESHOOTING.md](./k8s/openclaw/TROUBLESHOOTING.md).
+*   **Read**: [k8s/AGENTS.md](./k8s/AGENTS.md)
+
+### 3. [Terraform](./terraform/AGENTS.md) (`terraform/`)
+*   **Purpose**: Declarative core cloud infrastructure as code.
+    *   **DNS Zones**: `haumdaucher.de`, `moritzgraf.de`, `ameliegraf.de`, and others via OVH (`ovh/ovh`).
+    *   **Cloud Services**: Authentik SSO (`authentik.tf`), GCP APIs, Velero backup SA (`gcp_backups.tf`), OpenClaw calendar SA (`gcp_calendar.tf`), and Google Workspace MX/SPF/DKIM (`google_workspace.tf`).
+    *   **Usage**: `cd terraform && terraform apply`
+*   **Read**: [terraform/AGENTS.md](./terraform/AGENTS.md)
+
+### 4. Monitoring (`monitoring/`)
+*   **Purpose**: Physical ambient BLE sensor fleet monitoring across living areas.
+*   **Tool**: [ESPHome](https://esphome.io).
+*   **Devices**: Temperature & humidity sensors, Xiaomi BLE plants, ESP32 bridges (`muc-kitchen`, `muc-living`, `rgbg-living`).
+*   **Usage**:
+    ```bash
+    cd monitoring
+    esphome run config/<device>.secret.yml
+    ```
+*   **Read**: [monitoring/README.md](./monitoring/README.md)
+
+### 5. Knowledge Wiki & OpenClaw Tasks (`knowledge-wiki/`)
+*   **Purpose**: Shared personal knowledge wiki and task dispatch between OpenClaw (mobile/Telegram concierge) and Antigravity (local pair programming agent).
+*   **Location**: Git submodule `knowledge-wiki/` at root of `infrapuzzle` (Git repo: `https://git.moritzgraf.de/moritz/knowledge-wiki.git`, branch `main`).
+*   **Queue File**: `knowledge-wiki/tasks/antigravity.md`.
+*   **Workflow for Antigravity**:
+    - When the user asks to check tasks or pick up work from OpenClaw:
+      1. Pull latest: `git submodule update --remote knowledge-wiki` (or `git -C knowledge-wiki pull --rebase origin main`)
+      2. Read `knowledge-wiki/tasks/antigravity.md` to see pending items.
+      3. Implement the chosen task in the appropriate repository files.
+      4. Move the task to `## ✅ Completed Tasks` with completion details and commit info.
+      5. Push back: `git -C knowledge-wiki commit -am "chore(tasks): complete <task>" && git -C knowledge-wiki push origin main`.
+
+### 6. Architecture Decision Records (`docs/adr/`)
+*   **Purpose**: Formal record of architectural decisions, technical trade-offs, and design standards across the monorepo.
+*   **Format**: [MADR](https://adr.github.io/madr/) (Markdown Architectural Decision Records).
+*   **Consulting ADRs**: AI agents MUST consult `docs/adr/` before designing or altering system architecture, agent topologies, or tool integration patterns.
+*   **Key Records**:
+    *   [ADR 0001: Hybrid MCP Tool Search and Subagent Delegation for AI Agents](docs/adr/0001-hybrid-mcp-tool-search-and-subagents.md)
+    *   [ADR 0002: OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics](docs/adr/0002-openclaw-model-selection-tiering-and-economics.md)
+    *   [ADR 0003: Everything as Code (EaC) and Reproducible Declarative Infrastructure](docs/adr/0003-everything-as-code-and-reproducible-infrastructure.md)
+
+### 7. [Saschas Kronkorken Kreuzzug](./projects/sasch40/AGENTS.md) (`projects/sasch40/`)
+*   **Purpose**: Mobile-first 2.5D retro birthday adventure game for King Sascha's 40th jubilee (`https://sasch40.haumdaucher.de`).
+*   **Key Rules**: Prequel landing screen for QR codes, dynamic GameBoy arcade vs RPG layouts, real-photo character portraits, pure Web Audio synthesizer, competitive certificate scoring.
+*   **Read**: [projects/sasch40/AGENTS.md](./projects/sasch40/AGENTS.md)
+
+### 8. [Moritz Graf Personal Website](./projects/moritzgrafde/AGENTS.md) (`projects/moritzgrafde/`)
+*   **Purpose**: Personal homepage and portfolio for Moritz Graf (`https://moritzgraf.de`).
+*   **Manifest**: `k8s/web/moritzgrafde.yaml` (Namespace: `web`).
+*   **Image**: `registry.haumdaucher.de/moritzgrafde:latest`.
+*   **Key Rules**: Executive portfolio built with Vue 3, Vite, TypeScript, and Tailwind CSS served via Nginx in Docker.
+*   **Read**: [projects/moritzgrafde/AGENTS.md](./projects/moritzgrafde/AGENTS.md)
+
+### 9. [Haumdaucher Official Website](./projects/haumdaucherde/AGENTS.md) (`projects/haumdaucherde/`)
+*   **Purpose**: Official community platform and club portal for Haumdaucher Regensburg (`https://haumdaucher.de`).
+*   **Manifest**: `k8s/haumdaucher/haumdaucher.yaml` (Namespace: `haumdaucher`).
+*   **Image**: `registry.haumdaucher.de/haumdaucher-website:latest`.
+*   **Key Rules**: PWA built with Vue 3, Vite, TypeScript, and Firebase Auth; dynamic theming with NAT mode easter egg; strict compliance with German Vereinsrecht (§ 5 DDG Impressum for n.e.V.); user creation and mail forwarding Apps Script automations.
+*   **Read**: [projects/haumdaucherde/AGENTS.md](./projects/haumdaucherde/AGENTS.md)
+
+### 10. [Grabengasse 7 Home Portal](./projects/grabengasse/AGENTS.md) (`projects/grabengasse/`)
+*   **Purpose**: Local climate, weather, and environmental dashboard for Grabengasse 7 (`https://grabengasse7.de`).
+*   **Manifests**: `k8s/grabengasse/` (`frontend.yaml`, `backend.yaml`, `ingress.yaml` in namespace `grabengasse`).
+*   **Images**: `registry.haumdaucher.de/website/grabengasse_frontend:latest`, `registry.haumdaucher.de/website/grabengasse_backend:latest`.
+*   **Key Rules**: Vue 3 retro pixel art frontend + FastAPI backend querying cluster Prometheus for Shelly WS90 weather telemetry.
+*   **Read**: [projects/grabengasse/AGENTS.md](./projects/grabengasse/AGENTS.md)
+
+### 11. [Kitchen e-Paper Display](./projects/esphome_display_kitchen/AGENTS.md) (`projects/esphome_display_kitchen/`)
+*   **Purpose**: Dedicated ESP32 microcontroller driving the Waveshare e-Paper status display in the kitchen.
+*   **Integrations**: Home Assistant Native API (`input_text.epaper_message`), Prometheus client, SNTP time sync.
+*   **Key Tools**: Interactive `uv run flash.py` wizard and `./send_message.sh` CLI push messaging.
+*   **Read**: [projects/esphome_display_kitchen/AGENTS.md](./projects/esphome_display_kitchen/AGENTS.md)
+
+### 12. [DuckDB Backend](./projects/duckdb-backend/AGENTS.md) (`projects/duckdb-backend/`)
+*   **Purpose**: High-performance HTTP analytics and event tracking data service executing in-memory DuckDB queries against Knowledge Wiki JSONL datasets.
+*   **Manifests**: `k8s/family/` (`duckdb-backend.yaml`, `pvc.yaml`, `ingress.yaml` in namespace `family`).
+*   **Endpoints**: Curated dashboards (`/duckdb-api/v1/mila/dashboard`, `/duckdb-api/v1/moritz/dashboard`), generic SQL queries (`/duckdb-api/v1/query`), and event tracking writes with Git sync.
+*   **Read**: [projects/duckdb-backend/AGENTS.md](./projects/duckdb-backend/AGENTS.md)
+
+### 13. [Mila Tracking Web App](./projects/mila/AGENTS.md) (`projects/mila/`)
+*   **Purpose**: Reactive mobile-first PWA for Mila's baby sleep, feeding, diaper tracking, and growth percentiles (`https://family.haumdaucher.de/tracking/mila`).
+*   **Manifest**: `k8s/family/mila.yaml` (Namespace: `family`).
+*   **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; live wake timer; interactive 1-tap and custom-time tracking modal.
+*   **Read**: [projects/mila/AGENTS.md](./projects/mila/AGENTS.md)
+
+### 14. [Moritz Health & Nutrition Web App](./projects/moritz/AGENTS.md) (`projects/moritz/`)
+*   **Purpose**: Apple Health-inspired nutrition and metabolic balance dashboard (`https://family.haumdaucher.de/tracking/moritz`).
+*   **Manifest**: `k8s/family/moritz.yaml` (Namespace: `family`).
+*   **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; net calorie balance hero, Garmin health breakdown, and quick meal logger.
+*   **Read**: [projects/moritz/AGENTS.md](./projects/moritz/AGENTS.md)
+
+### 15. [Shelly Taupunktlüfter](./projects/shelly_taupunktluefter/AGENTS.md) (`projects/shelly_taupunktluefter/`)
+*   **Purpose**: Automated cellar ventilation controller ("Taupi") on Shelly Plug S Gen 3 to prevent mold growth using psychrometric calculations and BTHome BLE sensors.
+*   **Usage**: `cd projects/shelly_taupunktluefter && terraform apply`
+*   **Key Rules**: Sedlbauer LIM I decoupled mold danger evaluation, summer overheating and winter cold guards, optimal night drying window (03:00–07:00), and Prometheus metrics export.
+*   **Read**: [projects/shelly_taupunktluefter/AGENTS.md](./projects/shelly_taupunktluefter/AGENTS.md)
+
+### 16. Utility Scripts (`scripts/`)
+*   **Purpose**: Cross-cutting operational and administration CLI scripts that do not belong inside Kubernetes namespace directories.
+*   **Key Tools**:
+    *   `scripts/htpasswd.py`: Generates bcrypt-hashed htpasswd credentials for ingress basic auth secrets.
+
+### 17. Network & Router Integration (`fritzbox/`)
+*   **Purpose**: Site-to-Site WireGuard LAN coupling and DynDNS OVH synchronization for the home FritzBox router.
+*   **Cluster Endpoint**: `k8s/wireguard/` (`vpn.haumdaucher.de`).
+*   **Read**: [fritzbox/README.md](./fritzbox/README.md)
+
+## Global Constraints
+*   **Secrets**: `git-crypt` is used extensively. Ensure you have the unlock key before editing `*.secret` files.
+*   **Identity**:
+    *   **Cluster**: `haumdaucher`
+    *   **Primary Domain**: `haumdaucher.de`
+    *   **User Identity**: `moritzgraf.de`

README.md

@@ -1,25 +1,26 @@
-# Infrapuzzle
+# Infrapuzzle 🧩
 
-Infrapuzzle is the newly restructured way of implementing my private infrastructure. Currently the following things are in project scope:
+Infrapuzzle is the monorepo for personal infrastructure, Kubernetes cluster operations, DNS, IoT monitoring, and hosted applications.
 
-* single node k8s "cluster" implemented with kubespray
+## Repository Layout
-* development with Vagrant
-* host directory storage
-* Ingress via host network
 
-## terraform
+- **[`bootstrap/`](bootstrap/)**: Provisioning the single-node Kubernetes cluster (`haumdaucher`) using Kubespray.
+- **[`k8s/`](k8s/)**: Kubernetes workloads. **Every subdirectory represents a Kubernetes namespace.**
+- **[`terraform/`](terraform/)**: Declarative Infrastructure as Code for core cloud infrastructure (OVH DNS zones, Authentik SSO, Google Workspace, and GCP APIs).
+- **[`projects/`](projects/)**: Self-contained hosted applications and projects:
+  - `projects/shelly_taupunktluefter/`: Smart home Shelly Plug S ventilation controller ("Taupi") for cellar mold prevention.
+  - `projects/sasch40/`: Retro birthday adventure game (`sasch40.haumdaucher.de`).
+  - `projects/moritzgrafde/`: Personal homepage and executive portfolio (`moritzgraf.de`).
+  - `projects/haumdaucherde/`: Official community website & portal (`haumdaucher.de`).
+  - `projects/grabengasse/`: Climate & weather dashboard (`grabengasse7.de`).
+  - `projects/esphome_display_kitchen/`: ESP32 e-Paper kitchen status display & flashing wizard.
+  - `projects/duckdb-backend/`: In-memory analytics and tracking query API.
+  - `projects/mila/`: Baby tracking PWA (`family.haumdaucher.de/tracking/mila`).
+  - `projects/moritz/`: Health and nutrition dashboard (`family.haumdaucher.de/tracking/moritz`).
+- **[`monitoring/`](monitoring/)**: ESPHome fleet for BLE ambient room temperature & humidity sensors.
+- **[`fritzbox/`](fritzbox/)**: Home router Site-to-Site WireGuard VPN coupling and DynDNS synchronization.
+- **[`scripts/`](scripts/)**: Operational utility scripts (e.g. `htpasswd.py`).
+- **[`knowledge-wiki/`](knowledge-wiki/)**: Shared personal knowledge base and OpenClaw task queue submodule.
+- **[`docs/`](docs/)**: Architecture Decision Records (ADRs) following the MADR standard.
 
-[Documentation in subfolder](./terraform/README.md). It takes care of setting DNS records on OVH nameservers.
+For architectural standards and operational guidelines, refer to [AGENTS.md](AGENTS.md).
-
-## bootstrap
-
-[Documentation in subfolder](./bootstrap/README.md). It takes care of setting up the k8s cluster itself, including a vagrant dev environment.
-
-## k8s
-
-[Documentation in subfolder](./k8s/README.md). The services themselfes.
-
-## Links used
-
-* [ingress via host network](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/#via-the-host-network)
-* [Install K8s with Kubespray](https://kubernetes.io/docs/setup/production-environment/tools/kubespray/)

bootstrap/.gemini/settings.json

@@ -0,0 +1,15 @@
+{
+  "mcpServers": {
+    "kubespray-changelog": {
+      "command": "./bin/get_kubespray_changelog.py",
+      "args": [],
+      "trust": true
+    }
+  },
+  "general": {
+    "preferredEditor": "vscode"
+  },
+  "ui": {
+    "theme": "Default"
+  }
+}

bootstrap/.gemini/settings.json.orig

@@ -0,0 +1,16 @@
+{
+  "theme": "Default",
+  "preferredEditor": "vscode",
+  "mcpServers": {
+    //======================================================================
+    // *** FOR YOUR KUBESPRAY GOAL ***
+    // This is the config for the custom Python script we discussed.
+    // This is the tool that will *actually* search changelogs.
+    //======================================================================
+    "kubespray-changelog": {
+      "command": "./bin/get_kubespray_changelog.py",
+      "args": [],
+      "trust": true
+    }
+  }
+}

bootstrap/.gitignore

@@ -1,2 +1,4 @@
 kubespray
 venv
+credentials/
+inventory/**/credentials/

bootstrap/AGENTS.md

@@ -0,0 +1,83 @@
+# AGENTS.md
+
+> [!NOTE]
+> This directory handles the **bootstrapping, provisioning, and in-place upgrades** of the Haumdaucher Kubernetes cluster using **Kubespray**.
+
+## Project Overview
+*   **Tool**: [Kubespray](https://github.com/kubernetes-sigs/kubespray) (Ansible-based).
+*   **Topology**: **Single-node cluster** (`haumdaucher`, `136.243.23.215`). The single node acts simultaneously as control plane, etcd member, and worker.
+*   **Host OS**: Ubuntu 20.04.6 LTS.
+*   **Method**: The local `inventory/` is the source of truth, synced into the checked-out Kubespray repository.
+*   **Idempotency & Patching**: The `kubespray/` directory is ephemeral and managed by `init.sh`. Any essential Kubespray fixes are persisted in `patches/` and applied automatically.
+
+---
+
+## Architectural Invariants for Single-Node Upgrades
+
+> [!IMPORTANT]
+> The single-node topology imposes strict operational constraints that must be observed by all agents and operators:
+
+1. **NO DRAIN / NO CORDON**:
+   - Draining or cordoning a single-node cluster evicts all workloads with nowhere to schedule them, causing cascading failures and pending pods.
+   - `drain_nodes: false`, `upgrade_node_always_cordon: false`, and `upgrade_node_fail_if_drain_fails: false` MUST remain configured in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
+   - Use the in-place upgrade mechanism: `ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml`. This invokes `kubeadm-upgrade.yml` without cordon/drain overhead.
+2. **VERSION STRING FORMAT**:
+   - In `inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, `kube_version` must be specified **without** the leading `v` (e.g., `1.32.13`, NOT `v1.32.13`), as Kubespray 2.28 interpolates `v{{ kube_version }}` in download URLs and package names.
+3. **UBUNTU 20.04 OS CONSTRAINT**:
+   - Kubespray `release-2.28` is the **final release line** supporting Ubuntu 20.04 LTS. Upstream `release-2.29`+ drops Ubuntu 20.04 support. Do NOT bump `VERSION` in `init.sh` beyond `release-2.28` until the host operating system has been upgraded to Ubuntu 22.04 or 24.04.
+4. **ETCD BACKUPS**:
+   - Always take a manual etcd snapshot before initiating any cluster upgrade.
+
+---
+
+## Workflow & Scripts
+
+### 1. Source `init.sh`
+```bash
+source init.sh
+```
+**Actions performed**:
+- Clones / checks out the Kubespray branch defined by `VERSION` (`release-2.28`).
+- Applies monorepo patches in `patches/*.patch` (e.g., propagating `--ignore-preflight-errors` for `v1beta4` kubeadm configs).
+- Activates the `infrapuzzle-bootstrap` pyenv virtualenv and verifies Python requirements.
+- Syncs `./inventory/` to `./kubespray/inventory/`.
+- Exports the `sync-inventory` alias.
+- Changes working directory to `./kubespray`.
+
+### 2. Upgrading the Single-Node Cluster (Two-Phase Decoupled Workflow)
+
+#### Phase 1: Reconcile Kubespray Tooling Baseline (Keep existing k8s version)
+```bash
+cd kubespray
+ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+```
+*Verification*: Check node readiness and that no application pods crashed.
+
+#### Phase 2: Upgrade Kubernetes Version (e.g. to 1.32.13)
+1. Edit `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`:
+   ```yaml
+   kube_version: 1.32.13
+   ```
+2. Re-sync inventory from `bootstrap/` or `bootstrap/kubespray/`:
+   ```bash
+   sync-inventory
+   ```
+3. Run the in-place upgrade:
+   ```bash
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
+4. Verify the cluster:
+   ```bash
+   kubectl version
+   kubectl get nodes -o wide
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
+
+---
+
+## Directory Structure
+*   `init.sh`: The entrypoint script (sources environment, applies patches, sets up inventory).
+*   `patches/`: Monorepo patches applied automatically to ephemeral `kubespray/`.
+*   `inventory/`: Source of truth for cluster inventory and variables.
+*   `kubespray/`: (Ephemeral / Gitignored) Cloned Kubespray repository.

bootstrap/README.md

@@ -1,105 +1,165 @@
 # Bootstrap
 
-The following lines document how to initalize a fresh cluster. On a real cluster, or using Vagrant. It therefore assumes to clone kubespray to this folder. It will be excluded in *.gitignore* and all files are in this folder.
+This directory contains the automation, configurations, and playbooks used to provision and upgrade the single-node Kubernetes cluster **`haumdaucher`** (`136.243.23.215`) using **[Kubespray](https://github.com/kubernetes-sigs/kubespray)**.
 
-The version to be initialized in configured in the `init.sh` script.
+---
+
+## Architecture & Single-Node Cluster Constraints
+
+`haumdaucher` is a **single-node cluster** (control plane, etcd, and worker all run on the same machine). This topology requires specific configuration choices during cluster maintenance:
+
+> [!IMPORTANT]
+> ### Crucial Single-Node Invariants
+> 1. **Never Cordon or Drain the Node**:
+>    Draining a single-node cluster evicts all workloads with nowhere to schedule them, causing pods to stay in `Pending` state and breaking critical cluster services (such as DNS, CNI, ingress, and local storage).
+>    In [inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml), the following settings are enforced:
+>    ```yaml
+>    drain_nodes: false
+>    upgrade_node_always_cordon: false
+>    upgrade_node_fail_if_drain_fails: false
+>    ```
+> 2. **In-Place Upgrades via `cluster.yml`**:
+>    Instead of `upgrade-cluster.yml` (which defaults to draining nodes), single-node upgrades should use the in-place upgrade mechanism:
+>    ```sh
+>    ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+>    ```
+> 3. **Version Number Formatting**:
+>    Kubespray 2.28 interpolates `v{{ kube_version }}` across many internal tasks. Therefore, `kube_version` in `k8s-cluster.yml` MUST NOT include a leading `v`:
+>    ```yaml
+>    kube_version: 1.32.13   # Correct
+>    # kube_version: v1.32.13 # INCORRECT (leads to v1.32.13 -> vv1.32.13 errors)
+>    ```
+> 4. **Host OS Lifecycle Constraint (Ubuntu 20.04 LTS)**:
+>    Kubespray `release-2.28` is the **final release line supporting Ubuntu 20.04**. Kubespray `release-2.29`+ drops Ubuntu 20.04 support and requires Ubuntu 22.04 or 24.04. Do not bump `VERSION` in `init.sh` beyond `release-2.28` until the host OS is upgraded.
+
+---
+
+## Quickstart
+
+### 1. Environment Initialization
+Source `init.sh` from this directory. It clones Kubespray, applies patches from `patches/`, configures the Python virtualenv, and syncs `./inventory/` into `./kubespray/inventory/`:
 
 ```sh
-. ./init.sh
+source ./init.sh
 ```
 
-See [kubespray.io](https://kubespray.io/) on detailed information about kubespray. Though it seems to be a littlebit outdated.
+*(Note: `init.sh` also exports the `sync-inventory` alias to quickly sync local `./inventory/` modifications into `./kubespray/inventory/` without re-cloning).*
 
-## Vagrant
+---
 
-```sh
+## Step-by-Step Upgrade Runbook
-cd kubespray
-vagrant up
-# up and abkle to ssh
-vagrant ssh k8s-1
-```
 
-## Prod
+Upgrades are conducted in two decoupled phases:
+1. **Phase 1**: Upgrade the Kubespray tooling baseline while keeping the Kubernetes version unchanged $\rightarrow$ Apply $\rightarrow$ Verify.
+2. **Phase 2**: Upgrade Kubernetes components (API server, controller manager, scheduler, kubelet, kube-proxy) $\rightarrow$ Apply $\rightarrow$ Verify.
 
-Prepare server:
+### Step 0: Pre-Flight Checks & Backups
-* deactivate swap!
-* `moritz username     ALL=(ALL) NOPASSWD:ALL`
-* `
 
-```sh
+1. **Verify SSH and sudo access**:
-ssh centos@<ip>
+   ```sh
-# auth via pw
+   ssh moritz@136.243.23.215 "sudo kubectl get nodes -o wide"
-sudo su - root
+   ```
-adduser moritz
-visudo # add as sudo user
-su - moritz
-sudo yum -y install vim python3
-ssh-keygen
-vim .ssh/authorized_users # paste key
-chmod 644 .ssh/authorized_keys
-# check whether login works with ssh key
-sudo vim /etc/ssh/sshd_config # remove pw auth & root login
-sudo yum upgrade -y && sudo reboot
-```
 
-Install Kubernetes:
+2. **Check Pod Health**:
+   ```sh
+   ssh moritz@136.243.23.215 "sudo kubectl get pods -A | grep -v -E 'Running|Completed'"
+   ```
 
-```sh
+3. **Take a Manual etcd Snapshot**:
-. ./init.sh
+   ```sh
-# follow instructions from output, sth like:
+   ssh moritz@136.243.23.215 "sudo etcdctl \
-cd kubespray
+     --cacert=/etc/ssl/etcd/ssl/ca.pem \
-ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+     --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem \
-```
+     --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem \
+     snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db"
+   ```
 
-And get credentials:
+---
 
-```sh
+### Step 1: Reconcile Kubespray Baseline (Phase 1)
-ssh <ip>
-sudo su - root
-cd
-cp -r .kube /home/moritz/
-chown -R moritz. /home/moritz/.kube
-#ctrl + d
-kubectl get ns # test connection
-#ctrl + d
-scp haumdaucher.de:/home/moritz/.kube/config .kube/config
-```
 
-Foreward in k8s-directory.
+1. Verify `VERSION` in `init.sh` matches the target Kubespray branch (e.g. `release-2.28`).
+2. Run `source ./init.sh` to initialize the environment.
+3. Apply the baseline configuration:
+   ```sh
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini cluster.yml
+   ```
+4. Verify cluster health:
+   ```sh
+   kubectl get nodes -o wide
+   kubectl get pods -n kube-system
+   ```
 
-## Upgrade cluster
+---
 
-Check the current default value of `kube_version` in cloned repository.
+### Step 2: Upgrade Kubernetes Version (Phase 2)
 
-**FOR K8S VERSION - Edit /Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml**
+1. In `bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml`, update `kube_version`:
+   ```yaml
+   kube_version: 1.32.13
+   ```
 
-```sh
+2. Sync the configuration into the `kubespray/` workspace:
-cd kubespray
+   ```sh
-ansible-playbook -i inventory/prod/inventory.ini -e kube_version=v1.22.10 -e upgrade_cluster_setup=true cluster.yml
+   sync-inventory
-# or just the newest version
+   ```
-ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
-# upgrade to specific calico version (did not trigger/ failed)
-ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true -e calico_version=v3.15.2 cluster.yml --tags=network
-```
 
-History:
+3. Run the in-place upgrade playbook:
+   ```sh
+   cd kubespray
+   ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml
+   ```
 
-* 2020-04-18 kube_version=v1.16.8 kubespray_branch=release-2.12
+---
 
-## Add node
+### Step 3: Post-Upgrade Verification
 
-See [documentation](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/nodes.md).
+1. **Check component versions**:
+   ```sh
+   kubectl version
+   ```
+   Ensure both `Client Version` and `Server Version` report `v1.32.13`.
 
-Note: This was more or less a trial and error approach. Running different playbooks over and over again got it right at some point.
+2. **Verify Node status**:
+   ```sh
+   kubectl get nodes -o wide
+   ```
+   Ensure status is `Ready` and version matches `v1.32.13`.
 
-```sh
+3. **Verify System Pods**:
-ansible-playbook -i inventory/prod/inventory.ini --limit=ns3088070.ip-37-59-40.eu,ns3100058.ip-37-59-61.eu scale.yml
+   ```sh
-ansible-playbook -i inventory/prod/inventory.ini --limit=etcd,kube-master -e ignore_assert_errors=yes cluster.yml
+   kubectl get pods -n kube-system
-```
+   ```
+   All control plane components (`kube-apiserver`, `kube-controller-manager`, `kube-scheduler`, `coredns`, `kube-flannel`, `kube-proxy`, `traefik`) should be `1/1 Running`.
 
-This runs everything and is kind of idempotent:
+4. **Verify Workloads**:
+   ```sh
+   kubectl get pods -A | grep -v -E "Running|Completed"
+   ```
 
-```sh
+---
-ansible-playbook -i inventory/prod/inventory.ini cluster.yml
-```
 
+## Troubleshooting & Edge Cases
+
+### Kubeadm Preflight `CreateJob` Failure
+Kubeadm 1.31+ preflight executes an ephemeral `upgrade-health-check-...` batch job with a 15-second timeout. On single-node clusters or under load, this may exceed 15s. To prevent the upgrade from halting:
+- `CreateJob` is included in `kubeadm_ignore_preflight_errors` in [k8s-cluster.yml](file:///Users/moritz/src/infrapuzzle/bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml).
+- The patch `patches/kubeadm-upgrade-ignore-preflight.patch` ensures Kubespray's `v1beta4` kubeadm configuration propagates this flag during `upgrade apply`.
+
+### Kubelet Housekeeping Stalls / Static Pod Timeout
+If `kubeadm upgrade apply` times out waiting for static pod hash changes:
+- Check `journalctl -u kubelet -n 100` for `Housekeeping took longer than expected` or `orphaned pod ... failed to rmdir()`.
+- If an old container unmount left a stale bind-mount busy, lazily detach it:
+  ```sh
+  sudo umount -l /var/lib/kubelet/pods/<orphaned-pod-uid>/volumes/...
+  sudo systemctl restart kubelet
+  ```
+
+---
+
+## Directory Layout
+* `init.sh`: Environment bootstrapping script and upgrade instructions.
+* `inventory/`: Source-of-truth inventory and variable files.
+* `patches/`: Git patches applied to upstream Kubespray on checkout.
+* `kubespray/`: (Ephemeral) Cloned Kubespray repository.

bootstrap/explanation_cert_issue.md

@@ -0,0 +1,65 @@
+# Understanding and Resolving Kubernetes Certificate Expiration in Kubespray
+
+## Introduction: The Role of Certificates in Kubernetes
+
+A Kubernetes cluster relies heavily on TLS certificates to secure communication between its various components. The API server, controller manager, scheduler, etcd, and kubelets all use certificates to authenticate and encrypt traffic. These certificates are issued with a specific validity period (usually one year) for security reasons. When they expire, components can no longer trust each other, leading to a cluster-wide failure.
+
+The error message you encountered is a classic symptom of this problem:
+
+```
+E1116 13:47:01.271977 ... failed to verify certificate: x509: certificate has expired or is not yet valid
+```
+
+This indicates that `kubectl` (and other components) could not validate the certificate presented by the Kubernetes API server because the current date was past the certificate's expiration date.
+
+## The Core Problem: A "Chicken-and-Egg" Deadlock
+
+When the certificates expired, the initial and correct instinct was to use Kubespray's provided automation to fix it. In your version of Kubespray, the `upgrade-cluster.yml` playbook is the designated tool for this job, as it includes tasks to regenerate certificates.
+
+However, this approach led to a deadlock, manifesting as a timeout during the "Create kubeadm token for joining nodes" task. Here's a breakdown of why this happened:
+
+1.  **API Server is Down:** The primary certificate for the Kubernetes API server (`apiserver.crt`) had expired. This prevented the API server from starting correctly and serving traffic on its secure port (6443).
+2.  **Playbook Needs the API Server:** The `upgrade-cluster.yml` playbook, specifically the `kubeadm` tasks within it, needs to communicate with a healthy Kubernetes API server to perform its functions. To create a join token for other nodes, `kubeadm` must make a request to the API server.
+3.  **The Deadlock:** The playbook was trying to connect to the API server to fix the certificates, but it couldn't connect precisely *because* the certificates were already expired and the API server was unhealthy. This created a "chicken-and-egg" scenario where the automated solution couldn't run because the problem it was meant to fix was preventing it from running.
+
+## The Solution: Manual Intervention on the Control Plane
+
+To break this deadlock, we had to manually restore the core health of the control plane on the master node (`haumdaucher`) *before* letting the automation take over again. The process involved SSHing into the master node and using the `kubeadm` command-line tool to regenerate the essential certificates and configuration files.
+
+Here is a detailed look at the commands executed and why they were necessary:
+
+1.  **`sudo -i`**
+    *   **What it does:** Switches to the `root` user.
+    *   **Why:** Modifying files in `/etc/kubernetes/` requires root privileges.
+
+2.  **`mv /etc/kubernetes/pki /etc/kubernetes/pki.backup-...`**
+    *   **What it does:** Backs up the directory containing all the existing (and expired) cluster certificates.
+    *   **Why:** This is a critical safety measure. If the manual renewal process failed, we could restore the original state to diagnose the problem further.
+
+3.  **`kubeadm init phase certs all`**
+    *   **What it does:** This is the core of the manual fix. It tells `kubeadm` to execute *only* the certificate generation phase of the cluster initialization process. It creates a new Certificate Authority (CA) and uses it to sign a fresh set of certificates for all control plane components (API server, controller-manager, scheduler, etcd).
+    *   **Why:** This directly replaces the expired certificates with new, valid ones, allowing the API server and other components to trust each other again.
+
+4.  **`mv /etc/kubernetes/*.conf /etc/kubernetes/*.conf.backup-...`**
+    *   **What it does:** Backs up the kubeconfig files used by the administrator (`admin.conf`) and the control plane components.
+    *   **Why:** These files contain embedded client certificates and keys for authentication. Since we just created new certificates, we also need to generate new kubeconfig files that use them. Backing them up is a standard precaution.
+
+5.  **`kubeadm init phase kubeconfig all`**
+    *   **What it does:** This command generates new kubeconfig files (`admin.conf`, `kubelet.conf`, etc.) that use the newly created certificates from the previous step.
+    *   **Why:** This ensures that all components, as well as the administrator using `kubectl` on the node, can successfully authenticate to the now-healthy API server.
+
+6.  **`systemctl restart kubelet`**
+    *   **What it does:** Restarts the kubelet service on the master node.
+    *   **Why:** The kubelet is the agent that runs on each node and is responsible for managing pods. It needs to be restarted to load its new configuration (`kubelet.conf`) and re-establish a secure connection to the API server using the new certificates.
+
+## Why the Solution Works
+
+By performing these manual steps, we effectively gave the control plane a "jump-start." We manually created the valid certificates and configuration files needed for the API server to start up successfully.
+
+Once the API server was healthy and listening, the `upgrade-cluster.yml` playbook could be re-run. This time, when the `kubeadm` tasks within the playbook tried to connect to the API server to create join tokens, the connection succeeded. The playbook was then able to complete its remaining tasks, ensuring all nodes in the cluster were properly configured and joined.
+
+## Future Prevention and Best Practices
+
+1.  **Monitor Certificate Expiration:** Use tools like `kubeadm certs check-expiration` or monitoring solutions (e.g., Prometheus with `kube-state-metrics`) to track certificate expiry dates proactively.
+2.  **Consider Upgrading Kubespray:** Newer versions of Kubespray may include a dedicated `renew-certs.yml` playbook. This playbook is designed for certificate rotation specifically and is less disruptive than a full `upgrade-cluster.yml`, as it typically avoids rotating service account keys.
+3.  **Understand the Manual Process:** Keeping this guide handy will allow you to quickly resolve similar deadlocks in the future without extensive troubleshooting. The key is recognizing that automation sometimes needs a manual boost when the system it's trying to fix is too broken to respond.

bootstrap/init.sh

@@ -1,60 +1,78 @@
 #!/bin/sh
 
-# check whether script is sourced
+# Ensure script is sourced, not executed directly
-[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "script is  executed, but needs to be sourced" && exit 1
+[[ "${BASH_SOURCE[0]}" == "${0}" ]] && echo "Error: init.sh must be sourced: source ./init.sh (or . ./init.sh)" && exit 1
 
-echo "######################################################################################"
+BOOTSTRAP_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-echo "##  Reinit repository"
+VERSION="release-2.28" # Note: release-2.28 is the final release line supporting Ubuntu 20.04 LTS
-rm -rf kubespray
-VERSION="release-2.26"
-git clone --branch $VERSION https://github.com/kubernetes-sigs/kubespray.git
 
-echo "######################################################################################"
+echo "======================================================================================"
-echo "##  Activating pyenv venv"
+echo "## [1/4] Checking / Cloning Kubespray ($VERSION)"
+echo "======================================================================================"
+if [ ! -d "$BOOTSTRAP_DIR/kubespray/.git" ]; then
+  rm -rf "$BOOTSTRAP_DIR/kubespray"
+  git clone --branch "$VERSION" --depth 1 https://github.com/kubernetes-sigs/kubespray.git "$BOOTSTRAP_DIR/kubespray"
+else
+  echo "Existing kubespray checkout detected. Ensuring correct branch..."
+  git -C "$BOOTSTRAP_DIR/kubespray" fetch origin "$VERSION" --depth 1
+  git -C "$BOOTSTRAP_DIR/kubespray" checkout "$VERSION"
+fi
+
+if [ -d "$BOOTSTRAP_DIR/patches" ]; then
+  for p in "$BOOTSTRAP_DIR"/patches/*.patch; do
+    if [ -f "$p" ]; then
+      echo "## Applying patch $(basename "$p")..."
+      git -C "$BOOTSTRAP_DIR/kubespray" apply --check "$p" 2>/dev/null && git -C "$BOOTSTRAP_DIR/kubespray" apply "$p" || echo "Patch $(basename "$p") already applied or skipped."
+    fi
+  done
+fi
+
+echo "======================================================================================"
+echo "## [2/4] Activating Python Virtualenv"
+echo "======================================================================================"
 eval "$(pyenv init -)"
 eval "$(pyenv virtualenv-init -)"
-pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
+pyenv virtualenvs --bare | grep -q "^infrapuzzle-bootstrap$" || pyenv virtualenv 3.10.8 infrapuzzle-bootstrap
 pyenv activate infrapuzzle-bootstrap
-python -m pip install -r kubespray/requirements.txt
+python -m pip install -q -r "$BOOTSTRAP_DIR/kubespray/requirements.txt"
 
+echo "======================================================================================"
+echo "## [3/4] Syncing Inventory Configuration"
+echo "======================================================================================"
+rsync -a "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
 
-echo "######################################################################################"
+# Helper function for re-syncing changes made in ./inventory to ephemeral ./kubespray/inventory
-echo "##  Customizing vagrant dev env"
+sync-inventory() {
-mkdir -p kubespray/vagrant
+  rsync -av "$BOOTSTRAP_DIR/inventory/" "$BOOTSTRAP_DIR/kubespray/inventory/"
-cat << EOF > kubespray/vagrant/config.rb
+}
-\$instance_name_prefix = "k8s"
+export -f sync-inventory 2>/dev/null || true
-\$vm_cpus = 4
+alias sync-inventory="rsync -av '$BOOTSTRAP_DIR/inventory/' '$BOOTSTRAP_DIR/kubespray/inventory/'"
-\$num_instances = 1
-\$os = "centos"
-\$subnet = "10.0.20"
-\$network_plugin = "calico"
-\$shared_folders = { 'temp/docker_rpms' => "/var/cache/yum/x86_64/7/docker-ce/packages" }
-\$kube_node_instances_with_disks_number = 0
-EOF
 
-# make the rpm cache
+echo "======================================================================================"
-mkdir -p kubespray/temp/docker_rpms
+echo "## [4/4] Environment Ready! Single-Node Cluster Upgrade Workflow"
-
+echo "======================================================================================"
-echo "###############"
-echo "Now cd to kubespray and 'vagrant up'"
 echo ""
-echo "export KUBECONFIG=\"$( pwd )/kubespray/inventory/sample/artifacts/admin.conf\""
+echo "Current directory changed to: $BOOTSTRAP_DIR/kubespray"
-
+cd "$BOOTSTRAP_DIR/kubespray"
-
+echo ""
-echo "######################################################################################"
+echo "=== PRE-UPGRADE CHECKLIST ==="
-echo "## * syncing config in kubespray dir"
+echo "1. Verify SSH connection and sudo access:"
-rsync -a ./inventory/ ./kubespray/inventory/
+echo "   ssh moritz@136.243.23.215 'sudo kubectl get nodes -o wide'"
-echo "## * changing into kubespray dir"
+echo "2. Take manual etcd snapshot before upgrading:"
-cd kubespray
+echo "   ssh moritz@136.243.23.215 'sudo etcdctl --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-haumdaucher.pem --key=/etc/ssl/etcd/ssl/node-haumdaucher-key.pem snapshot save /var/backups/etcd/etcd-pre-upgrade-\$(date +%Y%m%d%H%M%S).db'"
-echo "## * execute the following command to force new settings"
+echo ""
-echo "ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
+echo "=== TWO-PHASE UPGRADE WORKFLOW ==="
-echo "## * execute the following command to force new settings with forced cluster upgrade"
+echo "Phase 1: Reconcile Kubespray baseline (keeps current k8s version):"
-echo "ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
+echo "   ansible-playbook -i inventory/prod/inventory.ini cluster.yml"
-
+echo ""
-# cp -r kubespray/inventory/sample kubespray/inventory/prod
+echo "Phase 2: Upgrade Kubernetes version (e.g. to 1.32.13):"
-# rm kubespray/inventory/prod/inventory.ini
+echo "   a) Update 'kube_version: <target>' in ../inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml"
-# cp ./prod.ini kubespray/inventory/prod/inventory.ini
+echo "   b) Sync inventory: sync-inventory"
-#gsed -i "s/kube_network_plugin: .*/kube_network_plugin: flannel/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-cluster.yml
+echo "   c) Run in-place single-node upgrade:"
-# gsed -i "s/# calico_iptables_backend: \"Legacy\"/calico_iptables_backend: \"NFT\"/" ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml
+echo "      ansible-playbook -i inventory/prod/inventory.ini -e upgrade_cluster_setup=true cluster.yml"
-#echo 'calico_iptables_backend: "NFT"' >> ./kubespray/inventory/prod/group_vars/k8s-cluster/k8s-net-cluster.yml
+echo ""
-#gsed -i "s/metrics_server_enabled: .*/metrics_server_enabled: true/" ./kubespray/inventory/prod/group_vars/k8s-cluster/addons.yml
+echo "=== POST-UPGRADE VERIFICATION ==="
+echo "   kubectl version"
+echo "   kubectl get nodes -o wide"
+echo "   kubectl get pods -A | grep -v -E 'Running|Completed'"
+echo "======================================================================================"

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-calico.yml

@@ -1,78 +0,0 @@
-# see roles/network_plugin/calico/defaults/main.yml
-
-## With calico it is possible to distributed routes with border routers of the datacenter.
-## Warning : enabling router peering will disable calico's default behavior ('node mesh').
-## The subnets of each nodes will be distributed by the datacenter router
-# peer_with_router: false
-
-# Enables Internet connectivity from containers
-# nat_outgoing: true
-
-# add default ippool name
-# calico_pool_name: "default-pool"
-
-# add default ippool blockSize (defaults kube_network_node_prefix)
-# calico_pool_blocksize: 24
-
-# add default ippool CIDR (must be inside kube_pods_subnet, defaults to kube_pods_subnet otherwise)
-# calico_pool_cidr: 1.2.3.4/5
-
-# Global as_num (/calico/bgp/v1/global/as_num)
-# global_as_num: "64512"
-
-# You can set MTU value here. If left undefined or empty, it will
-# not be specified in calico CNI config, so Calico will use built-in
-# defaults. The value should be a number, not a string.
-# calico_mtu: 1500
-
-# Configure the MTU to use for workload interfaces and tunnels.
-# - If Wireguard is enabled, set to your network MTU - 60
-# - Otherwise, if VXLAN or BPF mode is enabled, set to your network MTU - 50
-# - Otherwise, if IPIP is enabled, set to your network MTU - 20
-# - Otherwise, if not using any encapsulation, set to your network MTU.
-# calico_veth_mtu: 1440
-
-# Advertise Cluster IPs
-# calico_advertise_cluster_ips: true
-
-# Choose data store type for calico: "etcd" or "kdd" (kubernetes datastore)
-# calico_datastore: "etcd"
-
-# Choose Calico iptables backend: "Legacy", "Auto" or "NFT"
-#calico_iptables_backend: "NFT"
-
-# Use typha (only with kdd)
-# typha_enabled: false
-
-# Generate TLS certs for secure typha<->calico-node communication
-# typha_secure: false
-
-# Scaling typha: 1 replica per 100 nodes is adequate
-# Number of typha replicas
-# typha_replicas: 1
-
-# Set max typha connections
-# typha_max_connections_lower_limit: 300
-
-# Set calico network backend: "bird", "vxlan" or "none"
-# bird enable BGP routing, required for ipip mode.
-# calico_network_backend: bird
-
-# IP in IP and VXLAN is mutualy exclusive modes.
-# set IP in IP encapsulation mode: "Always", "CrossSubnet", "Never"
-# calico_ipip_mode: 'Always'
-
-# set VXLAN encapsulation mode: "Always", "CrossSubnet", "Never"
-# calico_vxlan_mode: 'Never'
-
-# If you want to use non default IP_AUTODETECTION_METHOD for calico node set this option to one of:
-# * can-reach=DESTINATION
-# * interface=INTERFACE-REGEX
-# see https://docs.projectcalico.org/reference/node/configuration
-# calico_ip_auto_method: "interface=eth.*"
-# Choose the iptables insert mode for Calico: "Insert" or "Append".
-# calico_felix_chaininsertmode: Insert
-
-# If you want use the default route interface when you use multiple interface with dynamique route (iproute2)
-# see https://docs.projectcalico.org/reference/node/configuration : FELIX_DEVICEROUTESOURCEADDRESS
-# calico_use_default_route_src_ipaddr: false

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-canal.yml

@@ -1,10 +0,0 @@
-# see roles/network_plugin/canal/defaults/main.yml
-
-# The interface used by canal for host <-> host communication.
-# If left blank, then the interface is choosing using the node's
-# default route.
-# canal_iface: ""
-
-# Whether or not to masquerade traffic to destinations not within
-# the pod network.
-# canal_masquerade: "true"

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-cilium.yml

@@ -1 +0,0 @@
-# see roles/network_plugin/cilium/defaults/main.yml

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-contiv.yml

@@ -1,20 +0,0 @@
-# see roles/network_plugin/contiv/defaults/main.yml
-
-# Forwarding mode: bridge or routing
-# contiv_fwd_mode: routing
-
-## With contiv, L3 BGP mode is possible by setting contiv_fwd_mode to "routing".
-## In this case, you may need to peer with an uplink
-## NB: The hostvars must contain a key "contiv" of which value is a dict containing "router_ip", "as"(defaults to contiv_global_as), "neighbor_as" (defaults to contiv_global_neighbor_as), "neighbor"
-# contiv_peer_with_uplink_leaf: false
-# contiv_global_as: "65002"
-# contiv_global_neighbor_as: "500"
-
-# Fabric mode: aci, aci-opflex or default
-# contiv_fabric_mode: default
-
-# Default netmode: vxlan or vlan
-# contiv_net_mode: vxlan
-
-# Dataplane interface
-# contiv_vlan_interface: ""

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-flannel.yml

@@ -1,18 +0,0 @@
-# see roles/network_plugin/flannel/defaults/main.yml
-
-## interface that should be used for flannel operations
-## This is actually an inventory cluster-level item
-# flannel_interface:
-
-## Select interface that should be used for flannel operations by regexp on Name or IP
-## This is actually an inventory cluster-level item
-## example: select interface with ip from net 10.0.0.0/23
-## single quote and escape backslashes
-# flannel_interface_regexp: '10\\.0\\.[0-2]\\.\\d{1,3}'
-
-# You can choose what type of flannel backend to use: 'vxlan' or 'host-gw'
-# for experimental backend
-# please refer to flannel's docs : https://github.com/coreos/flannel/blob/master/README.md
-# flannel_backend_type: "vxlan"
-# flannel_vxlan_vni: 1
-# flannel_vxlan_port: 8472

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-kube-router.yml

@@ -1,61 +0,0 @@
-# See roles/network_plugin/kube-router//defaults/main.yml
-
-# Enables Pod Networking -- Advertises and learns the routes to Pods via iBGP
-# kube_router_run_router: true
-
-# Enables Network Policy -- sets up iptables to provide ingress firewall for pods
-# kube_router_run_firewall: true
-
-# Enables Service Proxy -- sets up IPVS for Kubernetes Services
-# see docs/kube-router.md "Caveats" section
-# kube_router_run_service_proxy: false
-
-# Add Cluster IP of the service to the RIB so that it gets advertises to the BGP peers.
-# kube_router_advertise_cluster_ip: false
-
-# Add External IP of service to the RIB so that it gets advertised to the BGP peers.
-# kube_router_advertise_external_ip: false
-
-# Add LoadbBalancer IP of service status as set by the LB provider to the RIB so that it gets advertised to the BGP peers.
-# kube_router_advertise_loadbalancer_ip: false
-
-# Adjust manifest of kube-router daemonset template with DSR needed changes
-# kube_router_enable_dsr: false
-
-# Array of arbitrary extra arguments to kube-router, see
-# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md
-# kube_router_extra_args: []
-
-# ASN numbers of the BGP peer to which cluster nodes will advertise cluster ip and node's pod cidr.
-# kube_router_peer_router_asns: ~
-
-# The ip address of the external router to which all nodes will peer and advertise the cluster ip and pod cidr's.
-# kube_router_peer_router_ips: ~
-
-# The remote port of the external BGP to which all nodes will peer. If not set, default BGP port (179) will be used.
-# kube_router_peer_router_ports: ~
-
-# Setups node CNI to allow hairpin mode, requires node reboots, see
-# https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode
-# kube_router_support_hairpin_mode: false
-
-# Select DNS Policy ClusterFirstWithHostNet, ClusterFirst, etc.
-# kube_router_dns_policy: ClusterFirstWithHostNet
-
-# Array of annotations for master
-# kube_router_annotations_master: []
-
-# Array of annotations for every node
-# kube_router_annotations_node: []
-
-# Array of common annotations for every node
-# kube_router_annotations_all: []
-
-# Enables scraping kube-router metrics with Prometheus
-# kube_router_enable_metrics: false
-
-# Path to serve Prometheus metrics on
-# kube_router_metrics_path: /metrics
-
-# Prometheus metrics port to use
-# kube_router_metrics_port: 9255

bootstrap/inventory/prod/group_vars/k8s-cluster/k8s-net-weave.yml

@@ -1,58 +0,0 @@
-# see roles/network_plugin/weave/defaults/main.yml
-
-# Weave's network password for encryption, if null then no network encryption.
-# weave_password: ~
-
-# If set to 1, disable checking for new Weave Net versions (default is blank,
-# i.e. check is enabled)
-# weave_checkpoint_disable: false
-
-# Soft limit on the number of connections between peers. Defaults to 100.
-# weave_conn_limit: 100
-
-# Weave Net defaults to enabling hairpin on the bridge side of the veth pair
-# for containers attached. If you need to disable hairpin, e.g. your kernel is
-# one of those that can panic if hairpin is enabled, then you can disable it by
-# setting `HAIRPIN_MODE=false`.
-# weave_hairpin_mode: true
-
-# The range of IP addresses used by Weave Net and the subnet they are placed in
-# (CIDR format; default 10.32.0.0/12)
-# weave_ipalloc_range: "{{ kube_pods_subnet }}"
-
-# Set to 0 to disable Network Policy Controller (default is on)
-# weave_expect_npc: "{{ enable_network_policy }}"
-
-# List of addresses of peers in the Kubernetes cluster (default is to fetch the
-# list from the api-server)
-# weave_kube_peers: ~
-
-# Set the initialization mode of the IP Address Manager (defaults to consensus
-# amongst the KUBE_PEERS)
-# weave_ipalloc_init: ~
-
-# Set the IP address used as a gateway from the Weave network to the host
-# network - this is useful if you are configuring the addon as a static pod.
-# weave_expose_ip: ~
-
-# Address and port that the Weave Net daemon will serve Prometheus-style
-# metrics on (defaults to 0.0.0.0:6782)
-# weave_metrics_addr: ~
-
-# Address and port that the Weave Net daemon will serve status requests on
-# (defaults to disabled)
-# weave_status_addr: ~
-
-# Weave Net defaults to 1376 bytes, but you can set a smaller size if your
-# underlying network has a tighter limit, or set a larger size for better
-# performance if your network supports jumbo frames (e.g. 8916)
-# weave_mtu: 1376
-
-# Set to 1 to preserve the client source IP address when accessing Service
-# annotated with `service.spec.externalTrafficPolicy=Local`. The feature works
-# only with Weave IPAM (default).
-# weave_no_masq_local: true
-
-# Extra variables that passing to launch.sh, useful for enabling seed mode, see
-# https://www.weave.works/docs/net/latest/tasks/ipam/ipam/
-# weave_extra_args: ~

bootstrap/inventory/prod/group_vars/k8s_cluster/k8s-cluster.yml

@@ -27,7 +27,17 @@ kube_kubeadm_apiserver_extra_args:
 kube_api_anonymous_auth: true
 
 ## Change this to use another Kubernetes version, e.g. a current beta release
-kube_version: v1.29.5
+kube_version: 1.32.13
+
+# Single-node cluster settings: prevent draining or cordoning the only node
+drain_nodes: false
+upgrade_node_always_cordon: false
+upgrade_node_fail_if_drain_fails: false
+kubeadm_ignore_preflight_errors:
+  - ImagePull
+  - SystemVerification
+  - FileContent--proc-sys-net-bridge-bridge-nf-call-iptables
+  - CreateJob
 
 # kubernetes image repo define
 #kube_image_repo: "k8s.gcr.io"
@@ -122,6 +132,9 @@ kube_proxy_mode: iptables
 # Moritz: changed to iptables for tegra
 #kube_proxy_mode: ipvs
 
+# Expose kube-proxy metrics on all interfaces so Prometheus can scrape port 10249
+kube_proxy_metrics_bind_address: "0.0.0.0:10249"
+
 # configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
 # must be set to true for MetalLB to work
 kube_proxy_strict_arp: false

bootstrap/inventory/prod/inventory.ini

@@ -3,35 +3,16 @@
 # ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
 [all]
 haumdaucher
-#jetson1.dyndns.moritzgraf.de
-# container_manager=containerd resolvconf_mode=docker_dns
 
-#ns3088070.ip-37-59-40.eu ansible_host=37.59.40.95 ansible_become=yes ansible_become_method=sudo ansible_python_interpreter=/usr/bin/python3
+[kube_control_plane]
-#ns3100058.ip-37-59-61.eu ansible_host=37.59.61.198 ansible_become=yes ansible_become_method=sudo ansible_python_interpreter=/usr/bin/python3
-# node1 ansible_host=95.54.0.12  # ip=10.3.0.1 etcd_member_name=etcd1
-# node2 ansible_host=95.54.0.13  # ip=10.3.0.2 etcd_member_name=etcd2
-# node3 ansible_host=95.54.0.14  # ip=10.3.0.3 etcd_member_name=etcd3
-# node4 ansible_host=95.54.0.15  # ip=10.3.0.4 etcd_member_name=etcd4
-# node5 ansible_host=95.54.0.16  # ip=10.3.0.5 etcd_member_name=etcd5
-# node6 ansible_host=95.54.0.17  # ip=10.3.0.6 etcd_member_name=etcd6
-
-# ## configure a bastion host if your nodes are not directly reachable
-# bastion ansible_host=x.x.x.x ansible_user=some_user
-
-[kube-master]
 haumdaucher
-#ns3088070.ip-37-59-40.eu
 
 [etcd]
 haumdaucher
 
-[kube-node]
+[kube_node]
 haumdaucher
-#jetson1.dyndns.moritzgraf.de
 
-[calico-rr]
+[k8s_cluster:children]
-
+kube_control_plane
-[k8s-cluster:children]
+kube_node
-kube-master
-kube-node
-calico-rr

bootstrap/patches/kubeadm-upgrade-ignore-preflight.patch

@@ -0,0 +1,20 @@
+diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+index 523956d1d..75723a6e0 100644
+--- a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
++++ b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
+@@ -17,6 +17,7 @@
+     --force
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     {%- if kube_version is version('1.32.0', '>=') %}
+     --skip-phases={{ kubeadm_init_phases_skip | join(',') }}
+@@ -38,6 +39,7 @@
+     {% if kubeadm_patches | length > 0 %}--patches={{ kubeadm_patches_dir }}{% endif %}
+     {%- else %}
+     --config={{ kube_config_dir }}/kubeadm-config.yaml
++    --ignore-preflight-errors={{ kubeadm_ignore_preflight_errors | join(',') }}
+     {%- endif %}
+     --skip-phases={{ kubeadm_upgrade_node_phases_skip | join(',') }}
+   register: kubeadm_upgrade

docs/adr/0001-hybrid-mcp-tool-search-and-subagents.md

@@ -0,0 +1,148 @@
+---
+status: "accepted"
+date: 2026-09-12
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0001: Hybrid MCP Tool Search and Subagent Delegation for AI Agents
+
+## Context and Problem Statement
+
+OpenClaw operates as the personal AI concierge for the Graf family, communicating primarily via Telegram and web interfaces. It integrates external systems via the Model Context Protocol (MCP) (e.g., Home Assistant, Prometheus, Spotify, weather, shopping).
+
+When integrating comprehensive domain services like **Garmin Connect** via [`Taxuspt/garmin_mcp`](https://github.com/Taxuspt/garmin_mcp), the server exposes **148 distinct tools** spanning activity analysis, workout builders, physiological trends, sleep, stress, HRV, and gear tracking. 
+
+In standard MCP implementations, all tool schemas are transmitted in the LLM's `tools` array on *every single request turn*. For 148 tools, this imposes a **~25,000–30,000 token tax per turn** before a single word of user prompt is evaluated. This creates severe issues:
+1. **Severe Token & Cost Bloat**: Inflates inference costs on OpenRouter across thousands of daily family turns.
+2. **Turn Latency & Degradation**: Slows down response times on mobile Telegram and impairs the reasoning accuracy of fast conductor models (e.g. `deepseek-v4.1-flash`) due to tool schema noise.
+3. **Imbalance for Simple Checks**: A simple query like *"Wie war mein Schlaf?"* should not require dragging 148 tool definitions into context or, conversely, incurring the multi-second startup latency of a full subagent session.
+
+How do we grant OpenClaw full access to large, complex MCP tool catalogs without polluting everyday conversational context?
+
+---
+
+## Decision Drivers
+
+* **Token Economics**: Minimize idle token overhead on routine conversational turns.
+* **Low Latency**: Provide instant answers for routine, single-step lookups.
+* **Complex Multi-Step Capabilities**: Enable autonomous, deep reasoning for tasks like workout generation, periodization, and multi-source research without cluttering the primary conversation transcript.
+* **Protocol Standards**: Adhere to upstream MCP standards rather than writing fragile, non-standard CLI wrappers for every API.
+* **Architectural Consistency**: Establish an extensible pattern that can later be applied to refactor other domains (e.g. Supermarket shopping, Jellyfin media, and Regensburg Stadtrat local news research).
+
+---
+
+## Considered Options
+
+1. **Option 1: Global Eager MCP Registration**
+   * Register all 148 tools directly under `mcp.servers`.
+   * *Drawback*: Unacceptable ~30k token tax on every message; high cost and latency.
+
+2. **Option 2: Pure CLI / Script Wrapper Skill**
+   * Avoid MCP entirely; write custom Python CLI scripts in `mop-skills` executed via `bash`.
+   * *Drawback*: Re-invents the wheel, ignores the 1,150-star community MCP server, requires maintaining hundreds of lines of bespoke CLI parsing code.
+
+3. **Option 3: Subagent Delegation Only**
+   * Hide all domain tools from the main agent and mandate that *every* query spawns a subagent via `sessions_spawn`.
+   * *Drawback*: Adds unnecessary latency (2–4 seconds) and session overhead for trivial questions like *"Wie hoch ist meine Body Battery?"*.
+
+4. **Option 4: Hybrid Tool Search + Subagent Delegation (Selected)**
+   * Enable OpenClaw's native **Tool Search** (`tools.toolSearch.mode: "directory"`) for the main agent so large tool catalogs are deferred behind lightweight search/describe/call operations (~600 tokens).
+   * Empower the main conductor to dynamically decide:
+     * Execute directly via Tool Search for fast, single-step queries.
+     * Delegate to an isolated subagent (`sessions_spawn`) for deep, multi-step, stateful reasoning.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 4: Hybrid Tool Search + Subagent Delegation**.
+
+### 1. Dual-Path Architecture
+
+```
+User Query (Telegram)
+         │
+         ▼
+ ┌────────────────────────────────────────┐
+ │       OpenClaw Main Conductor          │
+ │       (DeepSeek v4.1 Flash)            │
+ └───────────────────┬────────────────────┘
+                     │
+         Is this a deep/multi-step task?
+        ┌────────────┴────────────┐
+        │ NO                      │ YES
+        ▼                         ▼
+ ┌───────────────┐        ┌─────────────────────────┐
+ │ Direct Tool   │        │ Subagent Delegation     │
+ │ Search        │        │ (sessions_spawn)        │
+ ├───────────────┤        ├─────────────────────────┤
+ │ • tool_search │        │ • Dedicated Context     │
+ │ • tool_call   │        │ • Full Tool Access      │
+ │ • Single Turn │        │ • Multi-Step Reasoning  │
+ └───────┬───────┘        └────────────┬────────────┘
+         │                             │
+         └──────────────┬──────────────┘
+                        ▼
+       ┌─────────────────────────────────┐
+       │   Taxuspt/garmin_mcp (148 Tools)│
+       │   Garmin Connect Cloud API      │
+       └─────────────────────────────────┘
+```
+
+### 2. Main Conductor Routing Heuristic
+
+The main conductor is guided by explicit system prompt rules:
+
+| Query Type | Routing Path | Examples |
+| :--- | :--- | :--- |
+| **Simple Vitals & Lookups** | **Direct Tool Search** | • *"Wie war mein Schlaf letzte Nacht?"*<br>• *"Wie ist meine Body Battery?"*<br>• *"Was habe ich gestern trainiert?"* |
+| **Deep Analysis & Creation** | **Subagent Delegation (`sessions_spawn`)** | • *"Erstelle mir ein 45-minütiges Oberkörper-Superset-Workout für heute auf meiner Uhr."*<br>• *"Analysiere mein Krafttraining der letzten 3 Wochen auf Progressive Overload und plane die nächste Woche."*<br>• *"Erstelle ein Periodisierungskonzept für den nächsten Monat."* |
+
+---
+
+## Consequences
+
+### Positive
+* **98% Token Reduction on Idle Turns**: The tool catalog is deferred behind OpenClaw's native Tool Search, dropping tool schema overhead from ~30,000 tokens to ~600 tokens.
+* **Instant Answers for Vitals**: Quick checks are answered in a single turn without waiting for subagent spin-up.
+* **Isolated Deep Reasoning**: Heavy multi-turn planning and JSON compilation occur inside temporary subagents, keeping the main Telegram conversation transcript lightweight and readable.
+* **Standard-Compliant**: Uses standard upstream MCP (`Taxuspt/garmin_mcp`) and standard OpenClaw features (`tools.toolSearch`, `sessions_spawn`).
+
+### Negative & Mitigations
+* **Two-Step Tool Search Latency**: For the first direct lookup of a tool, OpenClaw performs a search step before calling the tool.
+  * *Mitigation*: OpenClaw's `directory` mode caches known tool descriptors and allows direct calls when the exact tool name is known.
+* **Model Prompt Instruction**: Requires clear guidelines in the system prompt so the conductor consistently knows when to delegate vs. query directly.
+
+---
+
+## Future Roadmap: Monorepo Refactoring Blueprint
+
+This Hybrid MCP architecture serves as the official template for all agentic workflows in `infrapuzzle`. On 2026-09-16, this architecture was formally expanded into the **Full-Spectrum Multi-Agent Architecture** with 9 specialized subagents:
+
+### 1. Agents-as-Code in the Knowledge Wiki (`knowledge-wiki/agents/`)
+Subagent specifications, system prompts, tool assignments, and domain guidelines are decoupled from Kubernetes deployment manifests and stored as versioned Markdown files in `knowledge-wiki/agents/<name>.md`.
+* **Zero-Deployment Tuning**: Adding or refining an agent requires no Kubernetes pod restarts; changes synced via Git are immediately effective.
+* **Extensibility Blueprint**: Adding a 10th agent requires simply adding `agents/<new-agent>.md` with YAML frontmatter specifying model, tools, and owned concepts.
+
+### 2. The 9 Specialized Subagents Roster
+1. 🩺 **Digitale Hebamme Agent** (`agents/hebamme.md`): Baby milestones, sleep windows, BLW, pediatrician safety boundaries.
+2. 🍳 **Koch Agent** (`agents/koch.md`): Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization, exact recipes.
+3. 🏡 **Grabengasse Agent** (`agents/grabengasse.md`): Home Assistant, Philips Hue, Sonos multiroom, house systems.
+4. 🏰 **Regensburg Agent** (`agents/regensburg.md`): Regensburg & Stadtamhof insider, local places, RVV transit, city notices.
+5. ✈️ **Travel Agent** (`agents/travel.md`): Vacation planning, itineraries, packing checklists, on-the-road travel concierge.
+6. 🎬 **Media Agent** (`agents/media.md`): Jellyfin streaming/requests, Jellyseerr, Spotify audiobooks & playlists.
+7. 🔍 **Recherche Agent** (`agents/recherche.md`): Deep research engine (iterative multi-hop), ALLRIS Stadtrat, Friday bridge updates.
+8. 🛒 **Shopping Agent** (`agents/shopping.md`): Deal hunter, Geizhals all-time lows, supermarket deals, multi-tier webshop extraction.
+9. 🏋️ **Fitness Coach Agent** (`agents/fitness.md`): Garmin Connect (148 tools), Tacx Flux 2 (FTP 275 W), home gym supersets.
+
+### 3. Multi-Turn Continuity: Domain Stickiness & Rich State Bridge
+To prevent context loss during ongoing step-by-step dialogues (e.g. interactive cooking or workout design):
+* **Persistent Context Continuity**: Follow-up turns are routed within the dedicated subagent topic session using native conversational memory, keeping user-facing messages clean without embedding raw JSON/HTML metadata tags.
+
+### 4. 3-Tier Session Lifecycle (Preventing Unbounded Session Growth)
+1. **Conversational Close**: Topic shifts or explicit user completion (*"Danke, das war's"*) reset the active domain pointer.
+2. **Ephemeral Inactivity Cleanup**: Subagent child sessions inactive for > 60 minutes (`subagents.archiveAfterMinutes: 60`) are automatically archived.
+3. **Deterministic Nightly Compaction**: Nightly 03:30 AM CET compaction vacuums expired child sessions from the SQLite store.
+

docs/adr/0002-openclaw-model-selection-tiering-and-economics.md

@@ -0,0 +1,258 @@
+---
+status: "accepted (fleet-wide primary: DeepSeek-V4.1-Flash-1017; updated 2026-10-17)"
+date: 2026-09-13
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0002: OpenClaw Full-Spectrum Model Selection, Multimodal Tiering, and Cost Governance
+
+## Context and Problem Statement
+
+OpenClaw serves as the 24/7 personal AI concierge, family assistant, and operational orchestrator for Moritz Graf and his family. It interacts across Telegram, voice notes, webhooks, and background automations, bridging to personal knowledge (Obsidian wiki), smart home systems (Home Assistant, Sonos), fitness tracking (Garmin Connect), and cluster monitoring (Prometheus, Gitea).
+
+During early September 2026, OpenClaw consumed **143.26 million tokens** over 7 days, accumulating **\$12.55+** in raw OpenRouter API costs. Forensic analysis revealed that over **80% of this spend was architectural waste**:
+1. **Unbounded Context Re-transmission**: An uncompacted session (`agent:main:main`) reached 381k tokens (2,650 events, 5.7 MB JSON). Every minor turn re-ingested this entire history, costing \$12.20 (97.2% of total spend).
+2. **Heartbeat Amplification Loop**: Background heartbeats firing every 30 minutes targeted the active user session (`target: last`), re-evaluating the 381k context 48 times/day simply to output `NO_REPLY` (\$5.16 wasted).
+3. **Expensive & Deprecated Model Allocations**: Frontier models (`claude-3.7-sonnet` and `deepseek-v4-pro`) were configured without clear cost-benefit boundaries, with subagents frequently timing out (>300s). In addition, legacy model generations (e.g., Gemini 2.5) lingered in configuration despite superior, newer generations being available.
+4. **Modality Conflation (Image Input vs. Image Generation)**: Vision perception (analyzing an uploaded photo) was conflated with text-to-image synthesis (drawing an image). Configuring chat models like `z-ai/glm-5.3-flash` for image synthesis caused runtime HTTP 404 failures because pure LLMs lack diffusion/raster generation heads.
+
+Moritz established two foundational requirements for the repository:
+> 1. *"I want high quality, but it should not be too expensive."*
+> 2. *"We always want to use the newest models (as they usually are the best)."*
+
+How do we architect the full-spectrum model selection across **all modalities** (text, subagents, heartbeats, vision, image generation, audio transcription, text-to-speech, web search, and vector embeddings) to deliver premium quality, sub-10s latency, and a total operating budget of **<\$2.00 / week**?
+
+---
+
+## Decision Drivers
+
+* **State-of-the-Art Recency**: Exclusively employ the latest generation models (e.g. DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, Sonar Pro). Completely decommission legacy generations (e.g. Gemini 2.5).
+* **Conversational & Reasoning Quality**: High emotional intelligence, nuanced German fluency, personal context recall, and strict adherence to tool-calling protocols (ADR 0001).
+* **Strict Modality Decoupling**: Each capability must route to its dedicated, best-in-class engine:
+  - Text & Agentic Reasoning
+  - Native Multimodal Vision (reading user photos)
+  - Text-to-Image Generation (synthesizing new pictures)
+  - Speech-to-Text / Audio Input (transcribing voice messages)
+  - Text-to-Speech / Audio Output (synthesizing briefings and podcasts)
+  - Web Search & Retrieval (grounded search)
+  - Semantic Embeddings (vector memory)
+* **Inference Economics**: Low per-token costs for large conversational contexts (15k–80k tokens), high prompt-cache discount efficiency (`cacheRead`), zero background bleed, and predictable billing.
+* **Latency & Turn Responsiveness**: Conversational turns on Telegram must return within **4–10 seconds**, avoiding provider queuing or massive prefill bottlenecks.
+* **Resilience & Fault Tolerance**: Automated fallback chains and strict quantization/provider pins (`fp8`/`bf16`, `tool_choice` support) across independent providers on OpenRouter.
+
+---
+
+## Full-Spectrum Model Architecture & Concrete Allocations
+
+```
+┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                       OpenClaw Ingress Gateway                                         │
+└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
+                                                   │
+         ┌───────────────────┬─────────────────────┼─────────────────────┬───────────────────┐
+         ▼                   ▼                     ▼                     ▼                   ▼
+┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
+│ Text Conductor  │ │ Subagent Engine │   │ Background &    │   │ Image Synthesis │ │ Voice & Audio   │
+│ & Vision Input  │ │ (sessions_spawn)│   │ Heartbeat       │   │ (Text-to-Image) │ │ (STT & TTS)     │
+├─────────────────┤ ├─────────────────┤   ├─────────────────┤   ├─────────────────┤ ├─────────────────┤
+│ Tier 1          │ │ Tier 2          │   │ Tier 3          │   │ Tier 4          │ │ Tier 5          │
+│ • Low Latency   │ │ • Deep Analysis │   │ • target: none  │   │ • Nano Banana 2 │ │ • Whisper V3    │
+│ • Tool Routing  │ │ • Autonomous    │   │ • Zero Bleed    │   │ • Dedicated     │ │ • Gemini 3.1    │
+│ • Decoupled OCR │ │ • Isolated Run  │   │ • 30m Polling   │   │   Raster Head   │ │   Flash TTS     │
+└────────┬────────┘ └────────┬────────┘   └────────┬────────┘   └────────┬────────┘ └────────┬────────┘
+         │                   │                     │                     │                   │
+         ▼                   ▼                     ▼                     ▼                   ▼
+┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
+│ Primary:        │ │ Primary:        │   │ Primary:        │   │ Primary:        │ │ STT:            │
+│ deepseek/       │ │ deepseek/       │   │ deepseek/       │   │ google/         │ │ openai/         │
+│ deepseek-v4.1-  │ │ deepseek-v4.1-  │   │ deepseek-v4.1-  │   │ gemini-3.1-     │ │ whisper-large-  │
+│ flash-1017      │ │ flash-1017      │   │ flash-1017      │   │ flash-image     │ │ v3-turbo        │
+│                 │ │                 │   │                 │   │                 │                   │
+│ Fallbacks:      │ │ Fallbacks:      │   │ Fallbacks:      │   │ Fallbacks:      │ │ TTS:            │
+│ 1. v4.1-flash   │ │ 1. v4.1-flash   │   │ 1. v4.1-flash   │   │ 1. gemini-3.1-  │ │ google/         │
+│ 2. glm-5.3-flash│ │ 2. glm-5.3-flash│   │ 2. glm-5.3-flash│   │    flash-preview│ │ gemini-3.1-     │
+│ 3. qwen3.8-flash│ │ 3. qwen3.8-flash│   │ 3. qwen3.8-flash│   │ 2. gemini-3-pro-│ │ flash-tts       │
+│                 │ │                 │   │                 │   │    image        │ │ (FB: MiniMax 2.8│
+│ Vision (OCR):   │ │                 │   │                 │   │                 │ │  Speech HD)     │
+│ google/         │ │                 │   │                 │   │                 │ │                 │
+│ gemini-3.8-     │ │                 │   │                 │   │                 │ │                 │
+│ flash           │ │                 │   │                 │   │                 │ │                 │
+└─────────────────┘ └─────────────────┘   └─────────────────┘   └─────────────────┘ └─────────────────┘
+```
+
+---
+
+## Modality Specifications & Cost-Benefit Analysis
+
+### 1. Interactive Conductor & Agentic Reasoning (Tier 1)
+* **Primary (Fleet-Wide Standard as of 2026-10-17)**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
+  - **1017 Snapshot Upgrade (Pareto Jump in Quality & Price)**: Released on October 17, 2026, `deepseek/deepseek-v4.1-flash-1017` boosts the Artificial Analysis Intelligence Index from **24.7 to 39.5** (+60%) via Reinforcement Learning with Self-Feedback (RLSF) while cutting DeepInfra `fp8` pricing by an additional **30%** (`$0.098/M` prompt, `$0.294/M` completion, and **`$0.00294/M` `cacheRead`** vs. `$0.140/$0.420/$0.0042` on the 0910 baseline).
+  - **Provider Pinning & Tool-Compatibility Verification**:
+    - Pinned `order: ["deepinfra", "cloudflare", "inference-net", "fireworks", "parasail"]`, `quantizations: ["fp8", "bf16", "unknown"]`, and `ignore: ["morph", "baseten", "deepseek", "open-inference", "decart", "sail-research", "wafer"]`.
+    - **Cloudflare (#2 Promotion)**: Promoted ahead of InferenceNet and Parasail because Cloudflare delivers a 97% cache discount (`$0.003/M` `cacheRead`), 3× lower completion costs (`$0.60/M` vs. `$1.20/M` on Parasail), and a massive **943,718** completion token limit.
+    - **Fireworks (#4 Addition)**: Onboarded as a high-reliability enterprise tier-1 provider (`$0.30/M` prompt, `$1.20/M` completion, `$0.006/M` `cacheRead`, 100% uptime, 943k max completion tokens, full tool choice and structured outputs support).
+    - **Completion Token Headroom Guardrail (`maxTokens: 65536`)**: Because reasoning models share the completion token budget between internal chain-of-thought (`reasoningTokens`, typically 2k–4k tokens) and the actual tool-call payload, omitting `maxTokens` previously allowed OpenClaw/OpenRouter to default to an 8,192 token ceiling. During large multi-kilobyte code or website generation, hitting 8,192 tokens abruptly truncated tool-call JSON mid-payload (`Provider returned an incomplete or malformed tool call`). Explicitly configuring `maxTokens: 65536` and `params.max_tokens: 65536` provides ample headroom for complex reasoning and large outputs.
+    - Live OpenRouter routing forensics verified that `morph/fp8` is filtered out by OpenRouter's `Filter by Tool Compatibility` when `tools` are passed, `deepseek` (official) is filtered by account privacy guardrails (`Paid model training violation`), and `baseten` returns `HTTP 429`. Meanwhile, `deepinfra`, `cloudflare`, `inference-net`, `fireworks`, and `parasail` deliver 100% tool-calling reliability and sub-second to 3s latency.
+  - **Post-Trial Evaluation of `z-ai/glm-5.3-flash` (2026-09-13 to 2026-10-17)**:
+    - While `z-ai/glm-5.3-flash` was trialed as primary conductor, 17-day production telemetry (Oct 1–17, 2026) revealed **12 mid-stream tool-call aborts** (`stopReason: "error"`, `errorMessage: "Provider returned an incomplete or malformed tool call"`) across 5 agents (`main`, `grabengasse_media`, `koch`, `hebamme`, `doktor`) when generating nested `tool_call({ id, args })` payloads after `tool_describe`, as well as occasional thinking-only whitespace stalls (Issue 23) and long-context attention drift (Issues 22 & 25).
+    - Promoting `deepseek/deepseek-v4.1-flash-1017` to primary across all agents eliminates malformed tool-call stream aborts AND lowers total monthly cost by >50% compared to GLM 5.3 Flash.
+  - **Decoupled Vision Override**: `deepseek/deepseek-v4.1-flash-1017`, `deepseek/deepseek-v4.1-flash`, and `z-ai/glm-5.3-flash` are registered with `input: ["text"]` under `models.providers.openrouter.models` so that OpenClaw always routes user photos through `tools.media.image` (`google/gemini-3.8-flash`) for high-precision German OCR before handing enriched text to the chat model (Issue 29).
+* **Fallbacks**:
+  1. `openrouter/deepseek/deepseek-v4.1-flash` (`0910` baseline across 25+ providers, pinned to `deepinfra`, `streamlake`, `inference-net`, `digitalocean`, `venice`, `nebius`).
+  2. `openrouter/z-ai/glm-5.3-flash`: Secondary fallback pinned to `fp8`/`bf16` providers (`morph`, `near-ai`, `digitalocean`, `venice`, `streamlake`).
+  3. `openrouter/qwen/qwen3.8-flash`: Provider diversity.
+
+### 2. Autonomous Subagents (Tier 2)
+* **All Subagents & Specialists**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
+  - **Execution**: Spawned via `sessions_spawn` with a 300-second timeout. Provides high-precision reasoning, rock-solid nested JSON tool loops (`tool_search` -> `tool_describe` -> `tool_call`), and zero syntax drift.
+* **Fallback**: `openrouter/deepseek/deepseek-v4.1-flash` / `openrouter/z-ai/glm-5.3-flash` / `openrouter/qwen/qwen3.8-flash`.
+
+### 3. Background Daemon & Heartbeats (Tier 3)
+* **Configuration**: `heartbeat.model: "openrouter/deepseek/deepseek-v4.1-flash-1017"`, `heartbeat.target: none`, `heartbeat.lightContext: true`.
+* **Rationale**: Because background heartbeats have a 75%+ `cacheRead` share (`22.86M` `cacheRead` vs. `7.73M` uncached input over 17 days), DeepSeek V4.1 Flash 1017's \$0.00294/M cache read rate on DeepInfra reduces 17-day heartbeat cost to <\$0.90 while keeping prompt cache warm on the primary provider.
+
+### 4. Vision Input vs. Text-to-Image Generation (Tier 4)
+* **Crucial Distinction**:
+  - **Vision Input (`imageModel` vs. Dedicated Primary)**: Reading photos sent to Telegram (e.g. food ingredients, nutrition tables, plants, receipts). Note: OpenClaw only invokes `imageModel` if the primary model *lacks* image input modality. Since `z-ai/glm-5.3-flash` advertises image modality on OpenRouter, OpenClaw does not auto-divert to `imageModel`. Therefore, vision-intensive agents (specifically `koch`) must explicitly configure `primary: "openrouter/google/gemini-3.8-flash"` to leverage Google's SOTA OCR and dense-table recognition.
+  - **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here.
+* **Image Synthesis Engine**:
+  - **Primary**: `openrouter/google/gemini-3.1-flash-image` (Nano Banana 2).
+    - **Cost**: \$0.00006 per generated image + \$0.50/1M prompt.
+    - **Latency**: ~4.5 seconds.
+  - **Fallbacks**:
+    1. `openrouter/google/gemini-3.1-flash-image-preview`
+    2. `openrouter/google/gemini-3-pro-image` (Nano Banana Pro)
+  - **Decommissioned**: Gemini 2.5 is completely removed from all image configurations.
+
+### 5. Voice, Audio & Podcast Synthesis (Tier 5)
+* **Speech-to-Text (Audio Input)**:
+  - **Engine**: `openrouter/openai/whisper-large-v3-turbo`
+  - **Capabilities**: Fast, highly accurate transcription of German and English Telegram voice notes, with local gateway fallback.
+* **Text-to-Speech (Audio Dossiers & Podcasts)**:
+  - **Engine**: `google/gemini-3.1-flash-tts-preview` via `text-to-audio` skill.
+  - **Voices**:
+    - German Solo/Host: `Puck` (warm, natural, dynamic).
+    - German Deep/Calm: `Charon`.
+    - German Co-Host / Expert: `Aoede`.
+    - English: `Zephyr`.
+  - **Fallback**: `minimax/speech-2.8-hd` for studio-grade multi-speaker podcasts when Google TTS is throttled.
+
+### 6. Grounded Web Search & Vector Embeddings
+* **Live Search**: `perplexity/sonar-pro` via OpenRouter (replacing raw web scraping with grounded citations).
+* **Memory Embeddings**: `text-embedding-3-small` via OpenRouter for the Obsidian personal knowledge wiki.
+
+---
+
+## Context Lifecycle & Daytime Headroom Policy
+
+To balance the user's desire to retain full conversation context across the entire day while avoiding the 70-second latency penalty that uncompacted transcripts impose on OpenRouter providers:
+
+1. **Daytime Headroom (`maxActiveTranscriptBytes: "3mb"`)**:
+   The active transcript is permitted to retain the day's back-and-forth interactions so OpenClaw recalls earlier topics and instructions naturally.
+2. **Deterministic Nightly Compaction**:
+   Scheduled daily at **03:30 AM Europe/Berlin** (`ce39bbf8-b83f-453e-be53-04e27db27b78`):
+   ```bash
+   openclaw sessions compact "agent:main:main" --max-lines 250
+   ```
+   Executes in **2.1 seconds** using **0 LLM tokens**, ensuring every morning begins with a lean, fast ~15k token transcript.
+
+---
+
+## Open Clarifying Questions for Moritz
+
+To align on remaining edge cases before final sign-off:
+
+1. **Active Trial with GLM-5.3-Flash as Primary Conductor**:
+   - As requested, `z-ai/glm-5.3-flash` is now deployed as the Primary Conductor for interactive chat and autonomous subagents, equipped with the prompt guardrails against direct tool calls and XML leakage.
+   - `deepseek/deepseek-v4.1-flash` stands ready as the zero-latency primary fallback.
+   - We will observe real-world performance, tool adherence, and cost over the next several days.
+2. **Thinking / Reasoning Mode on Telegram**:
+   - Currently, `thinking: medium` is enabled, emitting 100–300 reasoning tokens before generating conversational text.
+   - *Question*: Would you like to disable thinking mode (`thinking: none`) for standard Telegram chat turns (for near-instant replies) and keep reasoning active only for subagents?
+3. **On-Demand Frontier Model**:
+   - With all standard workloads handled by Flash-tier models, do you ever want an explicit escape hatch to a frontier model (e.g. `claude-3.7-sonnet` or `gemini-3.1-pro-preview` triggered exclusively via `/expert <query>`)?
+
+---
+
+## OpenRouter Provider Qualification Criteria & Re-Evaluation Playbook
+
+Whenever a new model snapshot (such as `deepseek/deepseek-v4.1-flash-1017`) is evaluated or re-checked as new hosting providers join OpenRouter, every candidate provider MUST pass the following **7-point qualification gate** before being added to `extra_body.provider.order` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml):
+
+### 1. Hard Technical Gates (Must Pass All)
+1. **Quantization Precision (`fp8` / `bf16` Only)**:
+   - **Rule**: Must run `fp8` or `bf16` (or verified full-precision `unknown` from trusted cloud hosts like Cloudflare).
+   - **Reject**: Any `fp4`, `nvfp4`, `mxfp4`, or `int4` quantization (`decart`, `sail-research`, `wafer`, `open-inference`, `phala` fp4). 4-bit quantizations severely degrade nested JSON tool-call syntax and German grammar.
+2. **OpenRouter Routing Funnel & Guardrail Compatibility (`tools` + `tool_choice: "auto"` Pass)**:
+   - **Rule**: Must not be stripped by OpenRouter's internal routing filters (`Filter by Tool Compatibility` or `Filter by Guardrails`) when a request includes `tools: [...]` and `tool_choice: "auto"`.
+   - **Why**: Metadata in `/api/v1/models/.../endpoints` can claim `supports_tool_choice: true` even when OpenRouter's live router blocks the provider on actual tool calls (e.g. `morph/fp8` is stripped by `Filter by Tool Compatibility`, and `deepseek` official is stripped by `Paid model training violation` account privacy settings).
+3. **Nested MCP `tool_call` Serialization Integrity (2/2 Live Pass)**:
+   - **Rule**: Must cleanly emit `finish_reason: "tool_calls"` with valid, parseable nested JSON arguments (`{"id": "mcp:...", "args": {...}}`) across back-to-back test runs without mid-stream SSE aborts (`Provider returned an incomplete or malformed tool call`) or HTTP `429` upstream rate limits.
+4. **Context Window & Completion Headroom**:
+   - **Rule**: `context_length >= 262,144` (1,048,576 preferred) and `max_completion_tokens >= 131,072`.
+   - **Why**: Active daytime sessions with system prompts and tool schemas reach 20k–80k tokens.
+5. **Uptime & Reliability**:
+   - **Rule**: `uptime_last_1d >= 95%` and `uptime_last_30m == 100%`.
+
+### 2. Economic & Latency Ranking (How to Order Passing Providers)
+Among providers passing all 5 technical gates, rank them in `extra_body.provider.order` by **effective production cost** and **turn latency**:
+1. **Implicit Prompt Cache Read Rate (`input_cache_read`) — Highest Weight**:
+   - Because **73%–94%** of OpenClaw's token volume (both interactive multi-turn chats and 30m background heartbeats) consists of cached prompt reads (`cacheRead`), a provider with `supports_implicit_caching: true` and a **95%–98% cache discount** (`<= $0.005 / 1M` tokens, e.g. `deepinfra` at `$0.00294/M` or `streamlake` at `$0.0030/M`) is vastly cheaper in net monthly spend than a provider with slightly cheaper uncached input but weak or missing `cacheRead` discounts (e.g. `nebius` with `cache_read: null` or `modal`/`primeintellect` at `$0.030/M`).
+2. **Base Prompt & Completion Pricing**:
+   - Target `prompt <= $0.18 / 1M` and `completion <= $0.65 / 1M` for top-priority slots; allow higher-priced providers (`<= $0.30 / $1.20`) only in lower fallback slots (#3/#4) for outage resilience.
+3. **End-to-End Tool-Call Latency**:
+   - Target **1.0s – 5.5s** for a ~300-token reasoning + tool-call turn.
+
+### 3. One-Shot Live Verification Command (For Future Checks)
+Any agent (including a fast/lightweight model) can audit all currently available OpenRouter endpoints for `deepseek/deepseek-v4.1-flash-1017` in one step by running:
+
+```bash
+# 1. Inspect current endpoint metadata, pricing, cache_read, and 24h uptime:
+curl -s https://openrouter.ai/api/v1/models/deepseek/deepseek-v4.1-flash-1017/endpoints | \
+  jq -r '.data.endpoints[] | "\(.provider_name) | tag=\(.tag) | quant=\(.quantization) | ctx=\(.context_length) | prompt=\(.pricing.prompt) | comp=\(.pricing.completion) | cache=\(.pricing.input_cache_read) | uptime1d=\(.uptime_last_1d)"'
+
+# 2. Live-test candidate providers inside openclaw-0 with real nested tool_call payloads:
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c '
+import os, json, time, urllib.request
+api_key = os.environ["OPENROUTER_API_KEY"]
+tools = [{"type": "function", "function": {"name": "tool_call", "parameters": {"type": "object", "properties": {"id": {"type": "string"}, "args": {"type": "object"}}, "required": ["id", "args"]}}}]
+candidates = ["deepinfra", "streamlake", "inference-net", "venice", "digitalocean", "novita", "parasail", "cloudflare", "nebius"]
+for prov in candidates:
+    req = urllib.request.Request(
+        "https://openrouter.ai/api/v1/chat/completions",
+        data=json.dumps({
+            "model": "deepseek/deepseek-v4.1-flash-1017",
+            "messages": [{"role": "user", "content": "Call tool_call with id=\"mcp:homeassistant:homeassistant__ha_get_state\" and args={\"entity_id\": \"media_player.bad\"}"}],
+            "tools": tools, "tool_choice": "auto", "max_tokens": 250,
+            "provider": {"order": [prov], "allow_fallbacks": False}
+        }).encode(),
+        headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
+    )
+    t0 = time.time()
+    try:
+        with urllib.request.urlopen(req, timeout=20) as r:
+            b = json.loads(r.read().decode())
+            tc = b["choices"][0]["message"].get("tool_calls")
+            args = tc[0]["function"]["arguments"] if tc else "NONE"
+            print(f"PASS | {prov:15s} | {time.time()-t0:4.2f}s | actual={b.get(\"provider\")} | args={args}")
+    except urllib.error.HTTPError as e:
+        print(f"FAIL | {prov:15s} | HTTP {e.code} | {e.read().decode()[:140]}")
+    except Exception as e:
+        print(f"FAIL | {prov:15s} | ERR {e}")
+'
+```
+
+---
+
+## Consequences
+
+### Positive
+* **Unified Latest-Generation Stack**: Completely eliminates legacy Gemini 2.5 models; standardizes exclusively on DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, and Sonar Pro.
+* **100% Reliable Image Generation**: Correctly routes image generation requests to Nano Banana 2 (`google/gemini-3.1-flash-image`), completely eliminating HTTP 404 generation errors.
+* **Cohesive Voice & Audio Pipeline**: Clear architectural documentation of the Whisper V3 Turbo and Gemini 3.1 Flash TTS integration for dossiers and podcasts.
+* **Guaranteed Cost Ceiling**: Drops weekly expenditure from **\$12.55 to ~\$0.80–\$1.40** (~90% savings) through DeepSeek V4.1 Flash 1017's `$0.00294/M` cache reads, decoupled heartbeats, and nightly compaction.
+
+### Negative & Mitigations
+* **OpenRouter Provider Rollout Lag on Day-0 Model Releases**: Newly released snapshots initially launch with 5–10 providers before the full fleet (25+ providers) joins over subsequent days.
+  * *Mitigation*: Codified the 7-point provider qualification gate and one-shot test script above, and configured automatic model fallback to the `0910` snapshot (`openrouter/deepseek/deepseek-v4.1-flash`).

docs/adr/0003-everything-as-code-and-reproducible-infrastructure.md

@@ -0,0 +1,153 @@
+---
+status: "accepted"
+date: 2026-09-17
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0003: Everything as Code (EaC) and Reproducible Declarative Infrastructure
+
+## Context and Problem Statement
+
+The **Infrapuzzle** monorepo governs the complete digital infrastructure for Moritz Graf and family: bare-metal server provisioning (Haumdaucher node), Kubernetes workloads, DNS management, mail routing, smart home/IoT edge devices, and AI concierge workflows (OpenClaw and specialized domain subagents).
+
+Historically, home labs and personal infrastructure suffer from the "snowflake" problem:
+1. **Undocumented ClickOps**: Settings altered manually in cloud provider consoles (e.g. OVH DNS, router web UIs) or via web dashboards (Home Assistant, Grafana) become unversioned, undocumented, and fragile.
+2. **Ephemeral Terminal Interventions**: Ad-hoc `kubectl edit`, hot-patched container environments, or manual secret injections deviate from repository state, rendering disaster recovery nearly impossible.
+3. **Agent Prompt Fragmentation**: As AI agents proliferate (e.g. OpenClaw conductor, domain subagents like Hebamme, Koch, Grabengasse, Fitness, Recherche), defining prompts or tool configs across disparate Helm values, ConfigMaps, and local files leads to inconsistencies, stale behavior, and duplicate definitions.
+
+How do we establish a strict, maintainable architectural standard that ensures **everything is defined as code as far as feasible and reproducible**, while accommodating dynamic runtime state and secrets safely?
+
+---
+
+## Decision Drivers
+
+* **Determinism & Disaster Recovery**: The entire single-node cluster (`haumdaucher.de`), along with all services, DNS records, and edge devices, must be rebuildable from scratch purely from git repositories.
+* **Single Source of Truth**: Eliminate configuration drift across all operational domains.
+* **Exclusive Agent Storage**: Establish an unambiguous, single home for all AI agent personas, system prompts, role boundaries, and tool guidance.
+* **Declarative Secret Management**: Store all secrets in version control without risking plain-text credential leaks.
+* **Pragmatic Feasibility Boundary**: Clearly separate declarative desired state (in Git) from legitimate mutable runtime data (databases, dynamic chat histories, live OAuth session tokens) without breaking stateful applications.
+
+---
+
+## Considered Options
+
+1. **Option 1: Ad-Hoc / Best-Effort Setup (Status Quo Antipattern)**
+   * Store major K8s manifests in git, but allow manual tweaks in web consoles (OVH, Home Assistant, Shelly) and ad-hoc `kubectl` patches.
+   * *Drawbacks*: Severe drift, snowflake server syndrome, impossible disaster recovery, zero audit trail.
+
+2. **Option 2: Pure GitOps Immutability with Zero Runtime Mutation**
+   * Force every state mutation (including session compacting, OAuth token refreshes, and dynamic settings) to generate a Git commit.
+   * *Drawbacks*: Impractical for stateful workloads. High friction, commit noise, rate limits on git providers, and potential race conditions when multi-agent systems rapidly update local session caches.
+
+3. **Option 3: Comprehensive Everything-as-Code with Pragmatic State Governance (Selected)**
+   * Define 100% of infrastructure, workloads, DNS, IoT edge devices, and AI agents declaratively in Git.
+   * Enforce `git-crypt` for declarative secrets.
+   * Isolate mutable runtime state (databases, chat histories, active OAuth tokens) to persistent volumes (PVCs) governed by zero-drift merge mechanics (`mergeMode: merge`).
+   * Mandate `knowledge-wiki/agents/` as the **exclusive** home for all AI agent prompt definitions.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 3: Comprehensive Everything-as-Code with Pragmatic State Governance**.
+
+### 1. The Five Tiers of Everything as Code (EaC)
+
+```
+┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
+│                                    Infrapuzzle Monorepo (Git)                                    │
+└────────────────────────────────────────────────┬─────────────────────────────────────────────────┘
+                                                 │
+      ┌──────────────────┬───────────────────────┼───────────────────────┬──────────────────┐
+      ▼                  ▼                       ▼                       ▼                  ▼
+┌───────────┐      ┌───────────┐           ┌───────────┐           ┌───────────┐      ┌───────────┐
+│  Tier 1   │      │  Tier 2   │           │  Tier 3   │           │  Tier 4   │      │  Tier 5   │
+│Bare-Metal │      │Kubernetes │           │Cloud & DNS│           │Smart Home │      │Agents as  │
+│Bootstrap  │      │Workloads  │           │           │           │& IoT Edge │      │Code (Wiki)│
+├───────────┤      ├───────────┤           ├───────────┤           ├───────────┤      ├───────────┤
+│• Kubespray│      │• Helm     │           │• Terraform│           │• ESPHome  │      │• Exclusive│
+│• Ansible  │      │• Manifests│           │  (OVH DNS)│           │• Shelly   │      │  prompts  │
+│• init.sh  │      │• Operators│           │• Mail DNS │           │  Terraform│      │• Tool     │
+│           │      │  (OpenClaw│           │  (MX/SPF/ │           │• BLE &    │      │  bindings │
+│           │      │   CRDs)   │           │   DKIM)   │           │  Sensors  │      │• git-sync │
+└─────┬─────┘      └─────┬─────┘           └─────┬─────┘           └─────┬─────┘      └─────┬─────┘
+      │                  │                       │                       │                  │
+      ▼                  ▼                       ▼                       ▼                  ▼
+  Hetzner Node      Haumdaucher K8s         Public DNS &            Grabengasse 7        OpenClaw
+  Base OS Setup     Cluster Engine          Domain Records          Physical Devices     Conductor
+```
+
+#### Tier 1: Bare-Metal Server Provisioning (`bootstrap/`)
+* **Tooling**: Kubespray, Ansible, Bash (`init.sh`).
+* **Scope**: Operating system hardening, kernel modules, firewall rules, container runtime (containerd), and Kubernetes control plane setup on the single bare-metal node (Haumdaucher, `136.243.23.215`).
+* **Source of Truth**: `bootstrap/init.sh` and version-pinned Kubespray playbooks.
+
+#### Tier 2: Kubernetes Workloads & Operators (`k8s/`)
+* **Tooling**: Raw Kubernetes manifests, Helm 3 values, and Kubernetes Custom Resource Definitions (CRDs).
+* **Conventions**:
+  - Directory structure enforces namespace partitioning (`k8s/<namespace>/`).
+  - Workloads managed via declarative specifications (e.g. OpenClaw Operator `OpenClawInstance` CRD in `k8s/openclaw/openclaw-instance.secret.yaml`).
+  - Standardized ingress controller (`ingress-nginx` / `traefik`) with automated Let's Encrypt certificates (`cert-manager`).
+
+#### Tier 3: External Infrastructure, DNS & Mail (`terraform/`)
+* **Tooling**: Terraform with `ovh/ovh` provider.
+* **Scope**:
+  - Authoritative DNS zones (`haumdaucher.de`, `moritzgraf.de`, `ameliegraf.de`, and others).
+  - Google Workspace mail routing (MX, SPF TXT, DKIM, DMARC records).
+  - Cluster IP mapping to the bare-metal ingress endpoint.
+* **Execution**: Fully automated via `terraform plan` / `terraform apply` directly from version-controlled state.
+
+#### Tier 4: Smart Home & IoT Edge Hardware (`projects/shelly_taupunktluefter/`, `esphome/`, `monitoring/`)
+* **Tooling**: ESPHome, Shelly Terraform Provider, YAML configs.
+* **Scope**:
+  - Shelly relays and smart switches provisioned declaratively via Terraform.
+  - ESPHome firmware configurations for ESP32 Bluetooth proxies, Xiaomi BLE temperature/hygrometer sensors, and plant moisture monitors.
+  - All device configurations, pinouts, and network parameters committed to git; flashing performed deterministically from code.
+
+#### Tier 5: Agents-as-Code & Knowledge Systems (`knowledge-wiki/agents/`)
+* **Core Rule — Exclusive Single Source of Truth**:
+  > **All AI agent definitions (prompts, personas, operational boundaries, and tool calling instructions) MUST reside exclusively in [`knowledge-wiki/agents/<name>.md`](../../knowledge-wiki/agents/).**
+* **Strict Architecture**:
+  - Agent behavior is never hardcoded inside container images, ad-hoc Helm values, or untracked local directories.
+  - Adding or modifying an agent is done purely by creating or editing a markdown specification in the `knowledge-wiki/agents/` directory (e.g. `hebamme.md`, `koch.md`, `grabengasse.md`, `fitness.md`, `recherche.md`).
+  - Workloads (such as OpenClaw) ingest these definitions dynamically via `git-sync` sidecars or submodules, enabling instantaneous updates without requiring pod restarts or image rebuilds.
+  - If agent prompts or specifications are discovered outside of `knowledge-wiki/agents/`, they must be refactored and consolidated into this directory.
+
+---
+
+## The Feasibility Boundary & State Governance
+
+The user requirement explicitly dictates: *"we want to do everything as code as far as it is feasible and reproducible"*. To balance pure declarative code with stateful services, the boundary is defined as follows:
+
+| Dimension | Managed as Code (Git) | Managed on Persistent Storage (PVC) |
+| :--- | :--- | :--- |
+| **Workload Topology** | Pod specs, replicas, resource limits, CRDs, Helm releases. | — |
+| **Secrets & Keys** | Encrypted via `git-crypt` (`*.secret`, `*.secret.yaml`, `*.secret.values`). | Mounted into runtime memory/containers. |
+| **Agent Behavior** | System prompts, personas, roles, tool manifests in `knowledge-wiki/agents/`. | Dynamic short-term conversation context, user memory (`MEMORY.md`). |
+| **External Auth Tokens** | Initial static API keys and client credentials. | Volatile session tokens refreshed dynamically at runtime (e.g. Garmin OAuth tokens on PVC). |
+| **Data Storage** | Schema migrations, DB connection configs, backup cronjobs. | Raw databases (PostgreSQL, SQLite), media files, Nextcloud files. |
+| **Operator Integration** | Declarative base configurations (`spec.config.raw`). | Runtime mutations deep-merged safely via `mergeMode: merge` without overwriting runtime state. |
+
+### Operational Governance Rules
+
+1. **No Ad-Hoc `kubectl edit`**: Never modify cluster objects directly with `kubectl edit` or unrecorded commands. All modifications must be committed to git and applied via `kubectl apply -f <manifest>` or Helm.
+2. **Git-Crypt Security Enforcement**: Any file containing API keys, private passwords, or private identities must match the `.gitattributes` pattern (`*.secret*`) and be locked with `git-crypt`.
+3. **Reproducibility Guarantee**: In the event of total hardware loss, cloning `infrapuzzle` and unlocking `git-crypt` must be sufficient to run `bootstrap/init.sh`, apply Terraform, restore PVC backups from Velero/MinIO, and restore full operational state.
+
+---
+
+## Consequences
+
+### Positive
+* **100% Deterministic Disaster Recovery**: Eliminates mysterious configuration drift and allows full rebuilds of bare-metal, Kubernetes, DNS, and IoT systems.
+* **Unified AI Agent Lifecycle**: Developers and AI assistants edit agent personalities, prompts, and skills in a single, version-controlled markdown directory (`knowledge-wiki/agents/`).
+* **High Operational Velocity**: Changes to agent logic apply instantly via `git-sync` without container restarts or downtime.
+* **Audit Trail**: Every change to infrastructure, DNS, smart home switches, or agent system prompts has an author, timestamp, and commit message.
+
+### Negative & Mitigations
+* **Discipline Overhead**: Even minor configuration changes require a git commit and deployment workflow rather than a quick UI click.
+  * *Mitigation*: Tooling like OpenClaw Operator automated rolling updates and `git-sync` minimizes deployment friction to a single `git push`.
+* **Git-Crypt Dependency**: Operators and agents require the repository encryption key to modify secrets.
+  * *Mitigation*: Standardized symmetric key distribution and clear documentation in `k8s/AGENTS.md`.

docs/adr/0004-structured-personal-data-datastore.md

@@ -0,0 +1,141 @@
+---
+status: "accepted"
+date: 2026-10-06
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0004: Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)
+
+## Context and Problem Statement
+
+OpenClaw manages personal logging and family assistance for the Graf family across several daily domains:
+1. **Personal Nutrition:** Daily food intake, macro tracking (protein, carbs, sugar, fat, fiber, salt, calories) in `entities/nutrition-log/moritz/*.md`.
+2. **Baby Sleep & Routines:** Sleep logs, naps, night sleep, bottle feeds, and developmental milestones for Mila in `entities/people/mila-schlafprotokoll.md`.
+3. **Sports & Physiological Balance:** Activities, daily calorie expenditure (RMR + active), and net caloric surplus/deficit synced with Garmin Connect.
+4. **Culinary Knowledge:** Recipes with ingredient proportions, cooking steps, and hardware settings.
+5. **Points of Interest (POIs):** Local stores, farm stands (Hofläden), beer gardens, and playgrounds in Regensburg.
+
+Historically, this data was logged directly into Markdown tables and bulleted lists in the Knowledge Wiki (`knowledge-wiki/`). An analysis of existing log files revealed severe systemic failure modes:
+- **LLM Arithmetic Drift:** In `2026-09.md` and `2026-10.md`, the LLM consistently failed at daily macro sums (errors up to 75 kcal and 7.5 g protein per day). The derived "Vortags-Bilanz" (caloric deficit/surplus against Garmin) was calculated from incorrect totals.
+- **Structural and Chronological Degradation:** In `mila-schlafprotokoll.md`, dates drifted out of order (`29.09 → Sep Summary → 30.09 → 01.10 → 03.10 → 02.10 → 28.09`), and every night sleep was duplicated (once in the evening, once the next morning).
+- **Inability to Perform Spatial & Aggregate Queries:** Questions like *"What was Mila's average night sleep duration over the last 14 days?"* or *"Which beer gardens are within 500m of Grabengasse 7?"* required either multi-turn LLM hallucination or reading hundreds of lines of raw text.
+
+How do we establish a structured, reliable, and queryable data layer for OpenClaw that guarantees mathematical precision while upholding our Everything-as-Code (ADR 0003) and Git-as-Source-of-Truth architecture?
+
+---
+
+## Decision Drivers
+
+* **Mathematical Precision:** Zero LLM arithmetic. All summations, caloric balances, sleep durations, and nutrient rollups must be computed by a deterministic SQL engine.
+* **Git as the Single Source of Truth (ADR 0003):** No external database silos (PostgreSQL, standalone SQLite PVCs) that can drift from the git-backed wiki. All state must remain git-versioned, diffable, and merge-friendly.
+* **Separation of Concerns (Math vs. Narrative):** Delegate structured math, joins, and coordinates to DuckDB. Delegate natural-language understanding, nuance, packaging photo interpretation, and cooking advice to the LLM.
+* **Flexible Composability:** Foods can become ingredients in recipes, which can yield composite foods (e.g. wheat → baking mix → bread; tomatoes → sauce → pizza).
+* **Multi-Channel Presentation:** Auto-updated Markdown files for SilverBullet (`wiki.moritzgraf.de`), Telegram inline text with `chart-to-chat` PNGs, and mobile-friendly HTML dashboards on `hosting.haumdaucher.de/family/`.
+* **Zero-Tax Execution:** In-memory query execution in Python via `uv` without adding heavy runtime containers or pod memory overhead.
+
+---
+
+## Considered Options
+
+1. **Option 1: Stricter Markdown Prompts + Python Lint Rules**
+   * Keep Markdown tables; add linter checks in `okf_lint.py` to flag wrong sums.
+   * *Drawback*: Reactive detection rather than proactive prevention. Cannot do spatial searches or aggregate SQL queries.
+
+2. **Option 2: External Relational Database (PostgreSQL / PVC SQLite)**
+   * Deploy PostgreSQL or write to a binary SQLite database on the Kubernetes PVC.
+   * *Drawback*: Violates ADR 0003. Creates a separate data silo outside git, introduces backup synchronization risks, and binary files cannot be cleanly merged or diffed in Git.
+
+3. **Option 3: Pure Event Sourcing for All Data**
+   * Force every piece of data (including stores, recipes, and food labels) into an immutable event log.
+   * *Drawback*: Over-engineers static catalogs. Catalog entities (like places or recipes) become awkward to manage and query when forced into point-in-time event streams.
+
+4. **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git (Selected)**
+   * Store structured records in line-delimited JSON (`.jsonl`) inside the Knowledge Wiki repository (`data/`), using in-memory DuckDB as the calculation and query engine.
+   * Divide data into three natural archetypes: **Events**, **Entities**, and **Documents**.
+
+---
+
+## Decision Outcome
+
+Chosen Option: **Option 4: Three-Tier Archetype Architecture via DuckDB & JSONL in Git**.
+
+### 1. The Three Data Archetypes
+
+```
+┌────────────────────────────────────────────────────────────────────────┐
+│                        Three Data Archetypes                           │
+├─────────────────────────┬─────────────────────────┬────────────────────┤
+│ 1. EVENTS (Time-Series) │ 2. ENTITIES (Catalogs)  │ 3. DOCUMENTS (Prose│
+├─────────────────────────┼─────────────────────────┼────────────────────┤
+│ • data/events/nutrition/│ • data/entities/foods/  │ • entities/recipes/│
+│ • data/events/sleep/    │ • data/entities/recipes/│ • syntheses/places/│
+│ • data/events/feeding/  │ • data/entities/places/ │ • sources/*.pdf    │
+│ • data/events/body/     │ • data/entities/people/ │                    │
+│ [Point & Interval Logs] │ [Durable Slug Identifiers] [Markdown / Media]│
+└────────────┬────────────┴────────────┬────────────┴──────────┬─────────┘
+             │                         │                       │
+             ▼                         ▼                       │
+    ┌──────────────────────────────────────────────┐           │
+    │         In-Memory DuckDB SQL Engine          │           │
+    │  • Schema Validation (STRUCT & CHECK)        │           │
+    │  • SQL Aggregations & Daily Deficit Joins    │           │
+    │  • Spatial Queries (ST_DWithin / Haversine)  │           │
+    └──────────────────────┬───────────────────────┘           │
+                           │                                   │
+              ┌────────────┴────────────┐                      │
+              ▼                         ▼                      │
+     ┌──────────────────┐      ┌──────────────────┐            │
+     │ Telegram Output  │      │ Generated Views  │            │
+     │ • Text Summaries │      │ • Markdown Views ◄────────────┘
+     │ • chart-to-chat  │      │ • HTML Dashboards│ (Frontmatter:
+     └──────────────────┘      │   (/family/...)  │  datastore_ref)
+                               └──────────────────┘
+```
+
+1. **Events (Time-Series Logs):**
+   - Stored in `data/events/<domain>/<YYYY-MM>.jsonl`, sorted by `(start, id)`.
+   - Unified envelope: `id` (ULID), `kind`, `subject`, `actor`, `start`, `end` (supports open intervals where `end=NULL`), `estimated`, `meal`, `note`, `payload` (JSON cast to typed STRUCT), `source_ref`.
+2. **Entities (Structured Catalogs):**
+   - Stored in `data/entities/<collection>.jsonl`, sorted by slug `id`.
+   - `foods`: EU LMIV per-100g nutritional profile, ingredient lists, barcodes, density.
+   - `recipes`: Ingredients list (linking `foods` with exact grams or raw text), servings, calculated macros.
+   - `places`: Points of interest with `lat`, `lon`, category, address, tags, and linked documents.
+   - `people`: Family profile registry.
+3. **Documents (Prose & Media):**
+   - Stored as standard Markdown in `knowledge-wiki/`.
+   - Deterministic two-way link: Document YAML frontmatter contains `datastore_ref: "<collection>:<id>"`, while the entity JSONL record lists `documents: ["path/to/doc.md"]`.
+
+### 2. Dedicated Raw Garmin Mirror (`data/garmin/`)
+- Nightly zero-LLM Python sync script fetches raw Garmin API responses directly into:
+  - `data/garmin/activities/<YYYY-MM>.jsonl`
+  - `data/garmin/daily/<YYYY-MM>.jsonl`
+  - `data/garmin/sleep/<YYYY-MM>.jsonl`
+- DuckDB queries raw JSON directly with `read_json_auto()` and computes the daily caloric deficit/surplus by joining nutrition intake with Garmin total daily energy expenditure.
+
+### 3. Spatial & Geospatial Radius Lookups
+- DuckDB attempts to initialize the official `spatial` extension (`LOAD spatial;`).
+- If offline or binary installation fails, it transparently falls back to an internal pure-SQL Haversine macro `distance_m(lat1, lon1, lat2, lon2)`, ensuring zero-failure execution across any environment.
+
+### 4. Privacy & Hosting Routes (`hosting.haumdaucher.de`)
+- **Public (`/recipes/*`, `/places/*`, `/public/*`):** Publicly accessible recipe cards and shareable dossiers.
+- **Family (`/family/*`):** Protected by Traefik ForwardAuth linked to Authentik group `family` (nutrition dashboards, baby sleep charts).
+- **Private (`/private/*`):** Protected by Authentik for personal vital logs.
+
+---
+
+## Consequences
+
+### Positive
+- **Guaranteed Calculation Accuracy:** All calories, macronutrient totals, sleep statistics, and energy balances are computed in SQL. Zero LLM math errors.
+- **Git-Native & Mergable:** Text-based JSONL files partition data cleanly and produce clean 1-line git diffs.
+- **Offline & Low-Latency:** In-memory DuckDB reads months of JSONL in single-digit milliseconds.
+- **Preserved Human Readability:** SilverBullet (`wiki.moritzgraf.de`) and Obsidian continue to display updated Markdown tables generated directly from the database.
+
+### Negative & Mitigations
+- **Generated Markdown Views Must Not Be Hand-Edited:**
+  - *Mitigation*: Generated files include a prominent `<!-- DO NOT EDIT: GENERATED BY DATASTORE -->` warning comment.
+  - *Mitigation*: Git pre-commit hook and the weekly IT Guy audit verify that generated views match the underlying JSONL data.
+- **Dual-Step Agent Retrieval:** Agents execute `datastore query` for structured data, then `wiki_get` on linked documents if full narrative text is needed.
+  - *Mitigation*: Clear instructions and query helpers in the `datastore` skill prompt.

docs/adr/0005-personal-data-isolation-and-authentik-boundary.md

@@ -0,0 +1,156 @@
+---
+status: "accepted"
+date: 2026-10-11
+decision-makers: ["Moritz Graf", "Antigravity"]
+consulted: []
+informed: []
+---
+
+# ADR 0005: Personal Data Isolation, Authentik Authentication Boundaries, and Hosting Segregation
+
+## Context and Problem Statement
+
+On 2026-10-11, during the development of mobile-first PWA dashboards for baby monitoring (Mila Graf) and personal nutrition/Garmin energy balance (Moritz Graf), personal health, sleep, diaper, and nutrition data was temporarily made available at unauthenticated root URLs:
+- `https://hosting.haumdaucher.de/mila.html`
+- `https://hosting.haumdaucher.de/moritz.html`
+
+While [ADR 0004](0004-structured-personal-data-datastore.md) established the conceptual routing design (`/family/*` protected by Traefik ForwardAuth, public recipes at `/recipes/*`), a gap analysis revealed critical systemic vulnerabilities:
+1. **Unapplied Kubernetes Ingress:** The ForwardAuth Ingress (`minio-hosting-family`) declared in `k8s/minio/hosting-ingress.yaml` was never applied to the Kubernetes cluster.
+2. **Missing Authentik Identity Provider:** Authentik had no Proxy Provider or Application configured for `hosting.haumdaucher.de`, returning HTTP 404 on ForwardAuth checks.
+3. **Dual-Publishing Anti-Pattern:** Dashboard generator scripts (`update_website.py`, `update_moritz_website.py`) mirrored files to root (`/mila.html`, `/moritz.html`) for convenience, completely bypassing the planned `/family/` path boundary.
+4. **Lack of Automated Ingress & Storage Auditing:** No automated test or CI/Cron audit prevented personal data from being uploaded to public prefixes.
+
+How do we establish a permanent, verifiable, and automated security boundary that guarantees personal and family data can never be exposed publicly without authentication?
+
+---
+
+## Decision Drivers
+
+* **Zero Public Exposure for Personal Data:** Health, sleep, nutrition, body, and family data must NEVER be accessible without authentication.
+* **Defense-in-Depth:** Security must not rely on a single layer (e.g. secret URLs or developer memory). Protection must be enforced at identity provider, reverse proxy, storage bucket, and generator script levels.
+* **Declarative Infrastructure as Code (ADR 0003):** All Authentik providers, policies, groups, and Kubernetes Ingresses must be versioned in Git and applied via Terraform/Kubectl.
+* **Automated Guardrails & Fast Feedback:** Continuous audits must detect and quarantine any unauthorized files outside protected paths.
+* **Seamless PWA Experience:** The PWA capabilities (manifest, service worker, offline shell) must function within the authenticated cookie session without breaking ForwardAuth.
+
+---
+
+## Decision Outcome
+
+Chosen Strategy: **Multi-Layer Boundary Enforcement with ForwardAuth, Strict Storage Segregation, and Continuous Auditing**.
+
+### 1. Storage & URL Path Segmentation Architecture
+
+```
+Internet Request to https://hosting.haumdaucher.de
+                    │
+                    ▼
+          ┌───────────────────┐
+          │ Traefik Ingress   │
+          └─────────┬─────────┘
+                    │
+         Path Inspection & Priority
+                    │
+   ┌────────────────┴────────────────┐
+   │                                 │
+Path: /family/*                   Path: /* (Fallback)
+Priority: 100                     Priority: 10
+   │                                 │
+   ▼                                 ▼
+┌─────────────────────────┐       ┌─────────────────────────┐
+│ Authentik ForwardAuth   │       │ Public Ingress          │
+│ • Host: hosting.haumd...│       │ • ONLY /recipes/*       │
+│ • Allowed: 'family' &   │       │ • ONLY /showcase/*      │
+│   'infra-admins' groups │       │ • NO personal files     │
+└────────────┬────────────┘       └────────────┬────────────┘
+             │                                 │
+     HTTP 200 (Authed)                         │
+             │                                 │
+             ▼                                 ▼
+   ┌──────────────────┐               ┌──────────────────┐
+   │ MinIO Bucket     │               │ MinIO Bucket     │
+   │ key: family/*    │               │ key: public/*    │
+   │ (Protected Data) │               │ (Public Static)  │
+   └──────────────────┘               └──────────────────┘
+```
+
+#### Rule 1: Canonical Protected Prefix `/family/`
+- All dashboards containing personal, familial, or physiological metrics MUST reside under `/family/`:
+  - `https://hosting.haumdaucher.de/family/mila.html` (Baby sleep, feeds, diapers)
+  - `https://hosting.haumdaucher.de/family/moritz.html` (Personal nutrition, Garmin deficits, health)
+  - `https://hosting.haumdaucher.de/family/nutrition.html` (Family nutrition overview)
+  - `https://hosting.haumdaucher.de/family/sleep.html` (Sleep analytics)
+- Associated PWA assets (manifests, service workers) MUST also be scoped within `/family/` (e.g. `/family/mila.webmanifest`, `/family/sw.js`).
+
+#### Rule 2: Root Prefix `/` Strictly Restricted
+- The root of `hosting.haumdaucher.de` is restricted to public static assets (`/recipes/*`, `/showcase/*`).
+- Publishing scripts MUST NEVER write personal dashboards to the bucket root.
+
+---
+
+### 2. Declarative Authentik ForwardAuth Configuration
+
+Authentik configuration is codified in `terraform/authentik.tf`:
+1. **Proxy Provider (`authentik_provider_proxy.hosting_family`):**
+   - External Host: `https://hosting.haumdaucher.de`
+   - Mode: `forward_single`
+   - Bound to `authentik_outpost.embedded`
+   - Session duration: 30 days (`access_token_validity = "days=30"`)
+2. **Application (`authentik_application.hosting_family`):**
+   - Slug: `hosting-family`
+   - Protocol Provider: `authentik_provider_proxy.hosting_family.id`
+3. **Role-Based Policy Bindings:**
+   - Group `family` (Moritz & Amelie Graf): Granted access
+   - Group `infra-admins`: Granted access
+   - Unauthenticated requests are redirected (HTTP 302) to `https://auth.haumdaucher.de/application/o/authorize/...`
+
+---
+
+### 3. Kubernetes Ingress Deployment
+
+`k8s/minio/hosting-ingress.yaml` defines the two ingress tiers:
+- **`minio-hosting-family` (Priority 100):**
+  - Paths:
+    - `/family` (Prefix) -> Protected personal dashboards
+    - `/outpost.goauthentik.io` (Prefix) -> Authentik ForwardAuth OAuth callback & session negotiation
+  - Middlewares: `auth-authentik-forwardauth@kubernetescrd,minio-hosting-add-prefix@kubernetescrd`
+- **`minio-hosting` (Priority 10):**
+  - Path: `/` (Prefix)
+  - Middlewares: `minio-hosting-add-prefix@kubernetescrd`
+
+---
+
+### 4. Generator Scripts & Codebase Hardening
+
+1. **Elimination of Root Duplication:**
+   - `skills/datastore/scripts/update_website.py`: Removed all S3 upload and local sync calls to root `mila.html`. Target is strictly `family/mila.html`.
+   - `skills/datastore/scripts/update_moritz_website.py`: Removed S3 upload and local sync calls to root `moritz.html`. Target is strictly `family/moritz.html` (and `family/nutrition.html`).
+2. **PWA Scope Alignment:**
+   - Web App Manifests configure `start_url: "/family/mila.html"` and `scope: "/family/"`.
+   - Service workers are registered at `/family/sw.js`.
+3. **Telegram Link Consistency:**
+   - Agent prompts for `mila` (Topic 20) and `koch` (Topic 21) only broadcast the secure URL: `https://hosting.haumdaucher.de/family/...`.
+
+---
+
+### 5. Automated Continuous Leak Audit Guard
+
+An automated audit check runs continuously (via OpenClaw periodic audit and datastore maintenance):
+- Inspects the MinIO `hosting` bucket root (`prefix=""`).
+- Detects any forbidden files matching patterns: `*mila*`, `*moritz*`, `*nutrition*`, `*sleep*`, `*schlaf*`, `*health*`, `*diaper*`.
+- If detected:
+  1. Immediately deletes/quarantines the offending object from the public bucket root.
+  2. Raises a critical notification/event to the system log.
+
+---
+
+## Consequences
+
+### Positive
+- **Guaranteed Confidentiality:** Family and personal health data is physically impossible to access without passing Authentik ForwardAuth.
+- **Reproducible Infrastructure:** The entire authentication chain is codified in Terraform and Kubernetes manifests.
+- **Persistent Mobile PWA:** Authentik's 30-day session lifetime ensures seamless standalone mobile app operation without repetitive login prompts.
+- **Automated Safety Net:** Even if a developer or agent script mistakenly attempts to mirror a file to root, the automated audit deletes it.
+
+### Negative & Mitigations
+- **Authentication Step on First Mobile Launch:**
+  - *Mitigation*: When installing the PWA on iOS/Android, the user authenticates once via Authentik. The persistent session cookie maintains login across app launches.

docs/adr/README.md

@@ -0,0 +1,43 @@
+# Architecture Decision Records (ADRs)
+
+This directory documents the key architectural decisions made within the **Infrapuzzle** monorepo and its associated workloads (Kubernetes, OpenClaw AI agent, networking, DNS, and smart home systems).
+
+We follow the [Markdown Architectural Decision Records (MADR)](https://adr.github.io/madr/) format to capture intentional technical decisions, their context, considered options, trade-offs, and consequences.
+
+---
+
+## 📋 Record Index
+
+| ADR | Title | Status | Date | Area |
+| :--- | :--- | :---: | :---: | :--- |
+| [**0001**](0001-hybrid-mcp-tool-search-and-subagents.md) | **Hybrid MCP Tool Search and Subagent Delegation for AI Agents** | `accepted` | 2026-09-12 | OpenClaw / AI / MCP |
+| [**0002**](0002-openclaw-model-selection-tiering-and-economics.md) | **OpenClaw Multi-Tier Model Selection, Latency Optimization, and Inference Economics** | `proposed` | 2026-09-13 | OpenClaw / AI / Models |
+| [**0003**](0003-everything-as-code-and-reproducible-infrastructure.md) | **Everything as Code (EaC) and Reproducible Declarative Infrastructure** | `accepted` | 2026-09-17 | Architecture / Monorepo / EaC |
+| [**0004**](0004-structured-personal-data-datastore.md) | **Structured Personal Data Layer (DuckDB + JSONL in Knowledge Wiki)** | `accepted` | 2026-10-06 | OpenClaw / Data / Storage |
+| [**0005**](0005-personal-data-isolation-and-authentik-boundary.md) | **Personal Data Isolation, Authentik Authentication Boundaries, and Hosting Segregation** | `accepted` | 2026-10-11 | Security / Storage / Authentik |
+
+---
+
+## ✍️ How to Create a New ADR
+
+1. Copy the template structure into a new numbered markdown file:
+   ```bash
+   docs/adr/000X-short-title-in-kebab-case.md
+   ```
+2. Include the YAML frontmatter:
+   ```yaml
+   ---
+   status: "proposed | accepted | superseded | deprecated"
+   date: YYYY-MM-DD
+   decision-makers: ["Moritz Graf", "Antigravity"]
+   consulted: []
+   informed: []
+   ---
+   ```
+3. Clearly define:
+   - **Context & Problem Statement** (Why was this decision needed?)
+   - **Decision Drivers** (Constraints, costs, token limits, latency, maintainability)
+   - **Considered Options** (Alternatives evaluated)
+   - **Decision Outcome** (Chosen path with justification)
+   - **Consequences** (Positive effects, negative trade-offs, follow-up actions)
+4. Add the new record to the table in this `README.md`.

fritzbox/README.md

@@ -0,0 +1,51 @@
+# FritzBox Wireguard Setup
+
+This folder contains configuration and documentation for connecting your FritzBox router (home network) to the Kubernetes cluster via a Wireguard Site-to-Site VPN.
+
+## 1. Prerequisites
+- Your FritzBox must be running FRITZ!OS 7.50+ (Tested with 8.25).
+- The Kubernetes Wireguard endpoint (`k8s/wireguard`) must be deployed and running on `vpn.haumdaucher.de`.
+
+## 2. Connecting the FritzBox
+
+The FritzBox will be configured to connect to the cluster via a "LAN-to-LAN" coupling. Since we prefer "infrastructure as code", we have pre-generated the exact configuration file. For FritzBox specifically, this requires a manual import step.
+
+1. Locate the file `fritzbox-wireguard.secret.conf` in this directory. 
+2. Ensure you have unlocked `git-crypt` so you can read its decrypted contents.
+3. Open your FritzBox Web Interface (usually `http://fritz.box`).
+4. Navigate to **Internet > Permit Access > VPN (WireGuard)**.
+5. Click on **Add Connection** (or "Verbindung hinzufügen").
+6. Select **Connect networks or establish special connections** (Netzwerke koppeln oder spezielle Verbindungen herstellen).
+7. Ask if it has been set up on the other side -> choose **Yes** (or choose to upload a config file directly).
+8. Choose **Upload a configuration file** and select the decrypted `fritzbox-wireguard.secret.conf` file.
+9. Finish the setup.
+
+The FritzBox will immediately try to connect to `vpn.haumdaucher.de:51820`.
+
+## 3. Verifying the Connection
+
+### From the Kubernetes Side
+Connect to your cluster and check the Wireguard pod logs:
+```bash
+# Get the pod name
+kubectl get pods -n wireguard
+
+# Execute into the pod to check connection status
+kubectl exec -it <pod-name> -n wireguard -- wg show
+```
+You should see a peer connected and the `latest handshake` timestamp indicating a successful connection.
+
+### Bidirectional Ping Test
+1. **Cluster -> Home Network:**
+   Exec into any pod in your cluster (e.g., a toolbox or home-assistant pod) and ping a device on your local network:
+   ```bash
+   ping 192.168.10.1  # Ping your FritzBox local IP
+   ```
+2. **Home Network -> Cluster:**
+   From your laptop at home, try to ping a known K8s Service IP (e.g., `10.233.0.1` for kubernetes default service, or a specific pod IP):
+   ```bash
+   ping 10.233.0.1
+   ```
+
+## Backups
+Any future manual configurations, firmware backups, or notes related to the FritzBox should be stored within this `fritzbox/` folder. Use `.secret` extensions for any files containing sensitive tokens or passwords.

fritzbox/dyndns_ovh_setup.md

@@ -0,0 +1,48 @@
+# FritzBox OVH DynDNS (DynHost) Setup
+
+This document describes how to configure your FritzBox to automatically update an OVH DynHost record. 
+Since you have already set up `dyndns.moritzgraf.de` as a DynHost in your OVH control panel, the remaining step is to configure the FritzBox to push IP updates to OVH.
+
+## OVH DynHost Settings (Reference)
+The credentials for the DynHost are securely stored in the encrypted `dyndns_ovh.secret.yaml` file located in this directory. 
+Ensure you have unlocked `git-crypt` to view the `username` and `password`.
+
+*(Note: These are not your main OVH account credentials, but the specific credentials generated for the DynHost record!)*
+
+## FritzBox Configuration Steps
+
+1. Log into your FritzBox Web Interface (usually `http://fritz.box`).
+2. Navigate to **Internet > Permit Access > DynDNS**.
+3. Check the box **Use DynDNS** (DynDNS benutzen).
+4. From the **Dynamic DNS provider** dropdown, select **User-defined** (Benutzerdefiniert).
+5. Fill out the fields exactly as follows:
+
+   - **Update URL**:
+     ```text
+     https://www.ovh.com/nic/update?system=dyndns&hostname=<domain>&myip=<ipaddr>
+     ```
+     *(Make sure to copy exactly as written. The FritzBox will automatically replace `<domain>` and `<ipaddr>` with the actual values.)*
+
+   - **Domain name**: 
+     ```text
+     dyndns.moritzgraf.de
+     ```
+
+   - **Username**: 
+     *(Get the `username` value from `dyndns_ovh.secret.yaml`)*
+
+   - **Password**: 
+     *(Get the `password` value from `dyndns_ovh.secret.yaml`)*
+
+6. Click **Apply** (Übernehmen) to save the settings.
+
+## Verifying the Update
+After saving, the FritzBox should attempt an immediate update.
+1. In the FritzBox interface, go to **Overview** (Übersicht).
+2. Look under the **Connections** (Verbindungen) section. It should say:
+   `Dynamic DNS: enabled, <domain>, Status: successfully registered.`
+3. You can also manually ping `dyndns.moritzgraf.de` from an external network to verify it points to your router's current public IP.
+
+## Important Note on CGNAT and IPv6
+Since your ISP uses Carrier Grade NAT (CGNAT) for IPv4, your public IPv4 address might not be directly reachable from the outside. 
+OVH's legacy DynHost system primarily updates A (IPv4) records. If you specifically need external access to your home network via this dyndns domain, the incoming connection might fail on IPv4 due to CGNAT. (This is why we set up the Wireguard Site-to-Site VPN to the K8s cluster!)

k8s/AGENTS.md

@@ -0,0 +1,213 @@
+# AGENTS.md
+
+> [!NOTE]
+> This file describes the constraints and conventions for the `k8s` directory, which contains deployments for the **haumdaucher.de** Kubernetes cluster.
+
+## Project Overview
+This directory contains the Kubernetes manifests and Helm charts for a single-node Kubernetes cluster (Haumdaucher).
+*   **Domain**: `*.haumdaucher.de`
+*   **Orchestration**: Self-managed Kubernetes (single node).
+*   **Ingress**: `ingress-nginx`
+*   **SSL**: `cert-manager` (LetsEncrypt)
+
+## Directory Structure
+*   **Strict Namespace Mapping**: **Every top-level subdirectory in `k8s/` MUST correspond strictly to a Kubernetes namespace.**
+    *   Example: `mailu/` contains resources for the `mailu` namespace, `grabengasse/` for `grabengasse`, `haumdaucher/` for `haumdaucher`, etc.
+    *   **NO non-namespace folders**: Loose utility scripts belong in the root `scripts/` directory, application source code belongs in `projects/`, and general documentation belongs in namespace READMEs or `docs/`.
+*   **Documentation**: `README.md` is the **authoritative source** for deployment commands. Always check it before running commands.
+
+## Code Style & Conventions
+*   **Helm Version**: Helm 3 (`helm`) is used.
+*   **Implementation Order**: Top-down.
+*   **Naming**: Namespaces matches folder names.
+*   **Formatting**: Standard YAML conventions.
+
+## Security & Secrets
+> [!IMPORTANT]
+> **Git-Crypt is enforced.**
+> Do not touch encrypted files unless you have the key and know how to unlock them.
+
+**Encrypted File Patterns**:
+*   `*.secret`
+*   `*.secret.yaml`
+*   `*.secret.values`
+*   `*.secret.sh`
+
+## Remote Access
+It is possible to execute commands on the remote Linux node for information retrieval or troubleshooting.
+
+*   **Host**: `haumdaucher.de`
+*   **User**: `moritz` (local user)
+*   **Privileges**: Use `sudo` to gain root privileges.
+
+> [!CAUTION]
+> **SSH Identity Required**: The agent cannot enter an SSH passphrase.
+> If SSH commands fail with authentication errors, request the user to run `ssh-add` locally to load their identity.
+
+### Command Execution
+You can execute commands remotely via SSH. This is useful for checking node-level resources (memory, disk, etc.) that `kubectl` might not expose directly.
+
+**Example: Check Memory Usage**
+```bash
+ssh moritz@haumdaucher.de "free -h"
+```
+
+**Example: Check Disk Usage (with sudo)**
+```bash
+ssh -t moritz@haumdaucher.de "sudo df -h"
+```
+*Note: The `-t` flag forces pseudo-terminal allocation, which is often required for `sudo` prompts.*
+
+
+## Deployment Instructions
+**Always consult `README.md` first.** Deployments vary between Helm charts and raw manifests.
+
+### Common Patterns
+*   **Helm**:
+    ```bash
+    helm upgrade --install <release> <chart> -n <namespace> -f <values-file>
+    ```
+*   **Kubectl**:
+    ```bash
+    kubectl apply -f <folder>/<file>.yaml
+    ```
+
+### Post-Implementation Verification
+> [!IMPORTANT]
+> **Verification Workflow**:
+> After a new implementation or configuration change, always:
+> 1.  Run `kubectl apply -f <file>.yaml`.
+> 2.  Run `kubectl rollout restart deployment <deployment-name> -n <namespace>` if applying a ConfigMap/Secret that a deployment depends on.
+> 3.  Wait for 30 seconds to allow pods to roll over.
+> 4.  Check logs using `kubectl logs -n <namespace> -l <label> --tail=100`.
+>
+> The agent **must always ask the user** for permission to execute this verification workflow after making changes.
+
+### Operational Tasks
+*   **Cleanup Error Pods**:
+    ```bash
+    kubectl get pods | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod
+    ```
+
+## Ingress Configuration
+Ingress resources **must** follow these strict conventions to work with the cluster's ingress controller (`traefik`) and certificate manager (`cert-manager`).
+
+### Annotations
+All Ingress resources must include:
+```yaml
+annotations:
+  kubernetes.io/ingress.class: "traefik"
+  cert-manager.io/cluster-issuer: "letsencrypt-prod"
+  kubernetes.io/tls-acme: "true"
+  # Standard nginx tweaks (if using dual class) or traefik configurations
+  nginx.ingress.kubernetes.io/proxy-body-size: "0"
+  nginx.ingress.kubernetes.io/ssl-redirect: "true"
+  nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+```
+
+### Hostnames & TLS
+*   **Domain**: Use a subdomain of `haumdaucher.de` or `moritzgraf.de`.
+*   **TLS Secret Name**: Must use **hyphens** instead of dots.
+*   **Pattern**: `<subdomain>-<domain>-<tld>`
+*   **Example**: `n8n.moritzgraf.de` -> `n8n-moritzgraf-de`
+
+### Example
+```yaml
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - n8n.moritzgraf.de
+      secretName: n8n-moritzgraf-de
+  rules:
+    - host: n8n.moritzgraf.de
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: n8n
+                port:
+                  number: 5678
+```
+
+## Storage / Persistence
+The cluster uses **OpenEBS** for dynamic local storage provisioning.
+
+### PersistentVolumeClaims (PVC)
+*   **Provisioner**: `openebs.io/local` (or similar, managed via `openebs-hostpath`).
+*   **StorageClass**: `openebs-hostpath`.
+*   **AccessMode**: Typically `ReadWriteOnce` (RWO) as it's local storage.
+
+To request storage, simply create a PVC or configure Helm charts to use the default storage class (or explicitly `openebs-hostpath`).
+
+```yaml
+kind: PersistentVolumeClaim
+apiVersion: v1
+metadata:
+  name: my-data
+spec:
+  storageClassName: openebs-hostpath
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+```
+
+## Deployment Constraints
+*   **Resources**: Always define `requests` and `limits` for CPU and Memory to ensure fair scheduling on the single node.
+*   **Namespaces**: Every application gets its own namespace.
+*   **Secrets**: Encrypt all secrets using `git-crypt`.
+
+## Network Setup (VPN & Routing)
+
+The Kubernetes cluster (Haumdaucher) is connected to the home network via a dedicated WireGuard VPN tunnel.
+
+```mermaid
+graph TD
+    subgraph "Kubernetes Cluster (haumdaucher.de - 136.243.23.215)"
+        subgraph "monitoring namespace"
+            Exporter[fritzbox-exporter Pod]
+        end
+        HostNet[Host Network Namespace]
+        WgPod[wireguard Pod hostNetwork: true]
+    end
+
+    subgraph "Home Network (192.168.10.0/24)"
+        FB[FRITZ!Box Gateway - 192.168.10.1]
+        Taupi[Taupi Fan Shelly - 192.168.10.168]
+    end
+
+    Exporter -- "Queries 192.168.10.1" --> HostNet
+    HostNet -- "Route: 192.168.10.0/24 via wg0" --> WgPod
+    WgPod -- "WireGuard VPN Tunnel (51820/UDP)" --> FB
+    FB -- "Accesses local subnet" --> Taupi
+```
+
+### Components & Routing
+1. **Home Network**: `192.168.10.0/24`. Contains the home devices (e.g., Shelly plug at `192.168.10.168`) and the gateway FRITZ!Box at `192.168.10.1`.
+2. **Kubernetes Cluster Network**: Pod subnets (`10.233.64.0/24` etc.) running on the remote public server (`136.243.23.215`).
+3. **WireGuard VPN Link (`wg0`)**:
+   - The `wireguard` pod in the `wireguard` namespace is configured with `hostNetwork: true`. This exposes the `wg0` interface directly in the host's root network namespace.
+   - The cluster host has a static tunnel IP of `192.168.11.1/24` on `wg0`.
+   - The FRITZ!Box acts as the active peer client, initiating the connection to the host node at `136.243.23.215:51820`.
+   - The host routing table directs home network traffic over the tunnel:
+     ```bash
+     192.168.10.0/24 dev wg0 scope link
+     ```
+   - Pods inside the cluster (like `fritzbox-exporter`) query `192.168.10.1`. The traffic is forwarded by the host's default CNI routing to the host's network namespace, which matching the `192.168.10.0/24` subnet route and sends it over `wg0` to the FRITZ!Box.
+
+### FritzBox Exporter Egress Reject Workaround (Option B)
+
+**The Problem**:
+When the `fritzbox-exporter` pod queries the FRITZ!Box TR-064 API at `192.168.10.1` from outside the home LAN subnet (using the transit WireGuard subnet IP `192.168.11.1`), the FRITZ!Box's internal security policy redirects the client to its public WAN IP (e.g. `212.42.244.122`) for portal login/authentication (`login_sid.lua`). 
+When the dynamic WAN IP of the FRITZ!Box changes or the WAN ports are closed, these queries to the public IP timeout. Because Go's default HTTP client doesn't enforce a timeout, the exporter hangs for Go/Linux's default TCP connection handshake timeout of **2 minutes (120 seconds)**. This blocks all other metric scrapes and causes Prometheus target scrape timeouts.
+
+**The Solution**:
+Rather than disabling CPU/RAM/temperature metrics (`-nolua`) or modifying the remote FRITZ!Box WireGuard subnet configuration, we block the exporter pod from reaching the public WAN IP.
+- We add an `initContainer` in `fritzbox-exporter.yaml` running with `NET_ADMIN` privileges.
+- The `initContainer` installs `iptables` and adds rules directly to the pod's shared network namespace.
+- It permits traffic to local private subnets (RFC1918 ranges) but rejects all TCP egress to public WAN IPs on port 80/443 with a `TCP RST` (Reset) immediately.
+- This causes redirected authentication requests to fail in milliseconds instead of hanging for 2 minutes. The exporter immediately falls back to scrape TR-064 metrics (WAN sync speed, bytes) successfully, avoiding Prometheus scrapes timeouts.

k8s/README.md

@@ -34,14 +34,33 @@ kubcetl edit -n mailu secret sh.helm.release.v1.mailu.v8
 
 # Deployment (non persistent stuff)
 
-## [ingress-nginx](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx)
+## [Traefik Ingress Controller](https://github.com/traefik/traefik-helm-chart) (Replaces retired ingress-nginx)
 
-Apply with helm:
+Apply Dual IngressClass configuration (supporting both legacy `nginx` and new `traefik` names on the same controller):
 
 ```bash
-helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+kubectl apply -f traefik/ingress-classes.yaml
+```
+
+Apply with Helm:
+
+```bash
+helm repo add traefik https://traefik.github.io/charts
 helm repo update
-helm upgrade --install --create-namespace ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx -f ingress-nginx/ingress-nginx.yaml
+helm upgrade --install traefik traefik/traefik -n kube-system -f traefik/traefik-values.yaml
+```
+
+### [Legacy ingress-nginx (RETIRED)](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx)
+
+The community ingress-nginx is **retired** (EOL March 2026). During hot-swap, it was scaled down to 0 replicas. Keep it scaled down for a few days before deleting:
+
+```bash
+# To scale down (executed during migration)
+kubectl scale daemonset ingress-nginx-controller -n ingress-nginx --replicas=0
+
+# Permanent cleanup (execute after 3-7 days safety buffer)
+# helm uninstall ingress-nginx -n ingress-nginx
+# kubectl delete ns ingress-nginx
 ```
 
 ## [cert-manager](https://cert-manager.io/docs/tutorials/acme/ingress/)
@@ -102,19 +121,64 @@ Backup tool. See chart [README](https://github.com/vmware-tanzu/helm-charts/blob
 ```sh
 helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
 helm repo update
-helm upgrade --install --create-namespace --namespace velero -f ./velero/velero.secret.yaml velero vmware-tanzu/velero --version 5.3.0
+# Apply CRDs first (required on upgrades across major versions)
-kubectl create secret generic rclone-config --from-file=./velero/rclone.secret
+helm show crds vmware-tanzu/velero --version 12.1.0 | kubectl apply -f -
-kubectl apply -f velero/dropbox_sync.yml
+helm upgrade --install --create-namespace --namespace velero -f ./velero/velero.secret.yaml velero vmware-tanzu/velero --version 12.1.0
-# #
+# Secrets for rclone (minio + gdrive)
+kubectl create secret generic rclone-config --from-file=./velero/rclone.secret -n velero
+kubectl create secret generic gdrive-sa-key --from-file=gdrive_sa.json.secret=./velero/gdrive_sa.json.secret -n velero
+# Deploy GDrive sync cronjob
+kubectl apply -f velero/gdrive_sync.yml
+# Teardown
 helm delete velero -n velero
 kubectl delete ns velero
 ```
 
-A manual backup may be created executing the following command. **Note: Keep backuped namespaces in sync with config from helm chart!!!**
+A manual backup may be created executing the following command. **Note: Keep backed-up namespaces in sync with config from helm chart!!!**
 
 ```sh
 DATE=$( date +%Y%m%d )
-velero backup create $DATE --include-namespaces datalab,development,nextcloud,tt-rss,mailu --wait
+velero backup create $DATE --include-namespaces coder,home-assistant,n8n,influxdb,development,nextcloud,openclaw,auth,zigbee --wait
+```
+
+### Backup Retention Policy
+
+Three schedules run automatically, providing a tiered retention policy. Kopia (used for fs-backup) deduplicates at the block level, so each backup only stores changed data — incrementals come for free.
+
+| Schedule | Cron | TTL | Keeps |
+|---|---|---|---|
+| `velero-daily-backup` | `0 2 * * *` | 8 days | Last 7 daily backups |
+| `velero-weekly-backup` | `0 3 * * 0` | 43 days | Last 6 weekly backups (Sundays) |
+| `velero-monthly-backup` | `0 4 1 * *` | 183 days | Last 6 monthly backups (1st of month) |
+
+```sh
+# Check all schedules and their last run
+velero schedule get
+
+# Check current backups and their expiry
+velero backup get
+```
+
+
+## authentik (auth)
+
+Authentik is the central SSO and identity provider deployed at [auth.haumdaucher.de](https://auth.haumdaucher.de).
+See full documentation in [k8s/auth/README.md](./auth/README.md) and [k8s/auth/MACHINE_AUTH.md](./auth/MACHINE_AUTH.md).
+
+```sh
+# 1. Create namespace & CNPG PostgreSQL cluster
+kubectl create namespace auth
+kubectl apply -f auth/authentik-db.yaml
+
+# 2. Deploy Authentik via Helm
+helm repo add authentik https://charts.goauthentik.io
+helm repo update
+helm upgrade --install authentik authentik/authentik \
+  --namespace auth \
+  -f auth/authentik.secret.values.yaml
+
+# 3. Deploy Traefik ForwardAuth Middleware
+kubectl apply -f auth/traefik-forwardauth-middleware.yaml
 ```
 
 ## coder
@@ -160,8 +224,20 @@ USER='moritz'
 PASSWORD='xxx'
 docker run --entrypoint htpasswd --rm registry:2 -Bbn $USER $PASSWORD
 # #
-helm upgrade --install --create-namespace docker-registry stable/docker-registry -n development -f development/registry.secret.yaml
+# 1. Add the modern repo
-##kubectl apply -f development/registry.secret.yaml
+helm repo add twuni https://twuni.github.io/docker-registry.helm
+helm repo update
+
+# 2. Install the new one
+helm upgrade --install docker-registry twuni/docker-registry \
+  --namespace development \
+  --create-namespace \
+  -f development/registry.secret.yaml
+
+### 3. Verification
+Once deployed, verify you can login from your local machine:
+```bash
+docker login registry.haumdaucher.de -u moritz
 ```
 
 ### creating docker-pull-secret
@@ -169,14 +245,14 @@ helm upgrade --install --create-namespace docker-registry stable/docker-registry
 Create credentials secret [according to docu](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-by-providing-credentials-on-the-command-line):
 
 ```sh
-namespaces="datalab moritz web"
+namespaces="datalab moritz web haumdaucher"
 # the following is ONLY required to update the secret file!!
 for i in $( echo $namespaces ) ; do
   kubectl create secret docker-registry registry-haumdaucher-de \
     -n $i \
     --docker-server=registry.haumdaucher.de \
     --docker-username=moritz \
-    --docker-password='xxx' \
+    --docker-password='xxxxxxx' \
     --docker-email=moritz@moritzgraf.de \
     --dry-run -o yaml > ./${i}/docker-pull.yaml.secret
 done
@@ -310,12 +386,21 @@ helm repo add prometheus-community https://prometheus-community.github.io/helm-c
 helm repo update
 helm upgrade --install --create-namespace prometheus-operator prometheus-community/kube-prometheus-stack -n monitoring -f monitoring/prometheus-operator.secret.yml --version 56.6.1
 # alert configuration
-NAMESPACES_TO_ALERT=( kube-system monitoring cert-manager datalab ingress-nginx mailu minio velero web openebs )
+NAMESPACES_TO_ALERT=( kube-system monitoring cert-manager datalab ingress-nginx mailu minio velero web openebs n8n influxdb home-assistant )
 for i in "${NAMESPACES_TO_ALERT[@]}"; do
   kubectl apply -f monitoring/alertmanagerconfig.secret.yaml -n $i
 done
 ```
 
+### Checking alerts history
+
+A Python script is available to retrieve and analyze the Prometheus alert history (for the last 14 days) by querying the cluster's Prometheus API through the Grafana pod:
+
+```bash
+python3 k8s/monitoring/scripts/analyze_alerts.py
+```
+
+
 ### influxdb
 
 Used to store hass data long term.
@@ -352,7 +437,7 @@ helm repo update
 helm upgrade --install robusta robusta/robusta -n robusta -f ./robusta/robusta.yaml
 ```
 
-## gitea
+## gitea (old, no longer existant, do not use)
 
 In case my PRs have been accepted this is no longer necessary:
 
@@ -370,6 +455,23 @@ helm upgrade --install gitea ./gitea-chart -n development -f development/gitea.s
 helm upgrade --install gitea-phpmyadmin bitnami/phpmyadmin -n development -f development/gitea-phpmyadmin.yml
 ```
 
+## gitea (new set up)
+
+Chart used: [https://gitea.com/gitea/helm-gitea](https://gitea.com/gitea/helm-gitea)
+
+History: Manually deleted the old git server and reapplied a new one.
+
+```sh
+# 1. Add/Update Repo
+helm repo add gitea-charts https://dl.gitea.com/charts/
+helm repo update
+
+# 2. Install
+helm upgrade --install gitea gitea-charts/gitea \
+  --namespace development \
+  -f development/gitea.secret.yml
+```
+
 ### backup & restore
 
 See the backup cronjob in the `/backup/` folder.
@@ -705,11 +807,102 @@ Using helm chart from [https://github.com/8gears/n8n-helm-chart](https://github.
 kubectl create ns n8n
 helm upgrade --cleanup-on-fail --install mop-n8n \
 oci://8gears.container-registry.com/library/n8n \
---namespace n8n --values n8n/n8n.secret.yml --version 1.0.15
+--namespace n8n --values n8n/n8n.secret.yml --version 2.0.1
 ```
 
 To verify installation was correcet, use the following command:
 
-```
+```sh
 helm get manifest mop-n8n -n n8n | less
 ```
+
+
+## n8n-fabi
+
+> [!IMPORTANT]
+> Notice: This deployment is currently not deployed.
+
+```sh
+# kubectl create ns n8n-fabi
+# helm upgrade --cleanup-on-fail --install fabi-n8n \
+# oci://8gears.container-registry.com/library/n8n \
+# --namespace n8n-fabi --values n8n-fabi/n8n-fabi.secret.yml --version 2.0.1
+```
+
+
+## LLM (Ollama)
+Hosts a local LLM (Llama 3 8B) for inference.
+*   **Deploy**: `kubectl apply -f llm/ollama.yaml`
+*   **Verification**: Check pods in `llm` namespace.
+
+## OpenClaw
+An autonomous AI agent platform.
+1.  **Create Namespace**: `kubectl apply -f openclaw/namespace.yaml`
+2.  **Configure Secrets**:
+    *   Edit `openclaw/openclaw.secret.yaml`.
+    *   **Gemini**: Replace `change-me` with your Gemini API Key.
+    *   **Telegram**: Replace `telegram-bot-token` with your Bot Token.
+    *   **Gateway**: Token is pre-filled (randomly generated). Change if desired.
+    *   **Auth**: Generate `auth` string using `htpasswd -nb <user> <password>` and replace the placeholder.
+    *   **Encrypt**: Ensure the file is encrypted with `git crypt` before committing!
+3.  **Deploy**: `kubectl apply -f openclaw/openclaw.secret.yaml`
+4.  **Access**: `https://openclaw.haumdaucher.de` (Authentication required)
+
+## Zigbee2MQTT
+
+Zigbee2MQTT manages Zigbee devices via the SLZB-MR1U coordinator. It is deployed via the official Helm chart in the `zigbee` namespace.
+
+```bash
+helm repo add zigbee2mqtt https://charts.zigbee2mqtt.io
+helm repo update
+# Apply the secrets first for the Traefik basic auth middleware
+kubectl apply -f zigbee/zigbee2mqtt.secret.yaml
+# Install or upgrade the release
+helm upgrade --install zigbee2mqtt zigbee2mqtt/zigbee2mqtt --create-namespace -n zigbee -f zigbee/zigbee2mqtt.secret.values.yaml
+```
+
+To modify configurations (like `devices`, `throttle`, or `mqtt`), edit `zigbee/zigbee2mqtt.secret.values.yaml` and re-run the `helm upgrade` command. The chart automatically handles merging these values into the persistent volume on startup.
+
+## Grabengasse 7 Website (`grabengasse7.de`)
+
+Grabengasse 7 home website and backend service deployed in the `grabengasse` namespace. Exposes outdoor weather metrics from the Shelly WS90 station with interactive 24h charts.
+
+### Build and Push Container Images
+
+```bash
+# Login to local registry (if not already logged in)
+docker login registry.haumdaucher.de -u moritz -p "PPf10musSh!!"
+
+# Build and push images via root projects/grabengasse Makefile
+cd projects/grabengasse
+make build push
+
+# Or build individual components:
+cd projects/grabengasse/frontend && make all
+cd ../backend && make all
+```
+
+### Deploy to Kubernetes
+
+```bash
+# Create namespace and registry secret
+kubectl apply -f k8s/grabengasse/namespace.yaml
+kubectl apply -f k8s/grabengasse/registry.secret.yaml
+
+# Deploy backend, frontend, and ingress
+kubectl apply -f k8s/grabengasse/backend.yaml
+kubectl apply -f k8s/grabengasse/frontend.yaml
+kubectl apply -f k8s/grabengasse/ingress.yaml
+```
+
+### Verification
+
+```bash
+# Verify pod statuses
+kubectl get pods -n grabengasse
+
+# Check backend health
+curl -k https://grabengasse7.de/api/v1/healthz
+curl -k https://grabengasse7.de/api/v1/weather/current
+```
+

k8s/auth/MACHINE_AUTH.md

@@ -0,0 +1,65 @@
+# Machine & AI Assistant Authentication Guide
+
+This guide describes how to configure programmatic, machine-to-machine, and AI assistant authentication across services protected by **Authentik** and **Traefik ForwardAuth**.
+
+---
+
+## 🎯 Use Case
+When an AI assistant (or automation script, CLI tool, or cron job) needs to:
+* Inspect Zigbee2MQTT logs or device states via `https://zigbee2mqtt.moritzgraf.de`
+* Scrape or query metrics from `https://prometheus.haumdaucher.de`
+* Trigger or query APIs protected behind the Traefik ForwardAuth gateway
+
+Instead of interactive browser logins or hardcoding static basic auth credentials into individual ingress configs, Authentik supports **Service Accounts & API Tokens**.
+
+---
+
+## ⚙️ Setup Instructions
+
+### 1. Create a Service Account in Authentik
+1. Navigate to **Directory > Users** in Authentik (`https://auth.haumdaucher.de`).
+2. Click **Create Service Account**:
+   * **Username**: `ai-assistant` (or `mopbot`, `cli-agent`)
+   * **Create API Token**: Check this box.
+   * **Token Expiration**: Set to never expire (or desired TTL).
+3. Copy the generated **Token Secret** (e.g. `ak_tok_xxxx...`).
+
+### 2. Grant Access Permissions
+1. Go to **Directory > Groups**.
+2. Add the `ai-assistant` service account to the appropriate group (e.g. `infra-admins` or `zigbee-readers`).
+3. Ensure the Application/Proxy Provider policy allows this group.
+
+---
+
+## 💻 Making Authenticated API Calls
+
+### HTTP Bearer Token (Recommended)
+Pass the API token in the `Authorization` header:
+
+```bash
+curl -H "Authorization: Bearer <your-authentik-token>" \
+     https://zigbee2mqtt.moritzgraf.de/api/bridge/info
+```
+
+When Traefik ForwardAuth queries Authentik with this header, Authentik detects the Bearer token, authenticates the `ai-assistant` service account, checks permissions, and responds with HTTP `200 OK`. Traefik then proxies the request directly to the backend.
+
+### Python / Scripting Example
+
+```python
+import requests
+
+AUTH_TOKEN = "your-authentik-token"
+headers = {
+    "Authorization": f"Bearer {AUTH_TOKEN}",
+    "Accept": "application/json"
+}
+
+response = requests.get("https://zigbee2mqtt.moritzgraf.de/api/bridge/devices", headers=headers)
+print(response.json())
+```
+
+---
+
+## 🔄 Rotating & Revoking Tokens
+* To rotate a token, navigate to **Directory > Tokens and App passwords**.
+* You can view active tokens, inspect last-used timestamps, and revoke compromised tokens instantly with one click without restarting any Kubernetes pods or modifying ingress secrets.

k8s/auth/README.md

@@ -0,0 +1,188 @@
+# Authentik Identity & Access Management (SSO)
+
+Centralized authentication and identity provider for the `haumdaucher` Kubernetes cluster.
+
+* **Domain**: `https://auth.haumdaucher.de`
+* **Helm Chart**: `authentik/authentik` (Version 2026.8.0)
+* **Database**: CloudNative-PG (`cnpg`) PostgreSQL cluster (`authentik-db`)
+* **Ingress**: Traefik with Let's Encrypt TLS certificate (`auth-haumdaucher-de`)
+* **Capabilities**: OIDC/OAuth2 Provider, Traefik ForwardAuth, 2FA (WebAuthn/Passkey, TOTP), User Invitations, Machine/AI Service Accounts.
+
+---
+
+## 🏗️ Architecture Layout
+
+```mermaid
+graph TD
+    Client[Browser / User] -->|HTTPS| Traefik[Traefik Ingress]
+    AI[AI Assistant / CLI Script] -->|HTTPS + Bearer Token| Traefik
+    
+    subgraph "ForwardAuth Ingresses"
+        Traefik -->|ForwardAuth Check| ForwardAuthMW[Traefik ForwardAuth Middleware]
+        ForwardAuthMW -->|Verify Session or Token| AuthServer[Authentik Server]
+        Traefik -->|Allow| Z2M[Zigbee2MQTT]
+        Traefik -->|Allow| Prom[Prometheus & Alertmanager]
+        Traefik -->|Allow| PMA[phpMyAdmin]
+        Traefik -->|Allow| OC[OpenClaw]
+        Traefik -->|Allow| N8N[n8n Web UI]
+    end
+
+    subgraph "Native OIDC Workloads"
+        Grafana[Grafana] -->|OIDC Auth Code Flow| AuthServer
+        Coder[Coder v2] -->|OIDC Auth| AuthServer
+        Gitea[Gitea] -->|OIDC Auth| AuthServer
+        Nextcloud[Nextcloud user_oidc] -->|OIDC Auth| AuthServer
+        MinIO[MinIO Console] -->|OIDC Auth| AuthServer
+    end
+
+    subgraph "auth Namespace"
+        AuthServer --> DB[(CNPG Postgres: authentik-db)]
+        AuthWorker[Authentik Worker] --> DB
+    end
+```
+
+---
+
+## 🚀 Deployment Instructions
+
+### 1. Create Namespace & Provision CNPG Database
+
+```bash
+# Create auth namespace
+kubectl create namespace auth
+
+# Apply CNPG PostgreSQL cluster
+kubectl apply -f k8s/auth/authentik-db.yaml
+
+# Verify PostgreSQL cluster is ready (Status: Cluster in healthy state)
+kubectl get cluster -n auth authentik-db
+```
+
+### 2. Deploy Authentik via Helm
+
+```bash
+# Add Authentik repository
+helm repo add authentik https://charts.goauthentik.io
+helm repo update
+
+# Install or upgrade Authentik release
+helm upgrade --install authentik authentik/authentik \
+  --namespace auth \
+  -f k8s/auth/authentik.secret.values.yaml
+```
+
+### 3. Deploy Traefik ForwardAuth Middleware
+
+```bash
+kubectl apply -f k8s/auth/traefik-forwardauth-middleware.yaml
+```
+
+---
+
+## 🔑 Initial Setup & Admin Onboarding
+
+1. Open your browser and navigate to:
+   ```
+   https://auth.haumdaucher.de/if/flow/initial-setup/
+   ```
+2. Set a secure password for the default `akadmin` account.
+3. Configure your admin user profile and optionally enroll a WebAuthn / Passkey / TOTP 2FA device.
+
+---
+
+## 👥 User Management & Invitations
+
+### Creating Users & Sending Invites
+1. Navigate to **Directory > Users** in the Authentik Admin interface.
+2. You can create users manually or upload/provision users in batches.
+3. To send enrollment links, go to **Directory > Invitations**:
+   * Click **Create**.
+   * Set the target email and assign user groups (e.g. `family`, `friends`, `admins`).
+   * Copy the generated invitation URL or send it to the user.
+
+### Group Permissions
+* `authentik Admins`: Full administrative control over Authentik.
+* `infra-admins`: Access to infrastructure tools (Prometheus, Alertmanager, phpMyAdmin, Zigbee2MQTT).
+* `users`: Standard user group with access to shared web services (Grafana dashboards, Coder, Nextcloud, Gitea).
+
+👉 **For the complete guide on Terraform user management, bulk Excel/CSV imports, custom email templates, and RBAC/PBAC access limiting, see [USER_MANAGEMENT.md](./USER_MANAGEMENT.md).**
+
+---
+
+## 🤖 Machine & AI Agent Access (Programmatic Authentication)
+
+For automated tools, scripts, or AI assistants that need to inspect protected applications (e.g. querying Zigbee2MQTT API/logs or Prometheus metrics):
+
+1. **Create a Service Account**:
+   * In Authentik Admin, navigate to **Directory > Users**.
+   * Click **Create Service Account** (e.g. `ai-agent` or `automation-bot`).
+   * Create a permanent API token for the service account.
+2. **Accessing Protected Endpoints**:
+   * Pass the token in the HTTP `Authorization` header:
+     ```bash
+     curl -H "Authorization: Bearer <your-authentik-token>" https://zigbee2mqtt.moritzgraf.de/api/...
+     ```
+   * Authentik's ForwardAuth middleware validates the Bearer token and allows the request without redirecting to a web login page.
+
+See [MACHINE_AUTH.md](./MACHINE_AUTH.md) for full details.
+
+---
+
+## 🔒 Protecting Applications (Rollout Guide)
+
+### Pattern 1: Traefik ForwardAuth (Proxy Protection)
+Use this for applications without native OIDC (e.g. Zigbee2MQTT, Prometheus, phpMyAdmin, OpenClaw).
+
+1. In Authentik Admin, go to **Applications > Providers**:
+   * Create a **Proxy Provider**.
+   * Name: `<App Name> Proxy` (e.g. `Zigbee2MQTT Proxy`).
+   * Mode: `Forward auth (single application)`.
+   * External Host: `https://<app-domain>` (e.g. `https://zigbee2mqtt.moritzgraf.de`).
+2. Go to **Applications > Applications**:
+   * Create Application, link the Provider, and set access policies/groups.
+3. Add the middleware annotation to the application's Kubernetes Ingress:
+   ```yaml
+   annotations:
+     traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+   ```
+
+### Pattern 2: Native OIDC SSO
+Use this for applications supporting OpenID Connect (Grafana, Coder, Gitea, Nextcloud).
+
+1. In Authentik Admin, create an **OAuth2/OpenID Provider**:
+   * Name: `<App Name> OIDC`
+   * Client Type: `Confidential`
+   * Redirect URI: e.g. `https://grafana.haumdaucher.de/login/generic_oauth`
+2. Create an **Application** in Authentik and assign it to the Provider.
+3. Configure the client credentials (`client_id` and `client_secret`) in the application's Helm values / ConfigMap.
+
+---
+
+## 🎨 Branding & Haumdaucher UI Theme
+
+The Authentik login screen is customized with a **Haumdaucher** glassmorphism theme matching `haumdaucher.de`.
+
+* **Styling Highlights**:
+  * **Glassmorphic Card**: Frosted glass panel (`backdrop-filter: blur(20px)`), smooth rounded corners (`24px`), and elegant ambient background aura.
+  * **Typography**: Clean, modern typography using Google Font **Outfit**.
+  * **Logo & Entrance Animation**: Dynamic 3D scale-and-fade entry animation for the Haumdaucher crest logo (`https://haumdaucher.de/images/logo_classic.png`).
+  * **Dual Light & Dark Theme**: Automatic adaptation with dark mode styling featuring obsidian surfaces, gold accents (`#d4af37`), and glowing tactile buttons.
+  * **Branded Footer**: "HAUMDAUCHER Wurst und Spezialitäten n.e.V." subtle watermark.
+
+### Managing Theme via Terraform
+Branding is provisioned via the `authentik_brand` resource in `terraform/authentik.tf` using the stylesheet `terraform/haumdaucher-theme.css`:
+
+```bash
+cd terraform
+terraform apply -target=authentik_brand.haumdaucher
+```
+
+---
+
+## 💾 Backup & Disaster Recovery
+
+* **Database**: Managed by CloudNative-PG on `openebs-hostpath` persistent storage.
+* **Velero Integration**: The `auth` namespace is backed up during daily/weekly Velero schedules:
+  ```bash
+  velero backup create auth-backup-$(date +%Y%m%d) --include-namespaces auth --wait
+  ```

k8s/auth/USER_MANAGEMENT.md

@@ -0,0 +1,629 @@
+# Authentik User Management: Complete Roundup & Operations Guide
+
+This guide covers everything you need to know about managing users in Authentik for the **Haumdaucher** infrastructure (`https://auth.haumdaucher.de`): from manual creation and invitation workflows to Terraform IaC, custom email branding, fine-grained access control, handling custom attributes (e.g., Vereins-Mitgliedsnummer, second email), and bulk-importing users from an Excel / CSV spreadsheet.
+
+---
+
+## 📑 Table of Contents
+
+1. [Executive Summary: Quick Answers](#-executive-summary-quick-answers)
+2. [Method 1: Manual Web UI & Invitation Links](#-method-1-manual-web-ui--invitation-links)
+3. [Method 2: User Provisioning via Terraform](#-method-2-user-provisioning-via-terraform)
+4. [Method 3: Bulk Import from Excel / CSV (Python Script)](#-method-3-bulk-import-from-excel--csv-python-script)
+5. [Customizing Registration & Onboarding Emails](#-customizing-registration--onboarding-emails)
+6. [Access Control & Authorization Policies (Limiting App Access)](#-access-control--authorization-policies-limiting-app-access)
+7. [Custom Attributes & OIDC Claim Mappings](#-custom-attributes--oidc-claim-mappings)
+8. [Production Checklist & Best Practices](#-production-checklist--best-practices)
+
+---
+
+## ⚡ Executive Summary: Quick Answers
+
+| Question | Answer & Key Mechanism |
+| :--- | :--- |
+| **How to add new users?** | **4 Ways**: (1) Admin Web UI (`Directory > Users`), (2) **Invitations flow** (`Directory > Invitations` - recommended for self-service), (3) **Terraform** (`authentik_user` resource), or (4) **REST API / Python script** for bulk imports. |
+| **Can this be done in Terraform?** | **Yes.** Use `resource "authentik_user"` in `terraform/authentik.tf`. Supports custom attributes via `jsonencode()`, group assignments, and active status. |
+| **Can I change the email sent to the user?** | **Yes.** Configured in **Flows & Stages > Stages > Email Stage**. You can customize the subject line, token expiration, and HTML template (written in Django template syntax with access to user fields and custom attributes). |
+| **Can I limit what a user has access to?** | **Yes.** Through **Policy Bindings** attached to Applications: Group Membership Policies (e.g., `infra-admins` vs `users`), IP/Geo Policies, or Python Expression Policies checking custom attributes (e.g., only active club members). |
+| **How to handle Excel tables with custom attributes?** | Authentik stores arbitrary key-value pairs in `user.attributes` (JSON). You can import an Excel/CSV file with a Python script or Terraform `csvdecode()`, map fields like `haumdaucher_mitgliedsnummer` or `second_email`, and expose them to downstream apps (Nextcloud, Grafana, Gitea) via OIDC Scope Mappings. |
+
+---
+
+## 👤 Method 1: Manual Web UI & Invitation Links
+
+Authentik offers two UI methods to onboard users:
+
+### Option A: Direct User Creation (Admin sets password or triggers reset)
+1. Go to `https://auth.haumdaucher.de` and log in with your admin account.
+2. Navigate to **Directory > Users** and click **Create**.
+3. Fill in:
+   * **Username**: `max.mustermann`
+   * **Name**: `Max Mustermann`
+   * **Email**: `max@example.com`
+   * **Type**: `Internal`
+   * **Groups**: Select groups (e.g., `users`, `verein-mitglieder`).
+   * **Attributes (YAML/JSON)**:
+     ```json
+     {
+       "second_email": "max.privat@gmail.com",
+       "haumdaucher_mitgliedsnummer": "HM-2024-042",
+       "phone": "+49 170 1234567"
+     }
+     ```
+4. Click **Create**.
+5. To set credentials: Click on the user -> **Actions** -> **Set Password** (or **Explicitly trigger password reset flow** so the user receives a link via email).
+
+---
+
+---
+
+## ⚠️ Troubleshooting: "Flow does not apply to current user"
+
+If you encounter this screen when opening an invite link:
+
+> **"Flow does not apply to current user."**
+
+### Why this happens:
+1. **Wrong Flow Selected (`default-source-enrollment`)**:
+   In your URL (`/if/flow/default-source-enrollment/?itoken=...`), the flow used was `default-source-enrollment`. That flow is strictly designed for external identity providers (OAuth/SAML like Google/GitHub) and binds a policy that checks for an active external source in the context. When accessed directly with an `?itoken=...`, Authentik evaluates this policy, fails, and displays this error.
+2. **Missing Invitation Stage**:
+   Standard enrollment flows reject direct token access unless an **`Invitation Stage`** is bound to the beginning of the flow to validate `?itoken=...`.
+3. **Active Admin Session**:
+   If an administrator is currently logged in to the browser and clicks the enrollment link, flows with an "unauthenticated-only" policy will reject the session. Always test invitation URLs in an **Incognito / Private window**.
+
+---
+
+### Solution: The Haumdaucher Custom Invitation Flow (`haumdaucher-invitation-flow`)
+
+We provision a dedicated, streamlined flow in `terraform/authentik.tf`:
+
+```mermaid
+graph LR
+    Link[Invite Link with ?itoken=...] --> Stage1[1. Invitation Stage<br/>Validates Token]
+    Stage1 --> Stage2[2. Prompt Stage<br/>User inputs Username, Name, Password]
+    Stage2 --> Stage3[3. User Write Stage<br/>Writes to DB & assigns Groups]
+    Stage3 --> Stage4[4. User Login Stage<br/>Logs user into session]
+    Stage4 --> Stage5[5. Redirect Stage<br/>Redirects to haumdaucher.de]
+```
+
+#### Flow Behavior (Tailored to your setup):
+* **User Inputs**: Prompts the user to choose their preferred **Username**, **Full Name**, and **Password** (with confirmation).
+* **Email & Custom Attributes**: Pre-filled from the admin invitation (e.g. `haumdaucher_mitgliedsnummer`, `second_email`).
+* **MFA / 2FA**: Optional. The user registers immediately with their password and can enroll Passkeys/TOTP anytime in their profile settings.
+* **Automatic Redirect**: Upon completing registration, the user is automatically logged in and redirected to **`https://haumdaucher.de`**.
+
+---
+
+### How to Create an Invitation using the Custom Flow
+
+1. Go to `https://auth.haumdaucher.de` > **Directory > Invitations**.
+2. Click **Create** (or **New Invitation**).
+3. Set the fields:
+   * **Name**: `Invite - Max Mustermann`
+   * **Expires**: Set expiration (e.g., 7 days).
+   * **Flow**: Select **`Haumdaucher Invitation Flow`** (`haumdaucher-invitation-flow`).
+   * **Single use**: Checked (`true`).
+   * **Fixed Data / Attributes (JSON)**:
+     ```json
+     {
+       "email": "max@example.com",
+       "haumdaucher_mitgliedsnummer": "HM-2024-042",
+       "second_email": "max.privat@gmail.com"
+     }
+     ```
+4. Click **Create**.
+5. Copy the generated URL:
+   ```
+   https://auth.haumdaucher.de/if/flow/haumdaucher-invitation-flow/?itoken=<generated-token>
+   ```
+6. Send the link to the user. When they open it, they will see the branded Haumdaucher registration form asking for their Username, Name, and Password!
+
+---
+
+## 🏗️ Method 2: User Provisioning via Terraform
+
+You can manage users, groups, and permissions directly in `terraform/authentik.tf`.
+
+### 1. Single User Definition
+```hcl
+# Group for club members
+resource "authentik_group" "verein_mitglieder" {
+  name         = "verein-mitglieder"
+  is_superuser = false
+}
+
+# User with custom attributes
+resource "authentik_user" "max_mustermann" {
+  username   = "max.mustermann"
+  name       = "Max Mustermann"
+  email      = "max@example.com"
+  type       = "internal"
+  is_active  = true
+  groups     = [
+    authentik_group.users.id,
+    authentik_group.verein_mitglieder.id
+  ]
+  attributes = jsonencode({
+    "second_email"                 = "max.privat@gmail.com",
+    "haumdaucher_mitgliedsnummer" = "HM-2024-042",
+    "eintrittsdatum"               = "2024-01-15",
+    "status"                       = "aktiv"
+  })
+}
+```
+
+### 2. Batch Provisioning from CSV in Terraform
+If you maintain users in a CSV file (`terraform/users.csv`):
+
+```csv
+username,name,email,second_email,mitgliedsnummer,is_admin
+max.mustermann,Max Mustermann,max@example.com,max.privat@gmail.com,HM-2024-042,false
+anna.schmidt,Anna Schmidt,anna@example.com,anna.work@company.de,HM-2024-043,true
+```
+
+In `terraform/authentik.tf`:
+```hcl
+locals {
+  csv_users = csvdecode(file("${path.module}/users.csv"))
+}
+
+resource "authentik_user" "csv_members" {
+  for_each = { for u in local.csv_users : u.username => u }
+
+  username   = each.value.username
+  name       = each.value.name
+  email      = each.value.email
+  type       = "internal"
+  is_active  = true
+  groups = compact([
+    authentik_group.users.id,
+    authentik_group.verein_mitglieder.id,
+    each.value.is_admin == "true" ? authentik_group.infra_admins.id : null
+  ])
+  attributes = jsonencode({
+    "second_email"                 = each.value.second_email,
+    "haumdaucher_mitgliedsnummer" = each.value.mitgliedsnummer
+  })
+}
+```
+
+### ⚖️ Trade-offs: When to use Terraform for Users
+* **Pros**: Version-controlled in Git, declarative, reproducible, ideal for admins and service accounts (`ai_assistant`).
+* **Cons / Considerations**: Terraform cannot trigger interactive password onboarding emails out of the box; passwords must be set via recovery flow or first login reset.
+
+---
+
+## 📊 Method 3: Bulk Import from Excel / CSV (Python Script)
+
+If you have an Excel file (`mitglieder.xlsx`) or CSV, using Authentik's REST API is the cleanest approach because it can **create users, attach custom attributes, and automatically email them an invitation link to set their own password**.
+
+### Ready-to-Run Python Import Script
+Save this script (e.g. `k8s/auth/scripts/import_users_excel.py`):
+
+```python
+#!/usr/bin/env python3
+"""
+Bulk User Importer for Authentik (auth.haumdaucher.de)
+Reads an Excel (.xlsx) or CSV file and creates users with custom attributes and invitation links.
+"""
+
+import sys
+import os
+import requests
+import pandas as pd
+
+# Configuration
+AUTHENTIK_URL = os.getenv("AUTHENTIK_URL", "https://auth.haumdaucher.de")
+AUTHENTIK_TOKEN = os.getenv("AUTHENTIK_TOKEN")  # API Token (e.g. ai_assistant_token)
+
+if not AUTHENTIK_TOKEN:
+    print("❌ Error: AUTHENTIK_TOKEN environment variable is required.")
+    sys.exit(1)
+
+HEADERS = {
+    "Authorization": f"Bearer {AUTHENTIK_TOKEN}",
+    "Content-Type": "application/json",
+    "Accept": "application/json"
+}
+
+def get_or_create_group(group_name: str) -> str:
+    """Fetch group ID or create if it doesn't exist."""
+    res = requests.get(f"{AUTHENTIK_URL}/api/v3/core/groups/?name={group_name}", headers=HEADERS)
+    res.raise_for_status()
+    results = res.json().get("results", [])
+    if results:
+        return results[0]["pk"]
+    
+    # Create group
+    create_res = requests.post(f"{AUTHENTIK_URL}/api/v3/core/groups/", headers=HEADERS, json={"name": group_name})
+    create_res.raise_for_status()
+    return create_res.json()["pk"]
+
+def import_users_from_file(file_path: str, send_invitation: bool = True):
+    # Read Excel or CSV
+    if file_path.endswith((".xlsx", ".xls")):
+        df = pd.read_excel(file_path)
+    else:
+        df = pd.read_csv(file_path)
+
+    # Standardize column names (lowercase & stripped)
+    df.columns = [c.strip().lower() for c in df.columns]
+
+    print(f"📋 Found {len(df)} rows to process.")
+    
+    users_group_pk = get_or_create_group("users")
+    verein_group_pk = get_or_create_group("verein-mitglieder")
+
+    for idx, row in df.iterrows():
+        name = str(row.get("name", "")).strip()
+        email = str(row.get("email", "")).strip()
+        username = str(row.get("username", "")).strip() or email.split("@")[0].lower()
+        second_email = str(row.get("second_email", row.get("zweite_email", ""))).strip()
+        mitgliedsnummer = str(row.get("mitgliedsnummer", row.get("verein_mitgliedsnummer", ""))).strip()
+
+        if not email or email == "nan":
+            print(f"⚠️ Skipping row {idx+1}: missing email")
+            continue
+
+        # Prepare user payload with custom attributes
+        user_payload = {
+            "username": username,
+            "name": name if name != "nan" else username,
+            "email": email,
+            "type": "internal",
+            "is_active": True,
+            "groups": [users_group_pk, verein_group_pk],
+            "attributes": {
+                "second_email": second_email if second_email != "nan" else "",
+                "haumdaucher_mitgliedsnummer": mitgliedsnummer if mitgliedsnummer != "nan" else "",
+                "source": "excel_import"
+            }
+        }
+
+        # 1. Create or Update User in Authentik
+        res = requests.get(f"{AUTHENTIK_URL}/api/v3/core/users/?username={username}", headers=HEADERS)
+        res.raise_for_status()
+        existing = res.json().get("results", [])
+
+        if existing:
+            user_pk = existing[0]["pk"]
+            patch_res = requests.patch(f"{AUTHENTIK_URL}/api/v3/core/users/{user_pk}/", headers=HEADERS, json=user_payload)
+            patch_res.raise_for_status()
+            print(f"🔄 Updated existing user: {username} ({email})")
+        else:
+            create_res = requests.post(f"{AUTHENTIK_URL}/api/v3/core/users/", headers=HEADERS, json=user_payload)
+            if not create_res.ok:
+                print(f"❌ Failed to create {username}: {create_res.text}")
+                continue
+            user_pk = create_res.json()["pk"]
+            print(f"✅ Created user: {username} ({email}) [ID: {user_pk}]")
+
+        # 2. Generate Invitation / Password Reset Link
+        if send_invitation:
+            invitation_payload = {
+                "name": f"Onboarding {username}",
+                "single_use": True,
+                "fixed_data": {
+                    "email": email,
+                    "username": username
+                }
+            }
+            inv_res = requests.post(f"{AUTHENTIK_URL}/api/v3/stages/invitation/invitations/", headers=HEADERS, json=invitation_payload)
+            if inv_res.ok:
+                inv_key = inv_res.json()["pk"]
+                link = f"{AUTHENTIK_URL}/if/flow/default-enrollment-flow/?itoken={inv_key}"
+                print(f"   ✉️ Invitation Link: {link}")
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        print("Usage: python3 import_users_excel.py <path-to-excel-or-csv>")
+        sys.exit(1)
+    import_users_from_file(sys.argv[1])
+```
+
+---
+
+## ✉️ Customizing Registration & Onboarding Emails
+
+When an admin creates a user or triggers an enrollment/password recovery, Authentik sends transactional emails. You can customize their sender address, subject line, and full HTML appearance.
+
+### 1. SMTP & Sender Settings
+Authentik sends emails using the SMTP configuration defined in `k8s/auth/authentik.secret.values.yaml` or global environment variables:
+* `AUTHENTIK_EMAIL__HOST`: `smtp.gmail.com` (or Google Workspace / OVH relay)
+* `AUTHENTIK_EMAIL__PORT`: `587`
+* `AUTHENTIK_EMAIL__USE_TLS`: `true`
+* `AUTHENTIK_EMAIL__USERNAME`: `service@haumdaucher.de`
+* `AUTHENTIK_EMAIL__PASSWORD`: `<app-password>`
+* `AUTHENTIK_EMAIL__FROM`: `HAUMDAUCHER Identity <service@haumdaucher.de>`
+
+---
+
+### 2. Customizing Email Stages in Authentik
+Authentik controls outgoing emails via **Email Stages**:
+1. Open Admin UI: **Flows & Stages > Stages**.
+2. Locate the relevant email stage:
+   * `default-password-reset-email` (for password recovery & manual user activation)
+   * `default-enrollment-email` / `invitation-email` (for new user invitations)
+   * `default-user-settings-password-reset-email`
+3. Click **Edit** on the stage:
+   * **Subject**: `Willkommen bei Haumdaucher e.V. – Aktiviere deinen Zugang`
+   * **Token Expiry**: `Minutes=1440` (24 hours) or `Minutes=10080` (7 days)
+   * **Template**: Select standard template or your custom template path.
+
+---
+
+### 3. Customizing the HTML Email Template (Django Syntax)
+Authentik email templates use Django template syntax. Available variables in email templates include:
+* `{{ user.name }}` / `{{ user.username }}` / `{{ user.email }}`
+* `{{ user.attributes.haumdaucher_mitgliedsnummer }}`
+* `{{ url }}` (The activation / password reset link)
+* `{{ token_expires }}` (Expiry timestamp)
+* `{{ brand.branding_title }}` (`HAUMDAUCHER`)
+
+#### Example Custom Haumdaucher Welcome Template
+```html
+<!DOCTYPE html>
+<html>
+<head>
+  <style>
+    body { font-family: 'Outfit', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background-color: #0d0d12; color: #f0f0f5; margin: 0; padding: 40px; }
+    .card { background: #181824; border: 1px solid rgba(212, 175, 55, 0.3); border-radius: 16px; max-width: 540px; margin: 0 auto; padding: 32px; box-shadow: 0 12px 30px rgba(0,0,0,0.5); }
+    .header { text-align: center; margin-bottom: 24px; }
+    .logo { height: 60px; }
+    h1 { color: #d4af37; font-size: 22px; margin-top: 12px; }
+    p { color: #c0c0d0; line-height: 1.6; font-size: 15px; }
+    .btn { display: block; text-align: center; background: linear-gradient(135deg, #d4af37, #aa8520); color: #000; font-weight: bold; text-decoration: none; padding: 14px 24px; border-radius: 10px; margin: 28px 0; }
+    .meta { background: rgba(255,255,255,0.05); border-radius: 8px; padding: 12px 16px; font-size: 13px; color: #a0a0b0; }
+    .footer { text-align: center; font-size: 12px; color: #606070; margin-top: 30px; }
+  </style>
+</head>
+<body>
+  <div class="card">
+    <div class="header">
+      <img class="logo" src="https://haumdaucher.de/images/logo_classic.png" alt="Haumdaucher Logo" />
+      <h1>Willkommen beim Haumdaucher e.V.</h1>
+    </div>
+    
+    <p>Hallo <strong>{{ user.name }}</strong>,</p>
+    <p>dein zentraler Zugang für die Haumdaucher Cloud & Services wurde eingerichtet.</p>
+    
+    {% if user.attributes.haumdaucher_mitgliedsnummer %}
+    <div class="meta">
+      <strong>Mitgliedsnummer:</strong> {{ user.attributes.haumdaucher_mitgliedsnummer }}<br/>
+      <strong>Benutzername:</strong> {{ user.username }}
+    </div>
+    {% endif %}
+
+    <p>Klicke auf den Button unten, um dein Passwort festzulegen und deinen Zugang zu aktivieren:</p>
+    
+    <a class="btn" href="{{ url }}">Zugang aktivieren & Passwort setzen</a>
+    
+    <p style="font-size: 12px; color: #888;">Dieser Link ist 7 Tage lang gültig. Falls du die Registrierung nicht angefordert hast, kannst du diese E-Mail ignorieren.</p>
+    
+    <div class="footer">
+      HAUMDAUCHER e.V. &bull; <a href="https://haumdaucher.de" style="color: #d4af37;">haumdaucher.de</a>
+    </div>
+  </div>
+</body>
+</html>
+```
+
+---
+
+## 🔒 Access Control & Authorization Policies (Limiting App Access)
+
+By default, an application without an attached policy allows **any authenticated user** to access it. Authentik provides fine-grained Access Control (RBAC & PBAC) to restrict access per application.
+
+```mermaid
+graph TD
+    subgraph "Authentik Directory Groups"
+        Admin["👑 <b>infra-admins</b><br/>(Moritz & Wife / Full Admin)"]
+        Verein["🍺 <b>verein-mitglieder</b><br/>(Club Members)"]
+        Normal["👤 <b>users</b><br/>(Normal Authenticated Users)"]
+    end
+
+    subgraph "Tier 1: Admin & Ops Tools (ForwardAuth / OIDC)"
+        Z2M["Zigbee2MQTT"]
+        Prom["Prometheus & Alertmanager"]
+        PMA["phpMyAdmin (Nextcloud & Gitea)"]
+        N8N["n8n Automation"]
+        MinIO["MinIO Console"]
+        Coder["Coder v2"]
+        OpenClaw["OpenClaw Engine"]
+    end
+
+    subgraph "Tier 2: Daily Shared Services (OIDC)"
+        HASS["Home Assistant"]
+        Grafana["Grafana Dashboards"]
+        Nextcloud["Nextcloud Cloud"]
+        Gitea["Gitea Git Platform"]
+        ClubWeb["haumdaucher.de"]
+    end
+
+    %% Admin Links (Full Superadmin access everywhere)
+    Admin ==>|Full Access| Z2M
+    Admin ==>|Full Access| OpenClaw
+    Admin ==>|Admin| HASS
+    Admin ==>|Admin| Grafana
+    Admin ==>|Admin| Nextcloud
+    Admin ==>|Admin| Gitea
+    Admin ==>|Admin| ClubWeb
+
+    %% Normal User Links (Restricted)
+    Normal -->|Viewer (Read-Only)| Grafana
+    Normal -->|Dashboard View| HASS
+    Normal -->|Standard User| Nextcloud
+    Normal -->|Developer| Gitea
+
+    %% Club Members Links
+    Verein -->|Member Portal| ClubWeb
+    Verein -->|Club Shared Folder| Nextcloud
+    Verein -->|Viewer (Read-Only)| Grafana
+```
+
+### Master Permission Matrix
+
+| Application | Domain | Auth Type | `infra-admins` (Moritz & Wife) | `verein-mitglieder` (Club) | `users` (Normal Users) |
+| :--- | :--- | :--- | :---: | :---: | :---: |
+| **OpenClaw** | `openclaw.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Zigbee2MQTT** | `zigbee2mqtt.moritzgraf.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Prometheus / Alertmanager** | `prometheus.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **phpMyAdmin (All DBs)** | `*.phpmyadmin.haumdaucher.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **n8n** | `n8n.moritzgraf.de` | ForwardAuth | **Full Access** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **MinIO Console** | `minio-console.haumdaucher.de` | Native OIDC | **`consoleAdmin`** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Coder v2** | `coder.moritzgraf.de` | Native OIDC | **`Owner / Admin`** ✅ | ❌ *Denied* | ❌ *Denied* |
+| **Home Assistant** | `hass.moritzgraf.de` | Native OIDC | **`Administrator`** ✅ | ❌ *Denied* | **`Standard User`** |
+| **Grafana** | `grafana.haumdaucher.de` | Native OIDC | **`Admin`** ✅ | **`Viewer` (Read-Only)** | **`Viewer` (Read-Only)** |
+| **Nextcloud** | `cloud.haumdaucher.de` | Native OIDC | **`Admin`** ✅ | **`User` (Club Folder)** | **`User` (Standard)** |
+| **Gitea** | `git.moritzgraf.de` | Native OIDC | **`Admin`** ✅ | ❌ / Member | **`Developer`** |
+| **haumdaucher.de** | `haumdaucher.de` | ForwardAuth / OIDC | **`Admin`** ✅ | **`Member Access`** ✅ | ❌ *Denied* |
+
+### Policy Bindings in Terraform
+All policy bindings are enforced directly in `terraform/authentik.tf`:
+
+```hcl
+# 1. Admin-Only Tools (Restricted strictly to infra-admins)
+resource "authentik_policy_binding" "openclaw_admin_only" {
+  target = authentik_application.openclaw.uuid
+  group  = authentik_group.infra_admins.id
+  order  = 0
+}
+
+resource "authentik_policy_binding" "zigbee2mqtt_admin_only" {
+  target = authentik_application.zigbee2mqtt.uuid
+  group  = authentik_group.infra_admins.id
+  order  = 0
+}
+
+# 2. Shared & Member Applications (Accessible to users group)
+resource "authentik_policy_binding" "grafana_users" {
+  target = authentik_application.grafana.uuid
+  group  = authentik_group.users.id
+  order  = 0
+}
+
+resource "authentik_policy_binding" "nextcloud_users" {
+  target = authentik_application.nextcloud.uuid
+  group  = authentik_group.users.id
+  order  = 0
+}
+```
+
+---
+
+### 2. Attribute-Based Policy (Python Expression Policy)
+You can write Python expressions that evaluate custom user attributes dynamically:
+
+```hcl
+resource "authentik_policy_expression" "require_active_membership" {
+  name       = "require-active-haumdaucher-membership"
+  expression = <<EOF
+# Must have a valid Mitgliedsnummer and active status
+mitgliedsnummer = request.user.attributes.get('haumdaucher_mitgliedsnummer')
+status = request.user.attributes.get('status', 'aktiv')
+
+if not mitgliedsnummer or not mitgliedsnummer.startswith('HM-'):
+    ak_message("Keine gültige Haumdaucher Mitgliedsnummer hinterlegt.")
+    return False
+
+if status != 'aktiv':
+    ak_message("Deine Mitgliedschaft ist derzeit nicht aktiv.")
+    return False
+
+return True
+EOF
+}
+
+# Attach policy to Nextcloud application
+resource "authentik_policy_binding" "nextcloud_active_members" {
+  target = authentik_application.nextcloud.uuid
+  policy = authentik_policy_expression.require_active_membership.id
+  order  = 1
+}
+```
+
+---
+
+## 🏷️ Custom Attributes & OIDC Claim Mappings
+
+Authentik allows storing any arbitrary JSON object inside `user.attributes`.
+
+### 1. Recommended Attribute Schema for Haumdaucher
+```json
+{
+  "haumdaucher_mitgliedsnummer": "HM-2024-042",
+  "second_email": "max.privat@gmail.com",
+  "eintrittsdatum": "2024-01-15",
+  "verein_rolle": "Vorstand / Kassenwart / Mitglied",
+  "phone": "+49 170 1234567",
+  "telegram_handle": "@max_mustermann"
+}
+```
+
+---
+
+### 2. Exposing Custom Attributes via OIDC Claims (Terraform)
+To pass `haumdaucher_mitgliedsnummer` or `second_email` to downstream applications like **Nextcloud**, **Grafana**, or **Gitea**:
+
+```hcl
+# Define a custom Scope Mapping in Terraform
+resource "authentik_property_mapping_provider_scope" "haumdaucher_profile" {
+  name        = "Haumdaucher Extended Profile"
+  scope_name  = "haumdaucher_profile"
+  description = "Provides Vereins-Mitgliedsnummer and secondary email to OIDC clients"
+  expression  = <<EOF
+return {
+    "haumdaucher_mitgliedsnummer": request.user.attributes.get("haumdaucher_mitgliedsnummer", ""),
+    "second_email": request.user.attributes.get("second_email", ""),
+    "verein_rolle": request.user.attributes.get("verein_rolle", "Mitglied")
+}
+EOF
+}
+
+# Add this mapping to the OAuth2 provider (e.g. Nextcloud or Grafana)
+resource "authentik_provider_oauth2" "nextcloud" {
+  name               = "Nextcloud"
+  client_id          = "nextcloud"
+  client_secret      = random_password.nextcloud_oauth_client_secret.result
+  client_type        = "confidential"
+  authorization_flow = data.authentik_flow.default_authorization_flow.id
+  invalidation_flow  = data.authentik_flow.default_invalidation_flow.id
+  signing_key        = data.authentik_certificate_key_pair.generated.id
+  
+  property_mappings  = [
+    data.authentik_property_mapping_provider_scope.openid.id,
+    data.authentik_property_mapping_provider_scope.email.id,
+    data.authentik_property_mapping_provider_scope.profile.id,
+    authentik_property_mapping_provider_scope.haumdaucher_profile.id, # <-- Custom scope
+  ]
+}
+```
+
+---
+
+## 🛡️ Production Checklist & Best Practices
+
+When managing users in production, keep the following considerations in mind:
+
+### 1. Passwordless & Multi-Factor Authentication (MFA / 2FA)
+* Enforce **WebAuthn / Passkeys (FIDO2)** or **TOTP (Authenticator App)** for users in the `infra-admins` group.
+* In Authentik, this is done by binding an Authenticator Validation Stage to the default authentication flow or creating an Admin Authorization Flow.
+
+### 2. User Self-Service & Attribute Editing
+* Authentik provides a self-service portal at `https://auth.haumdaucher.de/if/user/#/settings`.
+* Users can change their password, register Passkeys, and edit user profile fields.
+* **Security Rule**: Keep administrative attributes (like `haumdaucher_mitgliedsnummer`) read-only by not exposing them to the user edit flow, while allowing users to freely edit personal contact details like `second_email` or `phone`.
+
+### 3. Offboarding & Member Deactivation
+* If a member leaves the club:
+  1. Set `is_active = false` (or remove from `verein-mitglieder` group).
+  2. Authentik immediately terminates active sessions and revokes ForwardAuth access and OIDC refresh tokens.
+  3. No need to manually reset passwords on Nextcloud, Grafana, or Gitea—SSO handles deprovisioning instantly.
+
+### 4. GDPR / DSGVO & Privacy
+* Store only necessary data in user attributes.
+* If a member requests account deletion, deleting the user in Authentik cascades identity termination across all connected services.
+
+### 5. Backups & Audit Logs
+* User database changes are stored in the PostgreSQL cluster (`authentik-db`) and backed up with daily Velero snapshot schedules.
+* Authentik logs all login attempts, failed logins, and administrative modifications under **Events > Logs**.

k8s/auth/authentik-db.yaml

@@ -0,0 +1,24 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: authentik-db
+  namespace: auth
+spec:
+  instances: 1
+  bootstrap:
+    initdb:
+      database: authentik
+      owner: authentik
+  postgresql:
+    parameters:
+      shared_buffers: "128MB"
+      max_connections: "100"
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 2Gi
+  storage:
+    storageClass: openebs-hostpath
+    size: 5Gi

k8s/auth/authentik.secret.values.yaml

@@ -0,0 +1,64 @@
+# --- Authentik Helm Configuration ---
+# Uses CloudNative-PG cluster (authentik-db) for PostgreSQL
+
+global:
+  env:
+    - name: AUTHENTIK_POSTGRESQL__HOST
+      value: "authentik-db-rw.auth.svc.cluster.local"
+    - name: AUTHENTIK_POSTGRESQL__NAME
+      value: "authentik"
+    - name: AUTHENTIK_POSTGRESQL__USER
+      value: "authentik"
+    - name: AUTHENTIK_POSTGRESQL__PASSWORD
+      valueFrom:
+        secretKeyRef:
+          name: authentik-db-app
+          key: password
+    - name: AUTHENTIK_SESSION__DURATION
+      value: "days=30"
+
+authentik:
+  # Secret key for cookie signing and session encryption
+  secret_key: "gU2ohe8eighaiph4quaeCh1ieM3iechoh2ohNg5yahz6eephoh"
+  error_reporting:
+    enabled: false
+  log_level: info
+
+server:
+  replicas: 1
+  resources:
+    requests:
+      cpu: 100m
+      memory: 512Mi
+    limits:
+      memory: 1536Mi
+
+  ingress:
+    enabled: true
+    ingressClassName: "traefik"
+    annotations:
+      kubernetes.io/ingress.class: "traefik"
+      cert-manager.io/cluster-issuer: "letsencrypt-prod"
+      kubernetes.io/tls-acme: "true"
+      nginx.ingress.kubernetes.io/ssl-redirect: "true"
+      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+      nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    hosts:
+      - auth.haumdaucher.de
+    tls:
+      - hosts:
+          - auth.haumdaucher.de
+        secretName: auth-haumdaucher-de
+
+worker:
+  replicas: 1
+  resources:
+    requests:
+      cpu: 100m
+      memory: 384Mi
+    limits:
+      memory: 768Mi
+
+# Disable bundled subchart databases (managed by CNPG)
+postgresql:
+  enabled: false

k8s/auth/phpldapadmin.yaml

@@ -1,25 +0,0 @@
-# # status: removed again
-
-# apiVersion: helm.fluxcd.io/v1
-# kind: HelmRelease
-# metadata:
-#   name: helm-phpldapadmin
-#   namespace: flux
-# spec:
-#   releaseName: phpldapadmin
-#   targetNamespace: auth
-#   chart:
-#     repository: https://cetic.github.io/helm-charts
-#     version: 0.1.3
-#     name: phpldapadmin
-#   values:
-#     service:
-#       type: ClusterIP
-#     ingress:
-#       enabled: true
-#       hosts:
-#         - "ldap.haumdaucher.de"
-#       annotations:
-#         cert-manager.io/cluster-issuer: "letsencrypt-prod"
-#       tls:
-#         - ldap-haumdaucher-de

k8s/auth/traefik-forwardauth-middleware.yaml

@@ -0,0 +1,26 @@
+apiVersion: traefik.io/v1alpha1
+kind: Middleware
+metadata:
+  name: authentik-forwardauth
+  namespace: auth
+spec:
+  forwardAuth:
+    address: http://authentik-server.auth.svc.cluster.local/outpost.goauthentik.io/auth/traefik
+    trustForwardHeader: true
+    authRequestHeaders:
+      - Authorization
+      - Cookie
+      - Accept
+    authResponseHeaders:
+      - X-authentik-username
+      - X-authentik-groups
+      - X-authentik-email
+      - X-authentik-name
+      - X-authentik-uid
+      - X-authentik-jwt
+      - X-authentik-meta-jwks
+      - X-authentik-meta-outpost
+      - X-authentik-meta-provider
+      - X-authentik-meta-app
+      - X-authentik-meta-version
+      - authorization

k8s/backup/job-gdrive-to-gphotos.yaml

@@ -0,0 +1,29 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: gdrive-to-gphotos
+  namespace: backup
+spec:
+  template:
+    spec:
+      containers:
+      - name: sync-agent
+        image: iptizer/swiss
+        volumeMounts:
+        - name: rclone-gdrive-config
+          mountPath: /rclone.conf
+          subPath: "rclone.conf"
+        command:
+        - "/bin/bash"
+        - "-c"
+        args:
+        - |
+          echo "Starting copy operation from Google Drive to Google Photos..."
+          cd /home/devbox && devbox run rclone --config /rclone.conf -v --stats 10s --stats-one-line copy --drive-root-folder-id="1f43AtWvfb7qd-eGr9nw-8vHraA5k9mVl" gdrive: "gphotos:album/All pictures of the last years"
+      volumes:
+      - name: rclone-gdrive-config
+        configMap:
+          name: rclone-gdrive-config
+      restartPolicy: Never
+  activeDeadlineSeconds: 86400
+  backoffLimit: 3

k8s/backup/verify_migration.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+"""
+Google Drive to Google Photos Migration Verification Tool
+
+This script analyzes the migration status of pictures from Google Drive to Google Photos.
+It compares Google Drive folder contents with Pod logs from Kubernetes (and/or Google Photos API)
+to determine exact file counts, total sizes, and completion status per subfolder.
+"""
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+from collections import defaultdict
+
+DEFAULT_SECRET_FILE = os.path.join(os.path.dirname(__file__), "rclone.secret.yml")
+DEFAULT_DRIVE_ROOT_ID = "1f43AtWvfb7qd-eGr9nw-8vHraA5k9mVl"
+DEFAULT_PHOTOS_ALBUM = "All pictures of the last years"
+
+def extract_rclone_conf(secret_file_path):
+    """Extract rclone.conf from Kubernetes ConfigMap secret YAML file."""
+    if not os.path.exists(secret_file_path):
+        print(f"Error: Secret file not found at {secret_file_path}")
+        sys.exit(1)
+    
+    with open(secret_file_path, "r", encoding="utf-8") as f:
+        content = f.read()
+    
+    match = re.search(r"rclone\.conf:\s*\|\n(.*)", content, re.DOTALL)
+    if not match:
+        print("Error: Could not find rclone.conf data in secret file.")
+        sys.exit(1)
+    
+    conf_data = match.group(1)
+    # Remove leading 4-space indentation if present
+    cleaned_lines = []
+    for line in conf_data.split("\n"):
+        if line.startswith("    "):
+            cleaned_lines.append(line[4:])
+        else:
+            cleaned_lines.append(line)
+    
+    tmp_conf = tempfile.NamedTemporaryFile(mode="w", suffix=".conf", delete=False)
+    tmp_conf.write("\n".join(cleaned_lines))
+    tmp_conf.close()
+    return tmp_conf.name
+
+def get_k8s_job_logs(namespace="backup", job_name="gdrive-to-gphotos"):
+    """Fetch logs from all pods associated with the Kubernetes job."""
+    print("Fetching execution logs from Kubernetes pods...")
+    try:
+        res = subprocess.run(
+            ["kubectl", "get", "pods", "-n", namespace, "-l", f"job-name={job_name}", "-o", "jsonpath={.items[*].metadata.name}"],
+            capture_output=True, text=True, check=True
+        )
+        pod_names = res.stdout.strip().split()
+        if not pod_names:
+            print(f"Warning: No pods found for job '{job_name}' in namespace '{namespace}'.")
+            return ""
+        
+        all_logs = []
+        for pod in pod_names:
+            print(f"  Retrieving logs for pod: {pod}")
+            log_res = subprocess.run(
+                ["kubectl", "logs", "-n", namespace, pod],
+                capture_output=True, text=True
+            )
+            all_logs.append(log_res.stdout)
+        return "\n".join(all_logs)
+    except Exception as e:
+        print(f"Warning: Could not fetch k8s logs directly ({e}).")
+        return ""
+
+def parse_copied_files_from_logs(log_text):
+    """Parse log text for confirmed uploaded files per subfolder."""
+    copied_files = defaultdict(set)
+    for line in log_text.splitlines():
+        if "Copied (new)" in line or "Copied (replaced)" in line:
+            m = re.search(r"INFO\s+:\s+(.*?)/(.*?):\s+Copied", line)
+            if m:
+                folder, filename = m.group(1), m.group(2)
+                copied_files[folder].add(filename)
+    return copied_files
+
+def get_gdrive_subfolders(config_path, root_id):
+    """List subdirectories in the source Google Drive folder."""
+    print("Querying subfolders from Google Drive...")
+    res = subprocess.run(
+        ["rclone", "--config", config_path, "lsd", f"--drive-root-folder-id={root_id}", "gdrive:"],
+        capture_output=True, text=True
+    )
+    if res.returncode != 0:
+        print(f"Error listing Google Drive folders: {res.stderr}")
+        return []
+    
+    subfolders = []
+    for line in res.stdout.strip().splitlines():
+        if line:
+            parts = line.split(maxsplit=4)
+            if len(parts) >= 5:
+                subfolders.append(parts[4])
+    return sorted(subfolders)
+
+def get_gdrive_folder_stats(config_path, root_id, subfolder):
+    """Get file count and size for a Google Drive subfolder using rclone size."""
+    res = subprocess.run(
+        ["rclone", "--config", config_path, "size", f"--drive-root-folder-id={root_id}", f"gdrive:{subfolder}"],
+        capture_output=True, text=True
+    )
+    if res.returncode != 0:
+        return {"count": 0, "size_str": "Unknown", "bytes": 0}
+    
+    output = res.stdout.strip()
+    # Handle formats like "Total objects: 2.563k (2563)" or "Total objects: 524"
+    count_match = re.search(r"Total objects:\s*(?:[\d\.\w]+\s*\()?(\d+)\)?", output)
+    size_match = re.search(r"Total size:\s*([\d\.]+\s*[KMGT]?i?B)\s*\((.*?)\s*Bytes\)", output)
+    
+    count = int(count_match.group(1)) if count_match else 0
+    size_str = size_match.group(1) if size_match else "0 B"
+    size_bytes = int(size_match.group(2)) if size_match else 0
+    
+    return {"count": count, "size_str": size_str, "bytes": size_bytes}
+
+def main():
+    parser = argparse.ArgumentParser(description="Verify Google Drive to Google Photos migration status.")
+    parser.add_argument("--secret-file", default=DEFAULT_SECRET_FILE, help="Path to rclone.secret.yml")
+    parser.add_argument("--root-id", default=DEFAULT_DRIVE_ROOT_ID, help="Google Drive root folder ID")
+    parser.add_argument("--local-log", help="Path to local pod log file (optional override)")
+    parser.add_argument("--csv", help="Path to save CSV report output (optional)")
+    args = parser.parse_args()
+
+    print("==========================================================")
+    print(" Google Drive to Google Photos Migration Verification Tool")
+    print("==========================================================")
+    
+    # 1. Extract Config
+    config_path = extract_rclone_conf(args.secret_file)
+    
+    try:
+        # 2. Get Logs
+        if args.local_log and os.path.exists(args.local_log):
+            print(f"Reading logs from local file: {args.local_log}")
+            with open(args.local_log, "r", errors="ignore") as f:
+                log_text = f.read()
+        else:
+            log_text = get_k8s_job_logs()
+        
+        copied_map = parse_copied_files_from_logs(log_text)
+        
+        # 3. Get Google Drive Subfolders
+        subfolders = get_gdrive_subfolders(config_path, args.root_id)
+        if not subfolders:
+            print("No subfolders found or unable to connect to Google Drive.")
+            sys.exit(1)
+        
+        print(f"\nAnalyzing {len(subfolders)} subfolders...\n")
+        
+        results = []
+        total_drive_files = 0
+        total_drive_bytes = 0
+        total_copied_files = 0
+        
+        hdr = f"{'Subfolder Name':<35} | {'GDrive Files':<12} | {'GDrive Size':<12} | {'Uploaded':<10} | {'Status':<15}"
+        print(hdr)
+        print("-" * len(hdr))
+        
+        for folder in subfolders:
+            stats = get_gdrive_folder_stats(config_path, args.root_id, folder)
+            copied_count = len(copied_map.get(folder, set()))
+            
+            drive_count = stats["count"]
+            drive_size = stats["size_str"]
+            total_drive_files += drive_count
+            total_drive_bytes += stats["bytes"]
+            total_copied_files += copied_count
+            
+            if drive_count > 0 and copied_count >= drive_count:
+                status = "COMPLETE"
+            elif copied_count > 0:
+                pct = (copied_count / drive_count * 100) if drive_count > 0 else 0
+                status = f"IN PROGRESS ({pct:.1f}%)"
+            else:
+                status = "QUEUED / WAITING"
+            
+            print(f"{folder:<35} | {drive_count:<12} | {drive_size:<12} | {copied_count:<10} | {status:<15}")
+            
+            results.append({
+                "folder": folder,
+                "drive_files": drive_count,
+                "drive_size": drive_size,
+                "uploaded_files": copied_count,
+                "status": status
+            })
+        
+        print("-" * len(hdr))
+        total_gb = total_drive_bytes / (1024 ** 3)
+        summary_str = f"TOTAL: {total_drive_files} files ({total_gb:.2f} GiB) | Uploaded: {total_copied_files} files"
+        print(f"\n{summary_str}")
+        
+        # Save CSV if requested
+        if args.csv:
+            import csv
+            with open(args.csv, "w", newline="", encoding="utf-8") as csvfile:
+                writer = csv.DictWriter(csvfile, fieldnames=["folder", "drive_files", "drive_size", "uploaded_files", "status"])
+                writer.writeheader()
+                writer.writerows(results)
+            print(f"Saved CSV report to: {args.csv}")
+
+    finally:
+        if os.path.exists(config_path):
+            os.remove(config_path)
+
+if __name__ == "__main__":
+    main()

k8s/cert-manager/production-issuer.yaml

@@ -18,4 +18,4 @@ spec:
     solvers:
     - http01:
         ingress:
-          class: nginx
+          ingressClassName: traefik

k8s/cert-manager/staging-issuer.yaml

@@ -18,4 +18,4 @@ spec:
     solvers:
     - http01:
         ingress:
-          class:  nginx
+          ingressClassName: traefik

k8s/datalab/corona-api.yaml

@@ -1,86 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: corona-api
-  labels:
-    app: corona-api
-  namespace: datalab
-spec:
-  selector:
-    matchLabels:
-      app: corona-api
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: corona-api
-    spec:
-      imagePullSecrets:
-      - name: "registry-haumdaucher-de"
-      containers:
-      - image: registry.haumdaucher.de/corona_plumber:latest
-        imagePullPolicy: Always
-        name: corona-api
-        ports:
-        - containerPort: 8000
-
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: corona-api
-  namespace: datalab
-spec:
-  ports:
-  - port: 8000
-    targetPort: 8000
-    protocol: TCP
-  selector:
-    app: corona-api
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-api-haumdaucher
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona-api.haumdaucher.de"
-     secretName: corona-api-haumdaucher-de
-  rules:
-  - host: corona-api.haumdaucher.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona-api
-          servicePort: 8000
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-api-moritzgraf
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona-api.moritzgraf.de"
-     secretName: corona-api-moritzgraf-de
-  rules:
-  - host: corona-api.moritzgraf.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona-api
-          servicePort: 8000

k8s/datalab/corona.yaml

@@ -1,86 +0,0 @@
----
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: corona
-  labels:
-    app: corona
-  namespace: datalab
-spec:
-  selector:
-    matchLabels:
-      app: corona
-  replicas: 1
-  template:
-    metadata:
-      labels:
-        app: corona
-    spec:
-      imagePullSecrets:
-      - name: "registry-haumdaucher-de"
-      containers:
-      - image: registry.haumdaucher.de/corona:latest
-        imagePullPolicy: Always
-        name: corona
-        ports:
-        - containerPort: 3838
-
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: corona
-  namespace: datalab
-spec:
-  ports:
-  - port: 3838
-    targetPort: 3838
-    protocol: TCP
-  selector:
-    app: corona
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-haumdaucher
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona.haumdaucher.de"
-     secretName: corona-haumdaucher-de
-  rules:
-  - host: corona.haumdaucher.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona
-          servicePort: 3838
----
-apiVersion: extensions/v1beta1
-kind: Ingress
-metadata:
-  name: corona-moritzgraf
-  namespace: datalab
-  annotations:
-    kubernetes.io/ingress.class: "nginx"    
-    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
-    cert-manager.io/cluster-issuer: "letsencrypt-prod"
-spec:
-  tls:
-   - hosts:
-     - "corona.moritzgraf.de"
-     secretName: corona-moritzgraf-de
-  rules:
-  - host: corona.moritzgraf.de
-    http:
-      paths:
-      - path: /
-        backend:
-          serviceName: corona
-          servicePort: 3838

k8s/datalab/redeploy.yaml

@@ -1,79 +0,0 @@
-apiVersion: v1
-kind: ServiceAccount
-metadata:
-  creationTimestamp: null
-  name: redeploy
-  namespace: datalab
----
-apiVersion: rbac.authorization.k8s.io/v1
-kind: Role
-metadata:
-  namespace: datalab
-  name: redeploy
-rules:
-- apiGroups: ["apps"]
-  resources: ["deployments"]
-  verbs: ["get",list","patch"]
----
-apiVersion: rbac.authorization.k8s.io/v1
-# This role binding allows "jane" to read pods in the "default" namespace.
-# You need to already have a Role named "pod-reader" in that namespace.
-kind: RoleBinding
-metadata:
-  name: redeploy-pods
-  namespace: datalab
-subjects:
-# You can specify more than one "subject"
-- kind: ServiceAccount
-  name: redeploy
-roleRef:
-  # "roleRef" specifies the binding to a Role / ClusterRole
-  kind: Role #this must be Role or ClusterRole
-  name: redeploy
-  apiGroup: rbac.authorization.k8s.io
-
----
-apiVersion: batch/v1beta1
-kind: CronJob
-metadata:
-  name: redeploy-corona-api
-  namespace: datalab
-spec:
-  schedule: "0 5/12 * * *"
-  jobTemplate:
-    spec:
-      template:
-        spec:
-          serviceAccountName: redeploy
-          imagePullSecrets:
-          - name: "registry-haumdaucher-de"
-          containers:
-          - name: redeploy-corona-api
-            image: registry.haumdaucher.de/redeploy:latest
-            env:
-            - name: DEPLOYMENT
-              value: "corona-api"
-          restartPolicy: Never
-
----
-apiVersion: batch/v1beta1
-kind: CronJob
-metadata:
-  name: redeploy-corona
-  namespace: datalab
-spec:
-  schedule: "15 5/12 * * *"
-  jobTemplate:
-    spec:
-      template:
-        spec:
-          serviceAccountName: redeploy
-          imagePullSecrets:
-          - name: "registry-haumdaucher-de"
-          containers:
-          - name: redeploy-corona
-            image: registry.haumdaucher.de/redeploy:latest
-            env:
-            - name: DEPLOYMENT
-              value: "corona"
-          restartPolicy: Never

k8s/development/gitea-phpmyadmin.yml

@@ -1,13 +1,74 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gitea-db-adminer
+  namespace: development
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: gitea-db-adminer
+  template:
+    metadata:
+      labels:
+        app: gitea-db-adminer
+    spec:
+      containers:
+        - name: adminer
+          image: adminer:latest
+          ports:
+            - containerPort: 8080
+              name: http
+          env:
+            - name: ADMINER_DEFAULT_SERVER
+              value: "gitea-postgresql"
+            - name: ADMINER_DESIGN
+              value: "dracula"
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              memory: 128Mi
 ---
-db:
+apiVersion: v1
-  host: gitea-mariadb
+kind: Service
-ingress:
+metadata:
-  enabled: true
+  name: gitea-db-adminer
-  hosts:
+  namespace: development
-  - path: "/"
+spec:
-    tls: true
+  type: ClusterIP
-    name: "gitea.phpmyadmin.haumdaucher.de"
+  ports:
-    tlsSecret: "gitea-phpmyadmin-haumdaucher-de"
+    - port: 80
+      targetPort: 8080
+      name: http
+  selector:
+    app: gitea-db-adminer
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea-phpmyadmin
+  namespace: development
   annotations:
     cert-manager.io/cluster-issuer: "letsencrypt-prod"
-    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    kubernetes.io/ingress.class: "traefik"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - gitea.phpmyadmin.haumdaucher.de
+      secretName: gitea-phpmyadmin-haumdaucher-de
+  rules:
+    - host: gitea.phpmyadmin.haumdaucher.de
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: gitea-db-adminer
+                port:
+                  number: 80

k8s/development/registry_ingress.yaml

@@ -0,0 +1,42 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: docker-registry
+  namespace: development
+  annotations:
+    # --- ADDED: Match the working configuration ---
+    kubernetes.io/tls-acme: "true"
+    # ----------------------------------------------
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    kubernetes.io/ingress.class: traefik
+    meta.helm.sh/release-name: docker-registry
+    meta.helm.sh/release-namespace: development
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+  labels:
+    app: docker-registry
+    app.kubernetes.io/managed-by: Helm
+    chart: docker-registry-1.9.2
+    heritage: Helm
+    release: docker-registry
+spec:
+  # --- ADDED: Critical for modern K8s ---
+  ingressClassName: traefik 
+  # --------------------------------------
+  rules:
+  - host: registry.haumdaucher.de
+    http:
+      paths:
+      - backend:
+          service:
+            name: docker-registry
+            port:
+              number: 5000
+        path: /
+        # --- CHANGED: Recommended for consistency ---
+        pathType: Prefix
+        # --------------------------------------------
+  tls:
+  - hosts:
+    - registry.haumdaucher.de
+    secretName: registry-haumdaucher-de

k8s/family/duckdb-backend.yaml

@@ -0,0 +1,154 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: duckdb-backend
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: duckdb-backend
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: duckdb-backend
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      initContainers:
+        - name: git-init
+          image: alpine/git:latest
+          command: ["/bin/sh", "-c"]
+          args:
+            - |
+              set -e
+              export HOME=/tmp
+              cd /wiki
+              if [ ! -d ".git" ]; then
+                echo "Initializing wiki repository on dedicated PVC..."
+                REPO_URL="https://${GIT_SYNC_USERNAME}:${GIT_SYNC_PASSWORD}@git.moritzgraf.de/moritz/knowledge-wiki.git"
+                git config --global http.sslVerify false
+                git config --global --add safe.directory /wiki
+                git config --global --add safe.directory '*'
+                git init
+                git remote add origin "$REPO_URL" || git remote set-url origin "$REPO_URL"
+                git fetch origin main
+                git checkout -f -B main origin/main
+                echo "Clone completed successfully."
+              else
+                echo "Repository already initialized on PVC, updating..."
+                REPO_URL="https://${GIT_SYNC_USERNAME}:${GIT_SYNC_PASSWORD}@git.moritzgraf.de/moritz/knowledge-wiki.git"
+                git config --global http.sslVerify false
+                git config --global --add safe.directory /wiki
+                git config --global --add safe.directory '*'
+                git remote set-url origin "$REPO_URL"
+                if [ -f .git/shallow ]; then
+                  echo "Unshallowing existing repository..."
+                  git fetch --unshallow origin main || git fetch origin main
+                else
+                  git fetch origin main
+                fi
+                git rebase --abort 2>/dev/null || true
+                git merge --abort 2>/dev/null || true
+                git reset --hard origin/main
+                git clean -fd
+              fi
+          env:
+            - name: GIT_SYNC_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: git-sync-username
+            - name: GIT_SYNC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: wiki-sync-token
+          volumeMounts:
+            - name: duckdb-wiki-data
+              mountPath: /wiki
+      containers:
+        - name: duckdb-backend
+          image: registry.haumdaucher.de/family/duckdb-backend:latest
+          imagePullPolicy: Always
+          env:
+            - name: DATA_DIR
+              value: "/wiki/data"
+            - name: DUCKDB_DATA_DIR
+              value: "/wiki/data"
+            - name: WIKI_DIR
+              value: "/wiki"
+            - name: GIT_REPO_DIR
+              value: "/wiki"
+            - name: GIT_REPO_URL
+              value: "https://git.moritzgraf.de/moritz/knowledge-wiki.git"
+            - name: GIT_REMOTE_URL
+              value: "https://git.moritzgraf.de/moritz/knowledge-wiki.git"
+            - name: GIT_SYNC_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: git-sync-username
+            - name: GIT_SYNC_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: wiki-git-credentials
+                  key: wiki-sync-token
+            - name: GIT_SYNC_INTERVAL_SECONDS
+              value: "60"
+            - name: PORT
+              value: "8000"
+            - name: HOST
+              value: "0.0.0.0"
+          ports:
+            - name: http
+              containerPort: 8000
+          volumeMounts:
+            - name: duckdb-wiki-data
+              mountPath: /wiki
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 1000m
+              memory: 1Gi
+          readinessProbe:
+            httpGet:
+              path: /duckdb-api/v1/healthz
+              port: 8000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /duckdb-api/v1/healthz
+              port: 8000
+            initialDelaySeconds: 15
+            periodSeconds: 20
+      volumes:
+        - name: duckdb-wiki-data
+          persistentVolumeClaim:
+            claimName: duckdb-wiki-data
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: duckdb-backend
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 8000
+      targetPort: 8000
+      name: http
+  selector:
+    app.kubernetes.io/name: duckdb-backend

k8s/family/ingress.yaml

@@ -0,0 +1,55 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: family-portal
+  namespace: family
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    traefik.ingress.kubernetes.io/router.middlewares: auth-authentik-forwardauth@kubernetescrd
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - family.haumdaucher.de
+      secretName: family-haumdaucher-de-tls
+  rules:
+    - host: family.haumdaucher.de
+      http:
+        paths:
+          - path: /duckdb-api
+            pathType: Prefix
+            backend:
+              service:
+                name: duckdb-backend
+                port:
+                  number: 8000
+          - path: /tracking/mila
+            pathType: Prefix
+            backend:
+              service:
+                name: mila
+                port:
+                  number: 80
+          - path: /tracking/moritz
+            pathType: Prefix
+            backend:
+              service:
+                name: moritz
+                port:
+                  number: 80
+          - path: /outpost.goauthentik.io
+            pathType: Prefix
+            backend:
+              service:
+                name: duckdb-backend
+                port:
+                  number: 8000
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: mila
+                port:
+                  number: 80

k8s/family/mila.yaml

@@ -0,0 +1,64 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: mila
+  namespace: family
+  labels:
+    app.kubernetes.io/name: mila
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: mila
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: mila
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: mila
+          image: registry.haumdaucher.de/family/mila:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 80
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /tracking/mila/
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /tracking/mila/
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: mila
+  namespace: family
+  labels:
+    app.kubernetes.io/name: mila
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app.kubernetes.io/name: mila

k8s/family/moritz.yaml

@@ -0,0 +1,64 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: moritz
+  namespace: family
+  labels:
+    app.kubernetes.io/name: moritz
+    app.kubernetes.io/part-of: family-portal
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: moritz
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: moritz
+        app.kubernetes.io/part-of: family-portal
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: moritz
+          image: registry.haumdaucher.de/family/moritz:latest
+          imagePullPolicy: Always
+          ports:
+            - name: http
+              containerPort: 80
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /tracking/moritz/
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /tracking/moritz/
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: moritz
+  namespace: family
+  labels:
+    app.kubernetes.io/name: moritz
+    app.kubernetes.io/part-of: family-portal
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app.kubernetes.io/name: moritz

k8s/family/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: family
+  labels:
+    name: family

k8s/family/pvc.yaml

@@ -0,0 +1,15 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: duckdb-wiki-data
+  namespace: family
+  labels:
+    app.kubernetes.io/name: duckdb-backend
+    app.kubernetes.io/part-of: family-portal
+spec:
+  accessModes:
+    - ReadWriteOnce
+  storageClassName: openebs-hostpath
+  resources:
+    requests:
+      storage: 5Gi

k8s/grabengasse/backend.yaml

@@ -0,0 +1,65 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grabengasse-backend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-backend
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grabengasse-backend
+  template:
+    metadata:
+      labels:
+        app: grabengasse-backend
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: backend
+          image: registry.haumdaucher.de/website/grabengasse_backend:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 8000
+              name: http
+          env:
+            - name: PROMETHEUS_URL
+              value: "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090"
+          resources:
+            requests:
+              cpu: 50m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
+          readinessProbe:
+            httpGet:
+              path: /healthz
+              port: 8000
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /healthz
+              port: 8000
+            initialDelaySeconds: 10
+            periodSeconds: 15
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grabengasse-backend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-backend
+spec:
+  type: ClusterIP
+  ports:
+    - port: 8000
+      targetPort: 8000
+      name: http
+  selector:
+    app: grabengasse-backend

k8s/grabengasse/frontend.yaml

@@ -0,0 +1,62 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grabengasse-frontend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-frontend
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: grabengasse-frontend
+  template:
+    metadata:
+      labels:
+        app: grabengasse-frontend
+    spec:
+      imagePullSecrets:
+        - name: registry-haumdaucher-de
+      containers:
+        - name: frontend
+          image: registry.haumdaucher.de/website/grabengasse_frontend:latest
+          imagePullPolicy: Always
+          ports:
+            - containerPort: 80
+              name: http
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 128Mi
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 15
+
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grabengasse-frontend
+  namespace: grabengasse
+  labels:
+    app: grabengasse-frontend
+spec:
+  type: ClusterIP
+  ports:
+    - port: 80
+      targetPort: 80
+      name: http
+  selector:
+    app: grabengasse-frontend

k8s/grabengasse/ingress.yaml

@@ -0,0 +1,35 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grabengasse7
+  namespace: grabengasse
+  annotations:
+    kubernetes.io/ingress.class: "traefik"
+    cert-manager.io/cluster-issuer: "letsencrypt-prod"
+    kubernetes.io/tls-acme: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+spec:
+  ingressClassName: traefik
+  tls:
+    - hosts:
+        - grabengasse7.de
+      secretName: grabengasse7-de
+  rules:
+    - host: grabengasse7.de
+      http:
+        paths:
+          - path: /api
+            pathType: Prefix
+            backend:
+              service:
+                name: grabengasse-backend
+                port:
+                  number: 8000
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: grabengasse-frontend
+                port:
+                  number: 80

k8s/grabengasse/namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: grabengasse
+  labels:
+    name: grabengasse

k8s/haumdaucher/haumdaucher.yaml

@@ -0,0 +1,71 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: haumdaucher-website
+  namespace: haumdaucher
+  labels:
+    app: haumdaucher
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: haumdaucher
+  template:
+    metadata:
+      labels:
+        app: haumdaucher
+    spec:
+      imagePullSecrets:
+      - name: registry-haumdaucher-de
+      containers:
+      - name: haumdaucher
+        image: registry.haumdaucher.de/haumdaucher-website:latest
+        imagePullPolicy: Always
+        ports:
+        - containerPort: 80
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: haumdaucher-service
+  namespace: haumdaucher
+spec:
+  selector:
+    app: haumdaucher
+  ports:
+    - protocol: TCP
+      port: 80
+      targetPort: 80
+  type: ClusterIP
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: haumdaucher-ingress
+  namespace: haumdaucher
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    kubernetes.io/ingress.class: nginx
+    kubernetes.io/tls-acme: "true"
+    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/proxy-body-size: "0"
+    nginx.ingress.kubernetes.io/proxy-buffering: "off"
+    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: haumdaucher.de
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: haumdaucher-service
+            port:
+              number: 80
+  tls:
+  - hosts:
+    - haumdaucher.de
+    secretName: haumdaucher-de-tls

k8s/home-assistant/README.md

@@ -0,0 +1,27 @@
+# Home Assistant
+
+Deployment for Home Assistant via the `pajikos` Helm chart.
+
+## Deployment
+
+1. Ensure the Helm repository is added and updated:
+```bash
+helm repo add pajikos https://pajikos.github.io/home-assistant-helm-chart/
+helm repo update
+```
+
+2. Apply the raw secrets (if they haven't been applied yet):
+```bash
+kubectl apply -f hass-code-auth.secret.yml
+```
+
+3. Upgrade/Install the Helm release:
+```bash
+helm upgrade --install home-assistant pajikos/home-assistant -n home-assistant -f home-assistant.secret.yaml
+```
+
+## Architecture & Configuration
+This setup perfectly balances **Infrastructure as Code (IaC)** with **Dynamic UI Configuration**:
+- **Core Settings:** Network proxies, InfluxDB tokens, and Ingress routing are strictly enforced via `home-assistant.secret.yaml`.
+- **Dynamic Config:** The helm chart utilizes an init-script to inject `!include` directives for `automations.yaml`, `scripts.yaml`, and `scenes.yaml` into the main config. This allows the Home Assistant UI to write dynamic automations without violating the IaC principles!
+- **Code Server Addon:** For advanced manual tweaks, a VS Code instance (`hass-coder.moritzgraf.de`) runs alongside the pod, allowing direct manipulation of the `/config` directory.