docs(openclaw): update Issue 16 with fallbacks, koch grounding, and prometheus rules

k8s/openclaw/TROUBLESHOOTING.md

@@ -26,6 +26,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
 | **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
 | **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
+| **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. |
 
 ---
 
@@ -472,3 +473,40 @@ This document captures fixed issues, operational quirks, and incident root-cause
   - Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
   - Rely on persistent session context and structured stores (`skills/datastore`) for state retention.
 
+---
+
+### Issue 16: Gateway Startup CrashLoop from Orphaned SQLite Foreign Keys & Silent Reasoning Drops in Telegram Topics
+
+- **Date**: 2026-10-06
+- **Severity**: Critical (Full Telegram bot outage due to pod `CrashLoopBackOff`, and silent dropped turns in `#Kochen` and `#Grabengasse & Media`)
+- **Symptom**:
+  1. `openclaw-0` pod was stuck in `CrashLoopBackOff` with 7+ restarts. Gateway logs reported:
+     ```text
+     Gateway failed to start: OpenClaw agent database /home/openclaw/.openclaw/agents/itguy/agent/openclaw-agent.sqlite is quarantined after integrity verification failed: SQLite foreign_key_check failed for ...: session_transcript_index_state row 5 references session_windows (foreign key 0); trajectory_runtime_events row 379 references session_windows ...
+     ```
+  2. Prior to the crash, Telegram Topic 21 (`#Kochen`) completely ignored an incoming user message containing 4 photos of *körniger Frischkäse* (Edeka Herzstücke & Philadelphia) with zero visible reply sent to Telegram.
+  3. When the pod was restarted during an active voice note turn in Topic 164 (`#Grabengasse & Media`), the session was left wedged in `status: running` with active recovery locks.
+- **Root Cause**:
+  1. **Orphaned Foreign Key Cascade Failure**: During cron session pruning in `setup-compaction-cron.sh`, records were deleted from `session_nodes` and `session_windows` without cascading into child tables (`trajectory_runtime_events`, `session_transcript_index_state`, `transcript_rewrite_watermarks`). Furthermore, Python's `sqlite3` module disables foreign keys by default (`PRAGMA foreign_keys = OFF`). On gateway startup, OpenClaw enables foreign keys and runs `PRAGMA foreign_key_check`. Detecting 46 violations in `itguy`, OpenClaw quarantined the DB in `openclaw-quarantine.sqlite` and crashed.
+  2. **InitContainer Execution Gap**: Although `openclaw-instance.secret.yaml` contained an inline Python snippet to repair foreign keys, `initContainers[0]` lacked `/home/node/.openclaw/.local/bin` in its `PATH`. `python3` failed with command not found, leaving the quarantine intact.
+  3. **Silent Reasoning Dropped Turn in `z-ai/glm-5.3-flash`**: In Topic 21, the model attempted to use the legacy `edit` tool on `nutrition-log/moritz/2026-10.md` to update a past yogurt entry. When `edit` failed, GLM-5.3-Flash emitted 63 reasoning tokens (*"Let me view the images first..."*), but then immediately terminated with `stopReason: "stop"` without generating any tool calls or visible text. OpenClaw marked the turn as `done`, delivering 0 text to Telegram.
+- **Resolution**:
+  1. **Database Repair**: Cleaned up all 46 orphaned child records in `itguy/agent/openclaw-agent.sqlite`, checkpointed the WAL journal, and cleared the quarantine record in `openclaw-quarantine.sqlite`.
+  2. **Session Normalization**: Repaired `grabengasse_media` (Topic 164) and `koch` (Topic 21), resetting `status: "done"`, clearing recovery run IDs, and restoring `entry_valid = 1`.
+  3. **InitContainer Self-Healing**: Injected `export PATH="/home/node/.openclaw/bin:/home/node/.openclaw/.local/bin:$PATH"` and added automated pre-flight self-healing to `initContainers[0]` in `openclaw-instance.secret.yaml`. On every container boot, the script now automatically cascades foreign key repairs, unlocks hung `running` sessions, and clears quarantine entries.
+  4. **Maintenance Script Hardening**: Updated `k8s/openclaw/setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` with `PRAGMA foreign_keys = ON;`, explicit deletion across child tables, and unconditional FK verification across all 12 agent databases.
+  5. **Model Fallback Chain for Subagents**: Replaced bare string model configurations (`model: "openrouter/z-ai/glm-5.3-flash"`) across all 11 subagents in `k8s/openclaw/openclaw-instance.secret.yaml` with structured objects specifying explicit fallbacks (`primary: "openrouter/z-ai/glm-5.3-flash"`, `fallbacks: ["openrouter/deepseek/deepseek-v4.1-flash", "openrouter/qwen/qwen3.8-flash"]`). This prevents reasoning dropouts on GLM-5.3-Flash from halting turns and eliminates the OpenClaw Doctor configuration warning.
+  6. **Koch Agent Grounding & Standard Foods Catalog**: 
+     - Added EDEKA Bio Herzstücke Körniger Frischkäse (150 g) and Philadelphia Der Körnige (200 g) to `knowledge-wiki/entities/standard-foods.md` and `knowledge-wiki/data/entities/foods.jsonl`.
+     - Hardened `knowledge-wiki/agents/koch.md` with explicit distinction: capturing packaging photos / ingredients is NOT intake, never touches `entities/nutrition-log/`, and must only update `foods.jsonl` / `standard-foods.md`. Intake logging is strictly prohibited from using `edit` and must use `datastore.py add --kind nutrition.intake`.
+  7. **Observability & Alerting**: Created `k8s/openclaw/monitoring.yaml` deploying a `PrometheusRule` (`openclaw-alerts`) covering:
+     - `OpenClawPodNotReady`: Alerts (critical) if `openclaw-0` is unready for >5 minutes.
+     - `OpenClawCrashLooping`: Alerts (warning) if container restarts exceed 2 within 15 minutes.
+     - `OpenClawDataVolumeLow`: Alerts (warning) if PVC `openclaw-data` has <15% free disk space.
+- **How to Avoid Recurrence**:
+  - Always ensure `PRAGMA foreign_keys = ON;` is set on every SQLite connection performing deletes.
+  - Rely on the initContainer self-healing hook to automatically heal state before the gateway daemon starts.
+  - Avoid legacy Markdown editing for nutrition logs; enforce structured datastore CLI tooling (`skills/datastore`).
+  - Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly.
+  - Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies.
+

k8s/openclaw/monitoring.yaml

@@ -0,0 +1,36 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: openclaw-alerts
+  namespace: openclaw
+  labels:
+    release: prometheus-operator
+    app.kubernetes.io/name: openclaw
+spec:
+  groups:
+    - name: openclaw.rules
+      rules:
+        - alert: OpenClawPodNotReady
+          expr: kube_pod_status_ready{namespace="openclaw", pod="openclaw-0", condition="true"} == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "OpenClaw pod openclaw-0 is not ready"
+            description: "Pod openclaw-0 in namespace openclaw has been unready for more than 5 minutes. Telegram and background agents may be offline."
+        - alert: OpenClawCrashLooping
+          expr: increase(kube_pod_container_status_restarts_total{namespace="openclaw", pod="openclaw-0"}[15m]) >= 2
+          for: 2m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenClaw container restart detected"
+            description: "Container {{ $labels.container }} in pod openclaw-0 has restarted {{ $value }} times in the last 15 minutes."
+        - alert: OpenClawDataVolumeLow
+          expr: kubelet_volume_stats_available_bytes{namespace="openclaw", persistentvolumeclaim="openclaw-data"} / kubelet_volume_stats_capacity_bytes{namespace="openclaw", persistentvolumeclaim="openclaw-data"} < 0.15
+          for: 15m
+          labels:
+            severity: warning
+          annotations:
+            summary: "OpenClaw PVC data volume is low on space"
+            description: "PersistentVolumeClaim openclaw-data has less than 15% space available."