fix(openclaw): add pre-flight self-healing init hook, repair FK cascade, and document Issue 16

k8s/openclaw/TROUBLESHOOTING.md

@@ -26,6 +26,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
 | **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
 | **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
+| **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. |
 
 ---
 
@@ -472,3 +473,30 @@ This document captures fixed issues, operational quirks, and incident root-cause
   - Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
   - Rely on persistent session context and structured stores (`skills/datastore`) for state retention.
 
+---
+
+### Issue 16: Gateway Startup CrashLoop from Orphaned SQLite Foreign Keys & Silent Reasoning Drops in Telegram Topics
+
+- **Date**: 2026-10-06
+- **Severity**: Critical (Full Telegram bot outage due to pod `CrashLoopBackOff`, and silent dropped turns in `#Kochen` and `#Grabengasse & Media`)
+- **Symptom**:
+  1. `openclaw-0` pod was stuck in `CrashLoopBackOff` with 7+ restarts. Gateway logs reported:
+     ```text
+     Gateway failed to start: OpenClaw agent database /home/openclaw/.openclaw/agents/itguy/agent/openclaw-agent.sqlite is quarantined after integrity verification failed: SQLite foreign_key_check failed for ...: session_transcript_index_state row 5 references session_windows (foreign key 0); trajectory_runtime_events row 379 references session_windows ...
+     ```
+  2. Prior to the crash, Telegram Topic 21 (`#Kochen`) completely ignored an incoming user message containing 4 photos of *körniger Frischkäse* (Edeka Herzstücke & Philadelphia) with zero visible reply sent to Telegram.
+  3. When the pod was restarted during an active voice note turn in Topic 164 (`#Grabengasse & Media`), the session was left wedged in `status: running` with active recovery locks.
+- **Root Cause**:
+  1. **Orphaned Foreign Key Cascade Failure**: During cron session pruning in `setup-compaction-cron.sh`, records were deleted from `session_nodes` and `session_windows` without cascading into child tables (`trajectory_runtime_events`, `session_transcript_index_state`, `transcript_rewrite_watermarks`). Furthermore, Python's `sqlite3` module disables foreign keys by default (`PRAGMA foreign_keys = OFF`). On gateway startup, OpenClaw enables foreign keys and runs `PRAGMA foreign_key_check`. Detecting 46 violations in `itguy`, OpenClaw quarantined the DB in `openclaw-quarantine.sqlite` and crashed.
+  2. **InitContainer Execution Gap**: Although `openclaw-instance.secret.yaml` contained an inline Python snippet to repair foreign keys, `initContainers[0]` lacked `/home/node/.openclaw/.local/bin` in its `PATH`. `python3` failed with command not found, leaving the quarantine intact.
+  3. **Silent Reasoning Dropped Turn in `z-ai/glm-5.3-flash`**: In Topic 21, the model attempted to use the legacy `edit` tool on `nutrition-log/moritz/2026-10.md` to update a past yogurt entry. When `edit` failed, GLM-5.3-Flash emitted 63 reasoning tokens (*"Let me view the images first..."*), but then immediately terminated with `stopReason: "stop"` without generating any tool calls or visible text. OpenClaw marked the turn as `done`, delivering 0 text to Telegram.
+- **Resolution**:
+  1. **Database Repair**: Cleaned up all 46 orphaned child records in `itguy/agent/openclaw-agent.sqlite`, checkpointed the WAL journal, and cleared the quarantine record in `openclaw-quarantine.sqlite`.
+  2. **Session Normalization**: Repaired `grabengasse_media` (Topic 164) and `koch` (Topic 21), resetting `status: "done"`, clearing recovery run IDs, and restoring `entry_valid = 1`.
+  3. **InitContainer Self-Healing**: Injected `export PATH="/home/node/.openclaw/bin:/home/node/.openclaw/.local/bin:$PATH"` and added automated pre-flight self-healing to `initContainers[0]` in `openclaw-instance.secret.yaml`. On every container boot, the script now automatically cascades foreign key repairs, unlocks hung `running` sessions, and clears quarantine entries.
+  4. **Maintenance Script Hardening**: Updated `k8s/openclaw/setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` with `PRAGMA foreign_keys = ON;`, explicit deletion across child tables, and unconditional FK verification across all 12 agent databases.
+- **How to Avoid Recurrence**:
+  - Always ensure `PRAGMA foreign_keys = ON;` is set on every SQLite connection performing deletes.
+  - Rely on the initContainer self-healing hook to automatically heal state before the gateway daemon starts.
+  - Avoid legacy Markdown editing for nutrition logs; enforce structured datastore CLI tooling (`skills/datastore`).
+