fix(openclaw): harden probes, runtime self-healing preflight and session recovery

k8s/openclaw/TROUBLESHOOTING.md

@@ -537,3 +537,43 @@ This document captures fixed issues, operational quirks, and incident root-cause
   - Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding.
   - Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks.
 
+---
+
+### Issue 18: Spinning Disk I/O Bottlenecks, In-Pod Restart Bypass, and Gateway Preflight Self-Healing
+
+- **Date**: 2026-10-07
+- **Severity**: Critical (Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart)
+- **Symptom**:
+  Moritz sent an audio message in Telegram Topic 21 (`#Kochen`):
+  *„Ich frühstücke wieder meine Oats, bestehend aus 50 Gramm meiner eigenen Mischung, 160 Milliliter Oatley Hafermilch und 50 Gramm Früchte gefroren.“*
+  The bot never responded. Subsequent messages in Topic 21 were completely ignored.
+- **Root Cause**:
+  1. **Spinning Disk I/O Latency**: Host `haumdaucher` uses mechanical HDD (`/dev/sda2`, `rotational: 1`). Under heavy background disk operations (nightly backups, index generation), random 4KB read operations are throttled to ~70–100 IOPS, blocking Node.js synchronous operations.
+  2. **Aggressive Liveness Probe Failure**: OpenClaw's liveness probe had a 120s tolerance window (`failureThreshold: 12, periodSeconds: 10`). An event-loop pause during disk I/O exceeded the deadline 12 consecutive times, causing Kubelet to kill the `openclaw` container mid-turn.
+  3. **The InitContainer Bypass Trap**: When Kubelet restarts an application container inside an existing Pod, **`initContainers` do NOT re-run**. The session repair and quarantine clearing scripts situated exclusively in `initContainers` were completely bypassed!
+  4. **Session Deadlock & Stale Lock**: Because container 1 was killed mid-turn:
+     - The session node `agent:koch:telegram:group:-1004326921510:topic:21` was left with `status: "running"` and `activeWriterRunId: "..."`. OpenClaw's `reply-admission-ticket.js` rejected all subsequent incoming Telegram messages on that topic as locked.
+     - Stale `/tmp/openclaw-1000/gateway.state.lock` caused container 2 to crash with `gateway.state.lock.reclaim: ENOENT`.
+  5. **Cold Database Opening Delays**: Container 3 started cold and was forced to open all 12 agent SQLite databases (~1.9 GB total) directly from HDD seeks, blocking the event loop for up to 50 minutes.
+- **Resolution**:
+  1. **Immediate State Recovery**:
+     - Pre-warmed all SQLite database files into the Linux OS page cache via sequential streaming (`cat ... > /dev/null`), dropping read latency from 50 minutes to 312 milliseconds.
+     - Unlocked the Koch session node and session window in SQLite (`status: "done"`, `activeWriterRunId: null`, `entry_valid = 1`).
+  2. **Intake Logged & Views Rendered**:
+     - Recorded the Oats breakfast into DuckDB via `datastore.py add --kind nutrition.intake` (260 g, ~311 kcal, 13.9 g Eiweiß, 37.9 g KH, 9.2 g Fett, 7.9 g Ballaststoffe, 0.24 g Salz) with `food_id = null` (composite dish) with timestamp `2026-10-07T07:55:00+02:00`.
+     - Regenerated Markdown views (`entities/nutrition-log/moritz/2026-10.md`) and HTML dashboards with `render_views.py`, committed and pushed to `knowledge-wiki`.
+     - Sent confirmation with complete macro totals directly to Telegram Topic 21 via `openclaw message send`.
+  3. **Permanent Gateway Preflight Self-Healing (`NODE_OPTIONS`)**:
+     - Configured `NODE_OPTIONS="--max-old-space-size=3072 --import=/home/node/.openclaw/scripts/gateway-preflight.mjs"`.
+     - Because Node.js evaluates `NODE_OPTIONS` on **every single process launch**, `gateway-preflight.mjs` executes before OpenClaw gateway starts—even during in-pod container restarts where `initContainers` are bypassed.
+     - Performs automatic lock cleanup, quarantine clearing, SQLite page cache pre-warming, and un-wedges any orphaned `running` sessions in <600ms.
+  4. **Probe Hardening**:
+     - Increased liveness tolerance window in `k8s/openclaw/openclaw-instance.secret.yaml` to 450s (`failureThreshold: 30, periodSeconds: 15, timeoutSeconds: 15`).
+     - Increased startup probe timeout to 15s to tolerate transient event-loop spikes without false kills.
+  5. **Nightly Compaction Cron Enhancement**:
+     - Updated `setup-compaction-cron.sh` and `compact_sessions.py` to run session recovery, `PRAGMA optimize;`, and `VACUUM;` nightly to prevent database bloat and optimize query plans.
+- **How to Avoid Recurrence**:
+  - Never place critical recovery logic solely in `initContainers`; use application runtime hooks or `NODE_OPTIONS` for in-pod restart resilience.
+  - Set probe failure thresholds and timeouts with high tolerance on rotational HDD nodes.
+  - Maintain page cache pre-warming for SQLite databases to convert slow random seek I/O into fast sequential streaming.
+

k8s/openclaw/setup-compaction-cron.sh

@@ -192,11 +192,29 @@ def checkpoint_wal_journals():
                 print(f"  ✗ Error checkpointing {db}: {e}")
     print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.")
 
+def vacuum_and_optimize():
+    print("=== Running SQLite VACUUM and OPTIMIZE across all databases ===")
+    dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
+    for db in dbs:
+        try:
+            con = sqlite3.connect(db, timeout=60.0)
+            con.execute("PRAGMA optimize;")
+            con.execute("VACUUM;")
+            con.close()
+        except Exception as e:
+            print(f"  ✗ Error vacuuming {db}: {e}")
+    print("SQLite VACUUM and optimization complete.")
+
 def main():
     prewarm_page_cache()
+    try:
+        subprocess.run(["python3", "/home/node/.openclaw/scripts/recover_sessions.py"], check=False)
+    except Exception:
+        pass
     prune_stale_cron_sessions()
     compact_active_sessions()
     checkpoint_wal_journals()
+    vacuum_and_optimize()
 
 if __name__ == "__main__":
     main()