diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index 8e6bec3..1286be4 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -537,3 +537,43 @@ This document captures fixed issues, operational quirks, and incident root-cause - Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding. - Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks. +--- + +### Issue 18: Spinning Disk I/O Bottlenecks, In-Pod Restart Bypass, and Gateway Preflight Self-Healing + +- **Date**: 2026-10-07 +- **Severity**: Critical (Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart) +- **Symptom**: + Moritz sent an audio message in Telegram Topic 21 (`#Kochen`): + *„Ich frühstücke wieder meine Oats, bestehend aus 50 Gramm meiner eigenen Mischung, 160 Milliliter Oatley Hafermilch und 50 Gramm Früchte gefroren.“* + The bot never responded. Subsequent messages in Topic 21 were completely ignored. +- **Root Cause**: + 1. **Spinning Disk I/O Latency**: Host `haumdaucher` uses mechanical HDD (`/dev/sda2`, `rotational: 1`). Under heavy background disk operations (nightly backups, index generation), random 4KB read operations are throttled to ~70–100 IOPS, blocking Node.js synchronous operations. + 2. **Aggressive Liveness Probe Failure**: OpenClaw's liveness probe had a 120s tolerance window (`failureThreshold: 12, periodSeconds: 10`). An event-loop pause during disk I/O exceeded the deadline 12 consecutive times, causing Kubelet to kill the `openclaw` container mid-turn. + 3. **The InitContainer Bypass Trap**: When Kubelet restarts an application container inside an existing Pod, **`initContainers` do NOT re-run**. The session repair and quarantine clearing scripts situated exclusively in `initContainers` were completely bypassed! + 4. **Session Deadlock & Stale Lock**: Because container 1 was killed mid-turn: + - The session node `agent:koch:telegram:group:-1004326921510:topic:21` was left with `status: "running"` and `activeWriterRunId: "..."`. OpenClaw's `reply-admission-ticket.js` rejected all subsequent incoming Telegram messages on that topic as locked. + - Stale `/tmp/openclaw-1000/gateway.state.lock` caused container 2 to crash with `gateway.state.lock.reclaim: ENOENT`. + 5. **Cold Database Opening Delays**: Container 3 started cold and was forced to open all 12 agent SQLite databases (~1.9 GB total) directly from HDD seeks, blocking the event loop for up to 50 minutes. +- **Resolution**: + 1. **Immediate State Recovery**: + - Pre-warmed all SQLite database files into the Linux OS page cache via sequential streaming (`cat ... > /dev/null`), dropping read latency from 50 minutes to 312 milliseconds. + - Unlocked the Koch session node and session window in SQLite (`status: "done"`, `activeWriterRunId: null`, `entry_valid = 1`). + 2. **Intake Logged & Views Rendered**: + - Recorded the Oats breakfast into DuckDB via `datastore.py add --kind nutrition.intake` (260 g, ~311 kcal, 13.9 g Eiweiß, 37.9 g KH, 9.2 g Fett, 7.9 g Ballaststoffe, 0.24 g Salz) with `food_id = null` (composite dish) with timestamp `2026-10-07T07:55:00+02:00`. + - Regenerated Markdown views (`entities/nutrition-log/moritz/2026-10.md`) and HTML dashboards with `render_views.py`, committed and pushed to `knowledge-wiki`. + - Sent confirmation with complete macro totals directly to Telegram Topic 21 via `openclaw message send`. + 3. **Permanent Gateway Preflight Self-Healing (`NODE_OPTIONS`)**: + - Configured `NODE_OPTIONS="--max-old-space-size=3072 --import=/home/node/.openclaw/scripts/gateway-preflight.mjs"`. + - Because Node.js evaluates `NODE_OPTIONS` on **every single process launch**, `gateway-preflight.mjs` executes before OpenClaw gateway starts—even during in-pod container restarts where `initContainers` are bypassed. + - Performs automatic lock cleanup, quarantine clearing, SQLite page cache pre-warming, and un-wedges any orphaned `running` sessions in <600ms. + 4. **Probe Hardening**: + - Increased liveness tolerance window in `k8s/openclaw/openclaw-instance.secret.yaml` to 450s (`failureThreshold: 30, periodSeconds: 15, timeoutSeconds: 15`). + - Increased startup probe timeout to 15s to tolerate transient event-loop spikes without false kills. + 5. **Nightly Compaction Cron Enhancement**: + - Updated `setup-compaction-cron.sh` and `compact_sessions.py` to run session recovery, `PRAGMA optimize;`, and `VACUUM;` nightly to prevent database bloat and optimize query plans. +- **How to Avoid Recurrence**: + - Never place critical recovery logic solely in `initContainers`; use application runtime hooks or `NODE_OPTIONS` for in-pod restart resilience. + - Set probe failure thresholds and timeouts with high tolerance on rotational HDD nodes. + - Maintain page cache pre-warming for SQLite databases to convert slow random seek I/O into fast sequential streaming. + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index c7bfa8f..33bb810 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/k8s/openclaw/setup-compaction-cron.sh b/k8s/openclaw/setup-compaction-cron.sh index a5104da..d81ad2b 100755 --- a/k8s/openclaw/setup-compaction-cron.sh +++ b/k8s/openclaw/setup-compaction-cron.sh @@ -192,11 +192,29 @@ def checkpoint_wal_journals(): print(f" ✗ Error checkpointing {db}: {e}") print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.") +def vacuum_and_optimize(): + print("=== Running SQLite VACUUM and OPTIMIZE across all databases ===") + dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite") + for db in dbs: + try: + con = sqlite3.connect(db, timeout=60.0) + con.execute("PRAGMA optimize;") + con.execute("VACUUM;") + con.close() + except Exception as e: + print(f" ✗ Error vacuuming {db}: {e}") + print("SQLite VACUUM and optimization complete.") + def main(): prewarm_page_cache() + try: + subprocess.run(["python3", "/home/node/.openclaw/scripts/recover_sessions.py"], check=False) + except Exception: + pass prune_stale_cron_sessions() compact_active_sessions() checkpoint_wal_journals() + vacuum_and_optimize() if __name__ == "__main__": main() diff --git a/knowledge-wiki b/knowledge-wiki index 2642060..3ef58a1 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 26420601349d25b2e64880c53ab94c92c8afc788 +Subproject commit 3ef58a1ce8c320cc5b1ce8faa33ff9b541563b02