fix(openclaw): harden 4Gi memory ceiling, increase liveness tolerance, add WAL checkpointing and cron delivery fix

k8s/openclaw/TROUBLESHOOTING.md

@@ -20,6 +20,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
 | **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
 | **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. |
+| **2026-10-02** | **Memory Cgroup OOMKill & Probe Flapping** | Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported `failed` delivery | `RESOLVED` | Injected `NODE_OPTIONS="--max-old-space-size=3072"`, raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added `--no-deliver` + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens). |
 
 ---
 
@@ -288,5 +289,33 @@ This document captures fixed issues, operational quirks, and incident root-cause
   - Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
   - Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.
 
+---
+
+### Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure
+
+- **Date**: 2026-10-02
+- **Severity**: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status)
+- **Symptom**:
+  1. Pod restarted at `2026-10-02T12:40:43Z` with `lastState: terminated (exitCode: 137, reason: OOMKilled)`.
+  2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (`context deadline exceeded while awaiting headers`).
+  3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit).
+  4. The nightly session compaction cron job `nightly-session-compaction` completed with `completionStatus: "failed"` due to unrouted announcement delivery.
+- **Root Cause**:
+  1. **Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting**: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit `--max-old-space-size` ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL.
+  2. **Aggressive Liveness Threshold on Single-Threaded Event Loop**: The liveness probe was set to `failureThreshold: 5, periodSeconds: 10` (50s tolerance). When disk I/O on the physical spinning HDD (`/dev/sda2`) or memory sync tasks delayed the Node.js event loop by 46 seconds (`overdue=16192ms elapsed=46192ms`), Kubelet killed the container.
+  3. **Unbound Cron Delivery**: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or `--no-deliver`, OpenClaw fails delivery closed (`Refusing implicit isolated cron delivery: delivery.to is required`).
+  4. **Uncheckpointed SQLite WAL Journals & Dead Sessions**: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens.
+- **Resolution**:
+  1. Injected `NODE_OPTIONS="--max-old-space-size=3072"` into `k8s/openclaw/openclaw-instance.secret.yaml` to enforce V8 garbage collection when heap reaches 3 GB.
+  2. Increased liveness probe tolerance to `failureThreshold: 12, timeoutSeconds: 15` (120s tolerance).
+  3. Lowered native safeguard compaction threshold `maxActiveTranscriptBytes` from `3mb` to `400kb` (~50k tokens).
+  4. Updated `k8s/openclaw/setup-compaction-cron.sh` with `--no-deliver` and automated nightly `PRAGMA wal_checkpoint(TRUNCATE)` across all agent DBs.
+  5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens.
+- **How to Avoid Recurrence**:
+  - Keep `NODE_OPTIONS="--max-old-space-size=3072"` configured in the container spec.
+  - Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases.
+  - Always register isolated automated cron jobs with `--no-deliver`.
+
+
 
 

k8s/openclaw/setup-compaction-cron.sh

@@ -72,6 +72,24 @@ def main():
                 
     print(f"Compaction complete: {compacted} compacted, {skipped} skipped.")
 
+    # 3. Nightly SQLite WAL Checkpoint across all agent databases
+    print("=== Checkpointing SQLite WAL journals across all agent databases ===")
+    import glob, sqlite3, os
+    dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
+    wal_freed = 0
+    for db in dbs:
+        wal = db + "-wal"
+        if os.path.exists(wal):
+            sz = os.path.getsize(wal)
+            try:
+                con = sqlite3.connect(db, timeout=15.0)
+                con.execute("PRAGMA wal_checkpoint(TRUNCATE);")
+                con.close()
+                wal_freed += sz
+            except Exception as e:
+                print(f"  ✗ Error checkpointing {db}: {e}")
+    print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.")
+
 if __name__ == "__main__":
     main()
 EOF
@@ -83,10 +101,11 @@ echo "Registering OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZO
 kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
   --name "Nightly Session Compaction" \
   --declaration-key "$JOB_NAME" \
-  --description "Nightly multi-agent session compaction for all active topics and main sessions" \
+  --description "Nightly multi-agent session compaction and SQLite WAL checkpointing" \
   --cron "$CRON_SCHEDULE" \
   --tz "$TIMEZONE" \
   --session "isolated" \
+  --no-deliver \
   --command "python3 /home/node/.openclaw/scripts/compact_sessions.py"
 
 echo "=== Compaction Cron automation successfully registered! ==="