diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index cecfd7b..fcc2fd1 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -20,6 +20,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. | | **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. | | **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. | +| **2026-10-02** | **Memory Cgroup OOMKill & Probe Flapping** | Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported `failed` delivery | `RESOLVED` | Injected `NODE_OPTIONS="--max-old-space-size=3072"`, raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added `--no-deliver` + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens). | --- @@ -288,5 +289,33 @@ This document captures fixed issues, operational quirks, and incident root-cause - Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502. - Review adding a secondary fallback STT provider if OpenRouter audio is unavailable. +--- + +### Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure + +- **Date**: 2026-10-02 +- **Severity**: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status) +- **Symptom**: + 1. Pod restarted at `2026-10-02T12:40:43Z` with `lastState: terminated (exitCode: 137, reason: OOMKilled)`. + 2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (`context deadline exceeded while awaiting headers`). + 3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit). + 4. The nightly session compaction cron job `nightly-session-compaction` completed with `completionStatus: "failed"` due to unrouted announcement delivery. +- **Root Cause**: + 1. **Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting**: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit `--max-old-space-size` ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL. + 2. **Aggressive Liveness Threshold on Single-Threaded Event Loop**: The liveness probe was set to `failureThreshold: 5, periodSeconds: 10` (50s tolerance). When disk I/O on the physical spinning HDD (`/dev/sda2`) or memory sync tasks delayed the Node.js event loop by 46 seconds (`overdue=16192ms elapsed=46192ms`), Kubelet killed the container. + 3. **Unbound Cron Delivery**: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or `--no-deliver`, OpenClaw fails delivery closed (`Refusing implicit isolated cron delivery: delivery.to is required`). + 4. **Uncheckpointed SQLite WAL Journals & Dead Sessions**: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens. +- **Resolution**: + 1. Injected `NODE_OPTIONS="--max-old-space-size=3072"` into `k8s/openclaw/openclaw-instance.secret.yaml` to enforce V8 garbage collection when heap reaches 3 GB. + 2. Increased liveness probe tolerance to `failureThreshold: 12, timeoutSeconds: 15` (120s tolerance). + 3. Lowered native safeguard compaction threshold `maxActiveTranscriptBytes` from `3mb` to `400kb` (~50k tokens). + 4. Updated `k8s/openclaw/setup-compaction-cron.sh` with `--no-deliver` and automated nightly `PRAGMA wal_checkpoint(TRUNCATE)` across all agent DBs. + 5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens. +- **How to Avoid Recurrence**: + - Keep `NODE_OPTIONS="--max-old-space-size=3072"` configured in the container spec. + - Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases. + - Always register isolated automated cron jobs with `--no-deliver`. + + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index 671c426..ffa8e01 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/k8s/openclaw/setup-compaction-cron.sh b/k8s/openclaw/setup-compaction-cron.sh index 98020ad..d42fa80 100755 --- a/k8s/openclaw/setup-compaction-cron.sh +++ b/k8s/openclaw/setup-compaction-cron.sh @@ -72,6 +72,24 @@ def main(): print(f"Compaction complete: {compacted} compacted, {skipped} skipped.") + # 3. Nightly SQLite WAL Checkpoint across all agent databases + print("=== Checkpointing SQLite WAL journals across all agent databases ===") + import glob, sqlite3, os + dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite") + wal_freed = 0 + for db in dbs: + wal = db + "-wal" + if os.path.exists(wal): + sz = os.path.getsize(wal) + try: + con = sqlite3.connect(db, timeout=15.0) + con.execute("PRAGMA wal_checkpoint(TRUNCATE);") + con.close() + wal_freed += sz + except Exception as e: + print(f" ✗ Error checkpointing {db}: {e}") + print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.") + if __name__ == "__main__": main() EOF @@ -83,10 +101,11 @@ echo "Registering OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZO kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \ --name "Nightly Session Compaction" \ --declaration-key "$JOB_NAME" \ - --description "Nightly multi-agent session compaction for all active topics and main sessions" \ + --description "Nightly multi-agent session compaction and SQLite WAL checkpointing" \ --cron "$CRON_SCHEDULE" \ --tz "$TIMEZONE" \ --session "isolated" \ + --no-deliver \ --command "python3 /home/node/.openclaw/scripts/compact_sessions.py" echo "=== Compaction Cron automation successfully registered! ==="