fix(openclaw): harden 4Gi memory ceiling, increase liveness tolerance, add WAL checkpointing and cron delivery fix
This commit is contained in:
Moritz Graf
@@ -20,6 +20,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
|
||||
| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
|
||||
| **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. |
|
||||
| **2026-10-02** | **Memory Cgroup OOMKill & Probe Flapping** | Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported `failed` delivery | `RESOLVED` | Injected `NODE_OPTIONS="--max-old-space-size=3072"`, raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added `--no-deliver` + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens). |
|
||||
|
||||
---
|
||||
|
||||
@@ -288,5 +289,33 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
|
||||
- Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.
|
||||
|
||||
---
|
||||
|
||||
### Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure
|
||||
|
||||
- **Date**: 2026-10-02
|
||||
- **Severity**: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status)
|
||||
- **Symptom**:
|
||||
1. Pod restarted at `2026-10-02T12:40:43Z` with `lastState: terminated (exitCode: 137, reason: OOMKilled)`.
|
||||
2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (`context deadline exceeded while awaiting headers`).
|
||||
3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit).
|
||||
4. The nightly session compaction cron job `nightly-session-compaction` completed with `completionStatus: "failed"` due to unrouted announcement delivery.
|
||||
- **Root Cause**:
|
||||
1. **Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting**: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit `--max-old-space-size` ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL.
|
||||
2. **Aggressive Liveness Threshold on Single-Threaded Event Loop**: The liveness probe was set to `failureThreshold: 5, periodSeconds: 10` (50s tolerance). When disk I/O on the physical spinning HDD (`/dev/sda2`) or memory sync tasks delayed the Node.js event loop by 46 seconds (`overdue=16192ms elapsed=46192ms`), Kubelet killed the container.
|
||||
3. **Unbound Cron Delivery**: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or `--no-deliver`, OpenClaw fails delivery closed (`Refusing implicit isolated cron delivery: delivery.to is required`).
|
||||
4. **Uncheckpointed SQLite WAL Journals & Dead Sessions**: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens.
|
||||
- **Resolution**:
|
||||
1. Injected `NODE_OPTIONS="--max-old-space-size=3072"` into `k8s/openclaw/openclaw-instance.secret.yaml` to enforce V8 garbage collection when heap reaches 3 GB.
|
||||
2. Increased liveness probe tolerance to `failureThreshold: 12, timeoutSeconds: 15` (120s tolerance).
|
||||
3. Lowered native safeguard compaction threshold `maxActiveTranscriptBytes` from `3mb` to `400kb` (~50k tokens).
|
||||
4. Updated `k8s/openclaw/setup-compaction-cron.sh` with `--no-deliver` and automated nightly `PRAGMA wal_checkpoint(TRUNCATE)` across all agent DBs.
|
||||
5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens.
|
||||
- **How to Avoid Recurrence**:
|
||||
- Keep `NODE_OPTIONS="--max-old-space-size=3072"` configured in the container spec.
|
||||
- Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases.
|
||||
- Always register isolated automated cron jobs with `--no-deliver`.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
@@ -72,6 +72,24 @@ def main():
|
||||
|
||||
print(f"Compaction complete: {compacted} compacted, {skipped} skipped.")
|
||||
|
||||
# 3. Nightly SQLite WAL Checkpoint across all agent databases
|
||||
print("=== Checkpointing SQLite WAL journals across all agent databases ===")
|
||||
import glob, sqlite3, os
|
||||
dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
|
||||
wal_freed = 0
|
||||
for db in dbs:
|
||||
wal = db + "-wal"
|
||||
if os.path.exists(wal):
|
||||
sz = os.path.getsize(wal)
|
||||
try:
|
||||
con = sqlite3.connect(db, timeout=15.0)
|
||||
con.execute("PRAGMA wal_checkpoint(TRUNCATE);")
|
||||
con.close()
|
||||
wal_freed += sz
|
||||
except Exception as e:
|
||||
print(f" ✗ Error checkpointing {db}: {e}")
|
||||
print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
EOF
|
||||
@@ -83,10 +101,11 @@ echo "Registering OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZO
|
||||
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
|
||||
--name "Nightly Session Compaction" \
|
||||
--declaration-key "$JOB_NAME" \
|
||||
--description "Nightly multi-agent session compaction for all active topics and main sessions" \
|
||||
--description "Nightly multi-agent session compaction and SQLite WAL checkpointing" \
|
||||
--cron "$CRON_SCHEDULE" \
|
||||
--tz "$TIMEZONE" \
|
||||
--session "isolated" \
|
||||
--no-deliver \
|
||||
--command "python3 /home/node/.openclaw/scripts/compact_sessions.py"
|
||||
|
||||
echo "=== Compaction Cron automation successfully registered! ==="
|
||||
|
||||
Reference in New Issue
Block a user