moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
1ffbf3d6b77988d099afacecc7dc1a54e8febba2
infrapuzzle/k8s/openclaw/TROUBLESHOOTING.md
T

40 KiB
Raw Blame History

OpenClaw Troubleshooting & Incident Playbook

This document captures fixed issues, operational quirks, and incident root-cause analyses for the OpenClaw deployment on Kubernetes.

Important

Rule for Agents & Operators: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, document it here immediately. This prevents recurring regressions and serves as the single source of truth for troubleshooting.

📋 Catalog of Incidents & Fixed Issues

Date Category Summary / Symptom Status Key Resolution
2026-09-20 Timeout & Execution Telegram #IT Guy (Topic 119) and #Recherche (Topic 117) request timed out before response generated (timeoutSeconds exceeded) RESOLVED Raised global agents.defaults.timeoutSeconds from 600 to 1800 (30 min).
2026-09-20 Config Schema Validation Pod crash with CrashLoopBackOff: agents.entries.<id>: Unrecognized key: "timeoutSeconds" RESOLVED timeoutSeconds is strictly global under agents.defaults.timeoutSeconds; do not place it under agents.entries.*.
2026-09-24 Database & Startup Latency Gateway crash/restart during initialization: slow OpenClaw agent database open leading to Startup/Liveness probe failures RESOLVED Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases.
2026-09-28 Security & Workspace Files Subagents rejected bootstrap files on every turn: bootstrap file is unreadable: ... reason=path must not be hardlinked RESOLVED Decoupled USER.md and MEMORY.md into standalone copies (nlink == 1) across all subagent workspaces via cp --remove-destination.
2026-09-28 Cron Trigger Scope Website watch cron job failed: cron trigger evaluation failed (internal_error): ReferenceError: state is not defined RESOLVED Guarded against undeclared state in JavaScript trigger sandbox via typeof state !== 'undefined' && state.
2026-09-28 Token Saturation & Model Degradation Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with Sleep-Hax0r) and Node.js event loop delayed 31s under 2GB RSS RESOLVED Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via setup-compaction-cron.sh.
2026-09-29 Hardlink Regression & Disk I/O Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in Dl state opening agent SQLite DBs RESOLVED Replaced recurring ln -f in cronjob-wiki-backup.secret.yaml with cp --remove-destination, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring 5/5 Running.
2026-09-29 Compaction Pitfall & Split-Turn Drift Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German RESOLVED Identified --max-lines 250 bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery.
2026-09-29 Audio & Speech-to-Text Telegram voice notes failed transcription with reason=ProviderHttpError, passing empty text (0 chars) and inducing bot hallucinations RESOLVED Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash.
2026-10-02 Memory Cgroup OOMKill & Probe Flapping Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported failed delivery RESOLVED Injected NODE_OPTIONS="--max-old-space-size=3072", raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added --no-deliver + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens).
2026-10-05 Disk I/O & Startup Latency Event loop starvation (48s freeze) and container restarts from MCP process storms & cold-start database integrity scans RESOLVED Implemented init-container page-cache pre-warming (cat ... > /dev/null) cutting DB open from 22 minutes to 11 seconds.
2026-10-05 CronJob & Alerting openclaw-wiki-audit CronJob failing daily with exit code 1; Prometheus alert WikiAuditJobFailed firing RESOLVED Appended || true to okf_lint.py in cronjob-wiki-audit.secret.yaml, preventing set -e abort when lint findings exist.
2026-10-05 Cron Trigger State rennerhof-website-watch (Regensburg Topic 116) stuck in status: error with ReferenceError: state is not defined RESOLVED Cleared stale error state from state_json in openclaw.sqlite and triggered successful verification run.
2026-10-05 State Deadlock & Session Corruption Telegram Topic 117 (#Recherche) wedged with restart recovery claim changed and entry_valid = 0 after crash RESOLVED Repaired session_nodes and session_windows (restored entry_valid = 1, synced updated_at, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added --max-lines 200 compaction fallback.
2026-10-06 Message Formatting & UX Subagents emitted raw HTML/XML comments (<!-- DOMAIN_STATE: { ... } -->) at the end of Telegram messages RESOLVED Removed obsolete DOMAIN_STATE State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails.

🔍 Detailed Incident Records

Issue 1: Telegram Agent Request Timeout (agents.defaults.timeoutSeconds)

  • Date: 2026-09-20
  • Severity: High (User-facing timeout in Telegram topics #IT Guy and #Recherche)
  • Symptom: Telegram replies with error message: 
    Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
  • Root Cause: The IT Guy agent (itguy) performs deep, sequential DevOps investigations (multiple exec / kubectl calls, file writes, and PR operations) using deepseek/deepseek-v4.1-flash. Similarly, #Recherche runs deep research chains. The global default was set to timeoutSeconds: 600 (10 minutes), causing active, non-stuck runs to be killed mid-flight.
  • Resolution: In k8s/openclaw/openclaw-instance.secret.yaml, configure: 
    spec:
  config:
    raw:
      agents:
        defaults:
          timeoutSeconds: 1800  # 30 minutes
  • How to Avoid Recurrence: Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.

Issue 2: Schema Crash on Per-Agent Timeout Override

  • Date: 2026-09-20
  • Severity: Critical (Deployment blocked with CrashLoopBackOff)
  • Symptom: Gateway container fails during startup: 
    Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
  • Root Cause: OpenClaw’s JSON schema strict validator does not permit timeoutSeconds inside individual agent entries under agents.entries.<name>.
  • Resolution: Remove timeoutSeconds from agents.entries.*. Configure it strictly at the root of agents.defaults.timeoutSeconds.
  • How to Avoid Recurrence: Before adding new keys to agent configurations in openclaw-instance.secret.yaml, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.

Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization

  • Date: 2026-09-24
  • Severity: High (Intermittent container kills on gateway restart)
  • Symptom: Container openclaw fails startup or liveness probes, causing kubelet to restart the container: 
    Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
Startup probe failed: context deadline exceeded / connection reset by peer
    Logs display repeated warnings: 
    [state/agent-db] slow OpenClaw agent database open
  • Root Cause: OpenClaw maintains separate SQLite databases for each specialist subagent (agents/*/agent/openclaw-agent.sqlite). Over time, databases and uncommitted WAL logs accumulate significant size:
    • agents/main/agent/openclaw-agent.sqlite: >220 MB
    • agents/fitness/, agents/itguy/, agents/hebamme/, agents/recherche/: >100 MB each + multi-megabyte WAL files. On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the startupProbe limit (failureThreshold * periodSeconds), Kubernetes kills the container before it finishes opening databases.
  • Resolution:
    1. Increase probes.startup.failureThreshold in openclaw-instance.secret.yaml to 240 (with 5s period = 1200s / 20 min).
    2. Periodically run PRAGMA wal_checkpoint(TRUNCATE) across all SQLite databases to compact WAL journals into main files.
  • How to Avoid Recurrence: Monitor SQLite database sizes under /home/openclaw/.openclaw/agents/*/agent/ and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.

Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary

  • Date: 2026-09-28
  • Severity: High (Subagents lose access to user profile and memory context)
  • Symptom: On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports: 
    [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/USER.md reason=path must not be hardlinked
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/MEMORY.md reason=path must not be hardlinked
  • Root Cause: OpenClaw’s embedded execution harness enforces a strict filesystem security check: stat.nlink > 1 triggers path must not be hardlinked. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files.
  • Resolution: Decouple all hardlinked files so that each workspace has its own standalone copy with nlink == 1: 
    for dir in /home/node/.openclaw/workspace-*; do
  for file in USER.md MEMORY.md; do
    cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file"
  done
done
    Ensure the init container in openclaw-instance.secret.yaml maintains this pattern with cp --remove-destination.
  • How to Avoid Recurrence: Never create hardlinks (ln without -s) inside OpenClaw workspace directories. Always use standalone file copies (cp --remove-destination).

Issue 5: Cron Trigger Sandbox Scope ReferenceError

  • Date: 2026-09-28
  • Severity: Medium (Automated recurring cron job fails to evaluate)
  • Symptom: In cron_run_receipts, the job rennerhof-website-watch failed with: 
    cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
    at <anonymous> (openclaw-code-mode:user.js:12:172)
  • Root Cause: The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access state && state.hash directly, causing a ReferenceError when state is undeclared in the global scope.
  • Resolution: Use typeof state !== "undefined" to safely guard against undeclared variables in the trigger script: 
    const s = (typeof state !== 'undefined' && state) ? state : {};
const prev = s.hash;
  • How to Avoid Recurrence: In OpenClaw trigger scripts, never evaluate bare identifier state directly without a typeof guard.

Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)

  • Date: 2026-09-28
  • Severity: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
  • Symptom:
    1. The #Mila (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang: 
      Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
    2. OpenClaw pod logs showed critical memory pressure (rss=1.98 GiB, threshold 1.64 GiB) and a massive Node.js event loop delay of 31.4 seconds (eventLoopDelayP99Ms=31406).
  • Root Cause:
    1. Uncompacted Topic Sessions: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached 753 messages and 127,607 tokens; IT Guy reached 204,670 tokens; Regensburg reached 160,029 tokens; Fitness reached 135,990 tokens.
    2. Model Context Saturation in openrouter/z-ai/glm-5.3-flash: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row 1062, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
    3. Compaction Gap: The existing nightly-session-compaction cron job only targeted "agent:main:main", neglecting all specialist subagent forum topic sessions. Furthermore, compaction.maxActiveTranscriptBytes: "3mb" in config requires ~700k raw text tokens to trigger natively.
  • Resolution:
    1. Immediately ran openclaw sessions compact across all bloated subagent topic sessions, bringing Hebamme from 127,607 down to 11,892 tokens and IT Guy from 204,670 to 250 lines.
    2. Created /home/node/.openclaw/scripts/compact_sessions.py on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
    3. Updated the OpenClaw Cron automation nightly-session-compaction to execute this script every night at 03:30 Europe/Berlin (30 3 * * *).
    4. Created k8s/openclaw/setup-compaction-cron.sh in git as the reproducible source of truth for the automation.
  • How to Avoid Recurrence: Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.

Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck

  • Date: 2026-09-29
  • Severity: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in Dl state)
  • Symptom:
    1. Telegram bot stopped answering messages entirely on the morning of Sep 29.
    2. openclaw-0 pod was killed by kubelet with exit code 137 (Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded).
    3. Upon restarting, the container remained in 4/5 Running for over 15 minutes, with the gateway process (openclaw-gateway, PID 15930) stuck in Linux uninterruptible sleep state Dl (generic_file_buffered_read) and logging repeated: 
      [state/agent-db] slow OpenClaw agent database open
  • Root Cause:
    1. Recurring Hardlink Regression: The nightly openclaw-wiki-backup CronJob runs daily at 03:30 UTC. Lines 41-42 in cronjob-wiki-backup.secret.yaml contained: 
      [ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true
[ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true
      This routinely re-created hardlinks (nlink == 11) across all subagent workspaces every night, directly undoing the nlink == 1 fix from Issue 4 and causing OpenClaw to reject USER.md and MEMORY.md. When a user message arrived in #Mila at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137).
    2. Spinning Disk Random I/O Bottleneck on Startup: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs verifyAndRepairCanonicalSqliteIndexes on each database. On rotational SATA hard drives (/dev/sda2, rotational=1), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold.
  • Resolution:
    1. Permanent Manifest Fix: Patched k8s/openclaw/cronjob-wiki-backup.secret.yaml to replace ln -f with cp --remove-destination, ensuring subagent workspace bootstrap files remain strict standalone files (nlink == 1). Applied the updated manifest to the cluster and pushed commit 74e32ed to git.
    2. Volume Decoupling: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies.
    3. Page Cache Pre-warming: Sequentially streamed each agent SQLite database into the Linux RAM page cache (cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent.
    4. The gateway successfully bound :18789, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy 5/5 Running state.
  • How to Avoid Recurrence:
    • Never use ln (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use cp --remove-destination.
    • When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (PRAGMA wal_checkpoint(TRUNCATE)), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.

Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction --max-lines Pitfall

  • Date: 2026-09-29
  • Severity: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German)
  • Symptom:
    1. A user sent a voice message to the #Einkaufen (Shopping) topic asking for recommendations for a kitchen scale (Küchenwaage).
    2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German: 
      Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel...
    3. The user was concerned whether the system had been compromised or hacked.
  • Root Cause:
    1. The --max-lines Compaction Flaw: The nightly compaction cron script (setup-compaction-cron.sh / compact_sessions.py) invoked: 
      openclaw sessions compact <key> --agent <agent> --max-lines 250
      According to the OpenClaw CLI specification, --max-lines <count> means: Truncate to the last N transcript lines instead of LLM summarization. If a session has \le 250 transcript lines, OpenClaw outputs No compaction needed for session ... and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached 38,000 to 95,000 tokens while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (shopping, recherche, grabengasse_media, etc.).
    2. Split-Turn Context Anchoring: When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a Turn Context (split turn) block indicating unfinished work: 
      Turn Context (split turn):
### Original Request
User korrigierte die Supermarkt-Liste der Wiki-Seite...
### Next Steps
1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen...
3. Änderung gegenüber Nutzer bestätigen.
      When the new voice note arrived at seq=231, the prompt presented to z-ai/glm-5.3-flash was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (seq=232) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding: 
      "No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction."
      It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German).
  • Resolution:
    1. Remove --max-lines for True LLM Compaction: Updated setup-compaction-cron.sh and /home/node/.openclaw/scripts/compact_sessions.py to remove --max-lines, invoking openclaw sessions compact <key> --agent <agent> so OpenClaw performs native LLM summarization.
    2. Compacted Live Sessions: Ran LLM compaction on shopping:topic:115, successfully shrinking it from 39,283 tokens down to 5,343 tokens (an 86% context reclamation).
    3. Event Loop Pacing: Added time.sleep(2) between consecutive session compactions in compact_sessions.py to allow Node.js event loop ticks and prevent Kubelet /healthz liveness probe timeouts during batch compactions.
  • How to Avoid Recurrence:
    • Never use --max-lines when the objective is semantic context reduction; omit --max-lines to let OpenClaw invoke LLM summarization.
    • Monitor session token sizes (openclaw sessions --all-agents --json) rather than line counts.
    • Keep setup-compaction-cron.sh aligned across git and live PVCs.

Issue 9: Speech-to-Text / Audio Transcription Failure (ProviderHttpError) & Empty Voice Note Transcripts

  • Date: 2026-09-29
  • Severity: High (Telegram voice notes fail transcription, passing empty messages to agents and prompting hallucinated/frantic behavior)
  • Symptom:
    1. Telegram voice notes fail transcription with: 
      [media-understanding] audio: failed (0/1) reason=ProviderHttpError
      in ~300–500ms.
    2. The inbound Telegram message is delivered to the agent with 0 chars text ((group, audio/ogg, 0 chars)).
    3. The agent receives an empty prompt or placeholder ([Audio attachment could not be analyzed]). It does not know what the user requested, resulting in:
      • No reply,
      • Completely hallucinated or unrelated replies, or
      • Frantic autonomous attempts by the agent to find and transcribe the audio file in /home/node/.openclaw/media/inbound/ via terminal commands (ffmpeg, python3, uv, free model APIs), producing errors and strange messages that give the appearance the bot was "hacked".
  • Root Cause:
    1. OpenRouter Audio Balance Floor ($0.50): OpenRouter requires at least $0.50 in credit buffer / remaining key limit for any audio transcription request (openai/whisper-large-v3-turbo or multimodal audio). When the monthly limit was set to $70.00 and usage reached ~$59.80 (leaving <$0.50 of head room under the prior limit), OpenRouter rejected audio requests with HTTP 402: 
      {"error":{"message":"This request requires at least $0.50 in balance for audio","code":402,"metadata":{"limit_source":"openrouter_key_limit"}}}
    2. Chat Completions Succeeded while Audio Failed: Because text completions cost only fractions of a cent ($0.0001–$0.001), regular chat queries continued to work normally. Only audio transcriptions were blocked by the $0.50 threshold.
    3. Error Truncation in OpenClaw: In runner.entries-LWAceiTW.js, the error was processed by summarizeDecisionReason(reason), which executes normalizeDecisionReason(reason).split(":")[0]. Because ProviderHttpError was not stripped by the regex, the error string ProviderHttpError: OpenRouter audio transcription failed (HTTP 402)... was truncated to ProviderHttpError. This obscured the HTTP 402 status code and key limit message in normal logs.
    4. Single Provider without Fallback: openclaw-instance.secret.yaml only declared a single audio provider (openrouter/openai/whisper-large-v3-turbo) with no fallback provider configured. When OpenRouter returned 402, OpenClaw marked the audio as failed and fell back to empty input.
  • Resolution:
    1. Buffer Headroom Verified: Confirmed OpenRouter key monthly limit has ample remaining buffer ($10.19 remaining), and re-tested live audio transcription of the exact failed 52-second Telegram voice files (b9e0da37... and aef0424f...), confirming fast, accurate transcription (HTTP 200).
    2. Recherche Agent Upgraded: Switched recherche agent from openrouter/z-ai/glm-5.3-flash to openrouter/deepseek/deepseek-v4.1-flash in openclaw-instance.secret.yaml and live openclaw.json to ensure robust analytical reasoning and prevent prompt confusion.
  • How to Avoid Recurrence:
    • Maintain a sufficient credit buffer on OpenRouter (at least $5–$10 remaining monthly headroom).
    • Interpret reason=ProviderHttpError on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
    • Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.

Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure

  • Date: 2026-10-02
  • Severity: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status)
  • Symptom:
    1. Pod restarted at 2026-10-02T12:40:43Z with lastState: terminated (exitCode: 137, reason: OOMKilled).
    2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (context deadline exceeded while awaiting headers).
    3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit).
    4. The nightly session compaction cron job nightly-session-compaction completed with completionStatus: "failed" due to unrouted announcement delivery.
  • Root Cause:
    1. Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit --max-old-space-size ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL.
    2. Aggressive Liveness Threshold on Single-Threaded Event Loop: The liveness probe was set to failureThreshold: 5, periodSeconds: 10 (50s tolerance). When disk I/O on the physical spinning HDD (/dev/sda2) or memory sync tasks delayed the Node.js event loop by 46 seconds (overdue=16192ms elapsed=46192ms), Kubelet killed the container.
    3. Unbound Cron Delivery: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or --no-deliver, OpenClaw fails delivery closed (Refusing implicit isolated cron delivery: delivery.to is required).
    4. Uncheckpointed SQLite WAL Journals & Dead Sessions: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens.
  • Resolution:
    1. Injected NODE_OPTIONS="--max-old-space-size=3072" into k8s/openclaw/openclaw-instance.secret.yaml to enforce V8 garbage collection when heap reaches 3 GB.
    2. Increased liveness probe tolerance to failureThreshold: 12, timeoutSeconds: 15 (120s tolerance).
    3. Lowered native safeguard compaction threshold maxActiveTranscriptBytes from 3mb to 400kb (~50k tokens).
    4. Updated k8s/openclaw/setup-compaction-cron.sh with --no-deliver and automated nightly PRAGMA wal_checkpoint(TRUNCATE) across all agent DBs.
    5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens.
  • How to Avoid Recurrence:
    • Keep NODE_OPTIONS="--max-old-space-size=3072" configured in the container spec.
    • Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases.
    • Always register isolated automated cron jobs with --no-deliver.

Issue 11: Event Loop Starvation and Probe Timeouts from MCP Process Storms & Cold-Start SQLite Verification on HDD Storage

  • Date: 2026-10-05
  • Severity: Critical (Pod flapping with startup/liveness probe timeouts, event loop frozen for 48s–70s)
  • Symptom:
    1. OpenClaw container restarted frequently; logs recorded massive timing gaps: 
      [health] host timing gap detected: process was frozen ~48830ms (clock jump: 48830ms, interval: 10000ms)
system-prompt:64391ms
[state/agent-db] slow OpenClaw agent database open
    2. Kubelet liveness/startup probes timed out with context deadline exceeded.
    3. Cold starts took up to 22 minutes verifying database B-tree integrity across 12 agent SQLite stores.
  • Root Cause:
    1. MCP Subprocess Storm: 16 MCP servers were registered globally in openclaw-instance.secret.yaml. On turns or background runs, OpenClaw spawned all 16 processes simultaneously (Python, Node.js, and Go). Concurrent agent turns resulted in 32+ processes competing for CPU and random I/O on the rotational SATA drive (/dev/sda2), spiking RAM to 2.43 GB and starving the Node.js event loop.
    2. Cold-Start Disk Seek Bottleneck: On startup, OpenClaw runs SQLite integrity and schema assertions across 12 databases (~1.8 GB). On rotational disk, thousands of random 4KB read operations take up to 22 minutes if the OS page cache is cold.
    3. ext4 Directory Inode Bloat: /home/node/.openclaw directory inode had grown to 17.8 MB from historical file deletions. ext4 does not shrink directory leaf blocks upon unlink, forcing the kernel to scan ~4,300 directory blocks on every stat/lookup.
  • Resolution:
    1. Implemented sequential page-cache pre-warming in the pod initContainer and nightly cron script: 
      cat /home/node/.openclaw/state/openclaw.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true
      Because sequential read throughput on the HDD is ~150 MB/s, the kernel streams all 1.8 GB into RAM in ~11 seconds. When the main container boots, index verification runs in 11 seconds instead of 22 minutes.
    2. Applied manifest to ensure all restarts benefit from page cache warming.
  • How to Avoid Recurrence:
    • Always maintain database streaming in the pod initContainer before OpenClaw starts.
    • Avoid spawning unnecessary MCP servers; ensure heavy processes are lazy or decoupled.

Issue 12: Knowledge Wiki Audit CronJob Failure Loop (set -e vs Lint Findings) and Prometheus Alert (WikiAuditJobFailed)

  • Date: 2026-10-05
  • Severity: High (Daily audit CronJob failed repeatedly; Prometheus alert firing; git repo left uncommitted)
  • Symptom:
    1. Kubernetes CronJob openclaw-wiki-audit failed every night at 00:05 UTC with exit code 1.
    2. Prometheus fired persistent alert WikiAuditJobFailed.
    3. Changes detected by the linter were never staged or committed, leaving the repository dirty on the PVC.
  • Root Cause: In k8s/openclaw/cronjob-wiki-audit.secret.yaml, the script had set -euo pipefail at line 25. Line 47 executed: 
    python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks
    okf_lint.py deliberately exits with code 1 whenever any lint findings or actionable tasks are added to Obsidian notes. Because of set -e, the shell script aborted immediately at line 47, skipping git commit, git push, and exiting 1.
  • Resolution: Patched line 47 to append || true: 
    python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks || true
    Applied manifest, deleted the stale failed job, and triggered a test job. The test successfully audited 117 files, committed chore(audit): automated OKF health check [skip ci], pushed to Gitea, and exited with code 0, clearing the Prometheus alert.
  • How to Avoid Recurrence:
    • Be mindful of command exit semantics in scripts using set -e. Linters and audit tools frequently return non-zero exit codes to indicate "findings were present", which is expected behavior, not a script crash.

Issue 13: Stale Cron Job Error State on rennerhof-website-watch (Regensburg Topic 116)

  • Date: 2026-10-05
  • Severity: Medium (Website watch cron job showed red error status in OpenClaw dashboard)
  • Symptom: The cron automation rennerhof-website-watch for Regensburg Topic 116 showed status: error with: 
    cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
  • Root Cause: Although the trigger code was previously patched, the historical error was preserved in state_json in /home/node/.openclaw/state/openclaw.sqlite, causing OpenClaw to report the job as failed.
  • Resolution: Cleared the error state from state_json in SQLite and executed a manual trigger evaluation: 
    "fire": false, "evals": 12, "last_result": null, "status": "running"
    Job state returned to healthy running with zero errors.
  • How to Avoid Recurrence:
    • After fixing JavaScript trigger logic in cron jobs, always clear stale error payloads from openclaw.sqlite or trigger an explicit run to update state.

Issue 14: Ungraceful Restart Deadlock on Telegram Topic 117 (#Recherche): entry_valid = 0 and Recovery Claim Deadlock

  • Date: 2026-10-05
  • Severity: Critical (Topic 117 completely unresponsive in Telegram; every message failed with error and retried endlessly)
  • Symptom:
    1. Inbound messages to Telegram Topic 117 failed with: 
      [telegram] dispatch failed: Error: restart recovery claim changed before agent adoption
[telegram] spooled update 985533189 failed; keeping for retry: restart recovery claim changed before agent adoption
    2. Running any CLI command against the recherche agent failed with: 
      SessionCanonicalKeyMigrationRequiredError: invalid persisted session row requires repair for agent:recherche:telegram:group:-1004326921510:topic:117; stop the Gateway and run openclaw doctor --fix
  • Root Cause:
    1. Active Writer Abort: When the pod was killed or restarted during an active message turn, session_nodes and entry_json were left with status: "running", activeWriterRunId, and restartRecoveryDeliveryRunId.
    2. Admission Deadlock: In reply-admission-ticket.js:469, when new messages arrive while activeClaimRunId is set and entry.status === "running", the runtime throws Error: restart recovery claim changed before agent adoption. Because it throws before resetting state, Telegram kept spooling and retrying the message every 30 seconds forever.
    3. Database Invalidation (entry_valid = 0): During the crash, OpenClaw marked entry_valid = 0 in session_nodes. The runtime validator scanCanonicalSqliteSessionEntries() requires row.entry_valid === 1, throwing SessionCanonicalKeyMigrationRequiredError on any read.
    4. Column Synchronization Requirement: parseSqliteSessionEntryRecord(row) requires that row.updated_at === record.updatedAt. Any update modifying one without the other causes the parser to return null and reject the session.
    5. Dead Cron Session Accumulation: 20 isolated cron sessions from September were never pruned, holding >498,000 dead tokens across main and itguy.
  • Resolution:
    1. Synchronized SQLite state in /home/node/.openclaw/agents/recherche/agent/openclaw-agent.sqlite:
      • Set entry_valid = 1.
      • Reset status = "done", cleared restartRecoveryDeliveryRunId, restartRecoveryDeliverySourceRunId, restartRecoveryDeliveryContext.
      • Synchronized updated_at on row and updatedAt in entry_json.
      • Updated session_windows to status = "done".
    2. Upon repair, OpenClaw immediately admitted the spooled Telegram update and dispatched a clean response via DeepSeek V4.1 Flash.
    3. Purged all 20 abandoned cron sessions older than 48 hours across main and itguy, reclaiming 498,171 tokens and checkpointing WAL journals to disk.
    4. Upgraded setup-compaction-cron.sh:
      • Added automatic pre-warming of SQLite page cache.
      • Added automated pruning of isolated cron sessions older than 48 hours.
      • Raised compaction RPC timeout from 10s to 120s (--timeout 120000).
      • Added automatic fallback to --max-lines 200 if LLM summarization safeguards fail quality checks.
  • How to Avoid Recurrence:
    • The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache.
    • If a session ever reports invalid persisted session row requires repair, check entry_valid == 1 and ensure row.updated_at == record.updatedAt.

Issue 15: Subagents Emitted Raw HTML/XML Comments (<!-- DOMAIN_STATE: { ... } -->) in Telegram Messages

  • Date: 2026-10-06
  • Severity: Medium (Visual/UX defect: raw HTML comments and JSON leaked into user-facing chat messages)
  • Symptom: Messages sent by specialist subagents in Telegram forum topics (#Mila, #Medizin, #Shopping, #Travel, #Sport, #Regensburg, etc.) consistently concluded with a raw HTML/XML comment block, for example: 
    <!-- DOMAIN_STATE: {
  "agent": "doktor",
  "topic_category": "symptom_check",
  "evidence_source": "AWMF",
  "triage_urgency": "routine"
} -->
  • Root Cause:
    1. Prompt Directive: In knowledge-wiki/agents/*.md, each subagent prompt contained a ## 🔄 State Bridge Schema instructing the LLM: „At the end of your response, output a compact state comment: <!-- DOMAIN_STATE: { ... } -->“.
    2. Telegram Formatting Mismatch: The schema was originally conceived in ADR 0001 under the assumption that <!-- ... --> would be invisible in chat rendering. However, Telegram's Bot API does not support hidden HTML comments. OpenClaw's text formatter safely escaped them to &lt;!-- ... --&gt;, causing the Telegram client to render the comments literally as visible text.
    3. Architectural Redundancy: Subagents operate inside persistent SQLite sessions with full conversation history and vector embeddings. Furthermore, structured domain states (sleep, nutrition, feeding, workouts) were transitioned to DuckDB/JSONL via skills/datastore (ADR 0004). No system component or code read DOMAIN_STATE.
  • Resolution:
    1. Removed ## 🔄 State Bridge Schema from all 9 subagent definition files in knowledge-wiki/agents/*.md (hebamme, doktor, shopping, travel, fitness, regensburg, recherche, itguy, grabengasse_media, and README.md).
    2. Replaced the directive with an explicit clean-output guardrail: 
      ## 💬 Output Formatting & Telegram Guardrails
- Output clean Markdown only for Telegram.
- NEVER append machine-readable metadata, HTML comments (`<!-- ... -->`), XML tags, or raw JSON payloads to your chat responses.
    3. Removed the Pass State Bridge (<!-- DOMAIN_STATE -->) clause from the Conductor system prompt in k8s/openclaw/openclaw-instance.secret.yaml.
    4. Pushed updates to knowledge-wiki git repository, pulled latest into /home/node/.openclaw/wiki/main, and synchronized live workspace files in /home/node/.openclaw/workspace-*/AGENTS.md.
    5. Applied updated openclaw-instance.secret.yaml to Kubernetes.
  • How to Avoid Recurrence:
    • Never instruct LLMs to embed machine-readable metadata or pseudo-comments (<!-- ... -->) in user-facing messenger outputs.
    • Rely on persistent session context and structured stores (skills/datastore) for state retention.
Reference in New Issue View Git Blame Copy Permalink