# OpenClaw Troubleshooting & Incident Playbook This document captures fixed issues, operational quirks, and incident root-cause analyses for the **OpenClaw** deployment on Kubernetes. > [!IMPORTANT] > **Rule for Agents & Operators**: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, **document it here immediately**. This prevents recurring regressions and serves as the single source of truth for troubleshooting. --- ## 📋 Catalog of Incidents & Fixed Issues | Date | Category | Summary / Symptom | Status | Key Resolution | | :--- | :--- | :--- | :---: | :--- | | **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). | | **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. | | **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. | | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. | | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. | | **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. | --- ## 🔍 Detailed Incident Records ### Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`) - **Date**: 2026-09-20 - **Severity**: High (User-facing timeout in Telegram topics `#IT Guy` and `#Recherche`) - **Symptom**: Telegram replies with error message: ```text Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config. ``` - **Root Cause**: The IT Guy agent (`itguy`) performs deep, sequential DevOps investigations (multiple `exec` / `kubectl` calls, file writes, and PR operations) using `deepseek/deepseek-v4.1-flash`. Similarly, `#Recherche` runs deep research chains. The global default was set to `timeoutSeconds: 600` (10 minutes), causing active, non-stuck runs to be killed mid-flight. - **Resolution**: In [`k8s/openclaw/openclaw-instance.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml), configure: ```yaml spec: config: raw: agents: defaults: timeoutSeconds: 1800 # 30 minutes ``` - **How to Avoid Recurrence**: Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops. --- ### Issue 2: Schema Crash on Per-Agent Timeout Override - **Date**: 2026-09-20 - **Severity**: Critical (Deployment blocked with `CrashLoopBackOff`) - **Symptom**: Gateway container fails during startup: ```text Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json: openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds" ``` - **Root Cause**: OpenClaw’s JSON schema strict validator does not permit `timeoutSeconds` inside individual agent entries under `agents.entries.`. - **Resolution**: Remove `timeoutSeconds` from `agents.entries.*`. Configure it strictly at the root of `agents.defaults.timeoutSeconds`. - **How to Avoid Recurrence**: Before adding new keys to agent configurations in `openclaw-instance.secret.yaml`, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path. --- ### Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization - **Date**: 2026-09-24 - **Severity**: High (Intermittent container kills on gateway restart) - **Symptom**: Container `openclaw` fails startup or liveness probes, causing kubelet to restart the container: ```text Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded Startup probe failed: context deadline exceeded / connection reset by peer ``` Logs display repeated warnings: ```text [state/agent-db] slow OpenClaw agent database open ``` - **Root Cause**: OpenClaw maintains separate SQLite databases for each specialist subagent (`agents/*/agent/openclaw-agent.sqlite`). Over time, databases and uncommitted WAL logs accumulate significant size: - `agents/main/agent/openclaw-agent.sqlite`: >220 MB - `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files. On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases. - **Resolution**: 1. Increase `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` to 240 (with 5s period = 1200s / 20 min). 2. Periodically run `PRAGMA wal_checkpoint(TRUNCATE)` across all SQLite databases to compact WAL journals into main files. - **How to Avoid Recurrence**: Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization. --- ### Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary - **Date**: 2026-09-28 - **Severity**: High (Subagents lose access to user profile and memory context) - **Symptom**: On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports: ```text [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-/USER.md reason=path must not be hardlinked [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-/MEMORY.md reason=path must not be hardlinked ``` - **Root Cause**: OpenClaw’s embedded execution harness enforces a strict filesystem security check: `stat.nlink > 1` triggers `path must not be hardlinked`. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files. - **Resolution**: Decouple all hardlinked files so that each workspace has its own standalone copy with `nlink == 1`: ```bash for dir in /home/node/.openclaw/workspace-*; do for file in USER.md MEMORY.md; do cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file" done done ``` Ensure the init container in `openclaw-instance.secret.yaml` maintains this pattern with `cp --remove-destination`. - **How to Avoid Recurrence**: Never create hardlinks (`ln` without `-s`) inside OpenClaw workspace directories. Always use standalone file copies (`cp --remove-destination`). --- ### Issue 5: Cron Trigger Sandbox Scope ReferenceError - **Date**: 2026-09-28 - **Severity**: Medium (Automated recurring cron job fails to evaluate) - **Symptom**: In `cron_run_receipts`, the job `rennerhof-website-watch` failed with: ```text cron trigger evaluation failed (internal_error): ReferenceError: state is not defined at (openclaw-code-mode:user.js:12:172) ``` - **Root Cause**: The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access `state && state.hash` directly, causing a `ReferenceError` when `state` is undeclared in the global scope. - **Resolution**: Use `typeof state !== "undefined"` to safely guard against undeclared variables in the trigger script: ```javascript const s = (typeof state !== 'undefined' && state) ? state : {}; const prev = s.hash; ``` - **How to Avoid Recurrence**: In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard. --- ### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens) - **Date**: 2026-09-28 - **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB) - **Symptom**: 1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang: ```text Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal). Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉 ``` 2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`). - **Root Cause**: 1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**. 2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands. 3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively. - **Resolution**: 1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**. 2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens. 3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`). 4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation. - **How to Avoid Recurrence**: Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.