moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
bdcf6a8d187b452498012dbc7004d9ad4e7dc193
infrapuzzle/k8s/openclaw/TROUBLESHOOTING.md
T

12 KiB
Raw Blame History

OpenClaw Troubleshooting & Incident Playbook

This document captures fixed issues, operational quirks, and incident root-cause analyses for the OpenClaw deployment on Kubernetes.

Important

Rule for Agents & Operators: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, document it here immediately. This prevents recurring regressions and serves as the single source of truth for troubleshooting.

📋 Catalog of Incidents & Fixed Issues

Date Category Summary / Symptom Status Key Resolution
2026-09-20 Timeout & Execution Telegram #IT Guy (Topic 119) and #Recherche (Topic 117) request timed out before response generated (timeoutSeconds exceeded) RESOLVED Raised global agents.defaults.timeoutSeconds from 600 to 1800 (30 min).
2026-09-20 Config Schema Validation Pod crash with CrashLoopBackOff: agents.entries.<id>: Unrecognized key: "timeoutSeconds" RESOLVED timeoutSeconds is strictly global under agents.defaults.timeoutSeconds; do not place it under agents.entries.*.
2026-09-24 Database & Startup Latency Gateway crash/restart during initialization: slow OpenClaw agent database open leading to Startup/Liveness probe failures RESOLVED Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases.
2026-09-28 Security & Workspace Files Subagents rejected bootstrap files on every turn: bootstrap file is unreadable: ... reason=path must not be hardlinked RESOLVED Decoupled USER.md and MEMORY.md into standalone copies (nlink == 1) across all subagent workspaces via cp --remove-destination.
2026-09-28 Cron Trigger Scope Website watch cron job failed: cron trigger evaluation failed (internal_error): ReferenceError: state is not defined RESOLVED Guarded against undeclared state in JavaScript trigger sandbox via typeof state !== 'undefined' && state.
2026-09-28 Token Saturation & Model Degradation Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with Sleep-Hax0r) and Node.js event loop delayed 31s under 2GB RSS RESOLVED Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via setup-compaction-cron.sh.

🔍 Detailed Incident Records

Issue 1: Telegram Agent Request Timeout (agents.defaults.timeoutSeconds)

  • Date: 2026-09-20
  • Severity: High (User-facing timeout in Telegram topics #IT Guy and #Recherche)
  • Symptom: Telegram replies with error message: 
    Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
  • Root Cause: The IT Guy agent (itguy) performs deep, sequential DevOps investigations (multiple exec / kubectl calls, file writes, and PR operations) using deepseek/deepseek-v4.1-flash. Similarly, #Recherche runs deep research chains. The global default was set to timeoutSeconds: 600 (10 minutes), causing active, non-stuck runs to be killed mid-flight.
  • Resolution: In k8s/openclaw/openclaw-instance.secret.yaml, configure: 
    spec:
  config:
    raw:
      agents:
        defaults:
          timeoutSeconds: 1800  # 30 minutes
  • How to Avoid Recurrence: Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.

Issue 2: Schema Crash on Per-Agent Timeout Override

  • Date: 2026-09-20
  • Severity: Critical (Deployment blocked with CrashLoopBackOff)
  • Symptom: Gateway container fails during startup: 
    Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
  • Root Cause: OpenClaw’s JSON schema strict validator does not permit timeoutSeconds inside individual agent entries under agents.entries.<name>.
  • Resolution: Remove timeoutSeconds from agents.entries.*. Configure it strictly at the root of agents.defaults.timeoutSeconds.
  • How to Avoid Recurrence: Before adding new keys to agent configurations in openclaw-instance.secret.yaml, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.

Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization

  • Date: 2026-09-24
  • Severity: High (Intermittent container kills on gateway restart)
  • Symptom: Container openclaw fails startup or liveness probes, causing kubelet to restart the container: 
    Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
Startup probe failed: context deadline exceeded / connection reset by peer
    Logs display repeated warnings: 
    [state/agent-db] slow OpenClaw agent database open
  • Root Cause: OpenClaw maintains separate SQLite databases for each specialist subagent (agents/*/agent/openclaw-agent.sqlite). Over time, databases and uncommitted WAL logs accumulate significant size:
    • agents/main/agent/openclaw-agent.sqlite: >220 MB
    • agents/fitness/, agents/itguy/, agents/hebamme/, agents/recherche/: >100 MB each + multi-megabyte WAL files. On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the startupProbe limit (failureThreshold * periodSeconds), Kubernetes kills the container before it finishes opening databases.
  • Resolution:
    1. Increase probes.startup.failureThreshold in openclaw-instance.secret.yaml to 240 (with 5s period = 1200s / 20 min).
    2. Periodically run PRAGMA wal_checkpoint(TRUNCATE) across all SQLite databases to compact WAL journals into main files.
  • How to Avoid Recurrence: Monitor SQLite database sizes under /home/openclaw/.openclaw/agents/*/agent/ and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.

Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary

  • Date: 2026-09-28
  • Severity: High (Subagents lose access to user profile and memory context)
  • Symptom: On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports: 
    [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/USER.md reason=path must not be hardlinked
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/MEMORY.md reason=path must not be hardlinked
  • Root Cause: OpenClaw’s embedded execution harness enforces a strict filesystem security check: stat.nlink > 1 triggers path must not be hardlinked. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files.
  • Resolution: Decouple all hardlinked files so that each workspace has its own standalone copy with nlink == 1: 
    for dir in /home/node/.openclaw/workspace-*; do
  for file in USER.md MEMORY.md; do
    cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file"
  done
done
    Ensure the init container in openclaw-instance.secret.yaml maintains this pattern with cp --remove-destination.
  • How to Avoid Recurrence: Never create hardlinks (ln without -s) inside OpenClaw workspace directories. Always use standalone file copies (cp --remove-destination).

Issue 5: Cron Trigger Sandbox Scope ReferenceError

  • Date: 2026-09-28
  • Severity: Medium (Automated recurring cron job fails to evaluate)
  • Symptom: In cron_run_receipts, the job rennerhof-website-watch failed with: 
    cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
    at <anonymous> (openclaw-code-mode:user.js:12:172)
  • Root Cause: The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access state && state.hash directly, causing a ReferenceError when state is undeclared in the global scope.
  • Resolution: Use typeof state !== "undefined" to safely guard against undeclared variables in the trigger script: 
    const s = (typeof state !== 'undefined' && state) ? state : {};
const prev = s.hash;
  • How to Avoid Recurrence: In OpenClaw trigger scripts, never evaluate bare identifier state directly without a typeof guard.

Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)

  • Date: 2026-09-28
  • Severity: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
  • Symptom:
    1. The #Mila (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang: 
      Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
    2. OpenClaw pod logs showed critical memory pressure (rss=1.98 GiB, threshold 1.64 GiB) and a massive Node.js event loop delay of 31.4 seconds (eventLoopDelayP99Ms=31406).
  • Root Cause:
    1. Uncompacted Topic Sessions: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached 753 messages and 127,607 tokens; IT Guy reached 204,670 tokens; Regensburg reached 160,029 tokens; Fitness reached 135,990 tokens.
    2. Model Context Saturation in openrouter/z-ai/glm-5.3-flash: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row 1062, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
    3. Compaction Gap: The existing nightly-session-compaction cron job only targeted "agent:main:main", neglecting all specialist subagent forum topic sessions. Furthermore, compaction.maxActiveTranscriptBytes: "3mb" in config requires ~700k raw text tokens to trigger natively.
  • Resolution:
    1. Immediately ran openclaw sessions compact across all bloated subagent topic sessions, bringing Hebamme from 127,607 down to 11,892 tokens and IT Guy from 204,670 to 250 lines.
    2. Created /home/node/.openclaw/scripts/compact_sessions.py on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
    3. Updated the OpenClaw Cron automation nightly-session-compaction to execute this script every night at 03:30 Europe/Berlin (30 3 * * *).
    4. Created k8s/openclaw/setup-compaction-cron.sh in git as the reproducible source of truth for the automation.
  • How to Avoid Recurrence: Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.
Reference in New Issue View Git Blame Copy Permalink