Files

T

Moritz Graf e45dbc2e19 feat(sasch40): improve Act III, add state persistence and fix portraits

- Fix character portrait crops and version assets to portrait_v2, ensuring correct identities (resolving Nat/Verena mismatch)
- Update Schankraum scene image and configure Nginx cache-control to prevent stale asset caching
- Make secret beer tap completely invisible in tavern until clicked
- Reposition room hotspots in Act III to eliminate overlapping click targets
- Add exit button [✕] to Bavarian Gstanzl duel minigame
- Implement localStorage persistence in gameStore to maintain progress across page reloads
- Enable clean game state and storage reset when clicking "Nochmal spielen"
- Expand automated playtest script to cover Nat dialogue, duel exit, reload persistence, and game restart

2026-09-24 11:03:22 +02:00

5.3 KiB

Raw Blame History

OpenClaw Troubleshooting & Incident Playbook

This document captures fixed issues, operational quirks, and incident root-cause analyses for the OpenClaw deployment on Kubernetes.

Important

Rule for Agents & Operators: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, document it here immediately. This prevents recurring regressions and serves as the single source of truth for troubleshooting.

📋 Catalog of Incidents & Fixed Issues

Date	Category	Summary / Symptom	Status	Key Resolution
2026-09-20	Timeout & Execution	Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded)	`RESOLVED`	Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min).
2026-09-20	Config Schema Validation	Pod crash with `CrashLoopBackOff`: `agents.entries.<id>: Unrecognized key: "timeoutSeconds"`	`RESOLVED`	`timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`.
2026-09-24	Database & Startup Latency	Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures	`RESOLVED`	Probe budget vs WAL checkpointing on large SQLite files (~1.5GB across subagents).

🔍 Detailed Incident Records

Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`)

Date: 2026-09-20
Severity: High (User-facing timeout in Telegram topics #IT Guy and #Recherche)

Symptom: Telegram replies with error message:

Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.

Root Cause: The IT Guy agent (itguy) performs deep, sequential DevOps investigations (multiple exec / kubectl calls, file writes, and PR operations) using deepseek/deepseek-v4.1-flash. Similarly, #Recherche runs deep research chains. The global default was set to timeoutSeconds: 600 (10 minutes), causing active, non-stuck runs to be killed mid-flight.

Resolution: In k8s/openclaw/openclaw-instance.secret.yaml, configure:

spec:
  config:
    raw:
      agents:
        defaults:
          timeoutSeconds: 1800  # 30 minutes

How to Avoid Recurrence: Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.

Issue 2: Schema Crash on Per-Agent Timeout Override

Date: 2026-09-20
Severity: Critical (Deployment blocked with CrashLoopBackOff)

Symptom: Gateway container fails during startup:

Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"

Root Cause: OpenClaw’s JSON schema strict validator does not permit timeoutSeconds inside individual agent entries under agents.entries.<name>.
Resolution: Remove timeoutSeconds from agents.entries.*. Configure it strictly at the root of agents.defaults.timeoutSeconds.
How to Avoid Recurrence: Before adding new keys to agent configurations in openclaw-instance.secret.yaml, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.

Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization

Date: 2026-09-24
Severity: High (Intermittent container kills on gateway restart)

Symptom: Container openclaw fails startup or liveness probes, causing kubelet to restart the container:

Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
Startup probe failed: context deadline exceeded / connection reset by peer

Logs display repeated warnings:

[state/agent-db] slow OpenClaw agent database open

Root Cause: OpenClaw maintains separate SQLite databases for each specialist subagent (agents/*/agent/openclaw-agent.sqlite). Over time, databases and uncommitted WAL logs accumulate significant size:
- agents/main/agent/openclaw-agent.sqlite: >220 MB
- agents/fitness/, agents/itguy/, agents/hebamme/, agents/recherche/: >100 MB each + multi-megabyte WAL files. On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the startupProbe limit (failureThreshold * periodSeconds), Kubernetes kills the container before it finishes opening databases.
Resolution:
1. Ensure probes.startup.failureThreshold in openclaw-instance.secret.yaml allows adequate startup headroom (e.g. at least 120 attempts with 5s period = 600s).
2. Periodically ensure SQLite VACUUM / WAL checkpoints run during maintenance windows or let the database open without interrupting it.
How to Avoid Recurrence: Monitor SQLite database sizes under /home/openclaw/.openclaw/agents/*/agent/ and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.

5.3 KiB Raw Blame History Unescape Escape