# OpenClaw Troubleshooting & Incident Playbook

This document captures fixed issues, operational quirks, and incident root-cause analyses for the **OpenClaw** deployment on Kubernetes.

> [!IMPORTANT]
> **Rule for Agents & Operators**: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, **document it here immediately**. This prevents recurring regressions and serves as the single source of truth for troubleshooting.

---

## 📋 Catalog of Incidents & Fixed Issues

| Date | Category | Summary / Symptom | Status | Key Resolution |
| :--- | :--- | :--- | :---: | :--- |
| **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). |
| **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.<id>: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. |
| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Probe budget vs WAL checkpointing on large SQLite files (~1.5GB across subagents). |

---

## 🔍 Detailed Incident Records

### Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`)

- **Date**: 2026-09-20
- **Severity**: High (User-facing timeout in Telegram topics `#IT Guy` and `#Recherche`)
- **Symptom**:
  Telegram replies with error message:
  ```text
  Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
  ```
- **Root Cause**:
  The IT Guy agent (`itguy`) performs deep, sequential DevOps investigations (multiple `exec` / `kubectl` calls, file writes, and PR operations) using `deepseek/deepseek-v4.1-flash`. Similarly, `#Recherche` runs deep research chains. The global default was set to `timeoutSeconds: 600` (10 minutes), causing active, non-stuck runs to be killed mid-flight.
- **Resolution**:
  In [`k8s/openclaw/openclaw-instance.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml), configure:
  ```yaml
  spec:
    config:
      raw:
        agents:
          defaults:
            timeoutSeconds: 1800  # 30 minutes
  ```
- **How to Avoid Recurrence**:
  Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.

---

### Issue 2: Schema Crash on Per-Agent Timeout Override

- **Date**: 2026-09-20
- **Severity**: Critical (Deployment blocked with `CrashLoopBackOff`)
- **Symptom**:
  Gateway container fails during startup:
  ```text
  Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
  openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
  ```
- **Root Cause**:
  OpenClaw’s JSON schema strict validator does not permit `timeoutSeconds` inside individual agent entries under `agents.entries.<name>`.
- **Resolution**:
  Remove `timeoutSeconds` from `agents.entries.*`. Configure it strictly at the root of `agents.defaults.timeoutSeconds`.
- **How to Avoid Recurrence**:
  Before adding new keys to agent configurations in `openclaw-instance.secret.yaml`, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.

---

### Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization

- **Date**: 2026-09-24
- **Severity**: High (Intermittent container kills on gateway restart)
- **Symptom**:
  Container `openclaw` fails startup or liveness probes, causing kubelet to restart the container:
  ```text
  Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
  Startup probe failed: context deadline exceeded / connection reset by peer
  ```
  Logs display repeated warnings:
  ```text
  [state/agent-db] slow OpenClaw agent database open
  ```
- **Root Cause**:
  OpenClaw maintains separate SQLite databases for each specialist subagent (`agents/*/agent/openclaw-agent.sqlite`). Over time, databases and uncommitted WAL logs accumulate significant size:
  - `agents/main/agent/openclaw-agent.sqlite`: >220 MB
  - `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files.
  On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases.
- **Resolution**:
  1. Ensure `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` allows adequate startup headroom (e.g. at least 120 attempts with 5s period = 600s).
  2. Periodically ensure SQLite VACUUM / WAL checkpoints run during maintenance windows or let the database open without interrupting it.
- **How to Avoid Recurrence**:
  Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.