# OpenClaw Troubleshooting & Incident Playbook

This document captures fixed issues, operational quirks, and incident root-cause analyses for the **OpenClaw** deployment on Kubernetes.

> [!IMPORTANT]
> **Rule for Agents & Operators**: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, **document it here immediately**. This prevents recurring regressions and serves as the single source of truth for troubleshooting.

---

## 📋 Catalog of Incidents & Fixed Issues

| Date | Category | Summary / Symptom | Status | Key Resolution |
| :--- | :--- | :--- | :---: | :--- |
| **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). |
| **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.<id>: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. |
| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. |
| **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
| **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |
| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
| **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. |
| **2026-10-02** | **Memory Cgroup OOMKill & Probe Flapping** | Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported `failed` delivery | `RESOLVED` | Injected `NODE_OPTIONS="--max-old-space-size=3072"`, raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added `--no-deliver` + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens). |
| **2026-10-05** | **Disk I/O & Startup Latency** | Event loop starvation (48s freeze) and container restarts from MCP process storms & cold-start database integrity scans | `RESOLVED` | Implemented init-container page-cache pre-warming (`cat ... > /dev/null`) cutting DB open from 22 minutes to 11 seconds. |
| **2026-10-05** | **CronJob & Alerting** | `openclaw-wiki-audit` CronJob failing daily with exit code 1; Prometheus alert `WikiAuditJobFailed` firing | `RESOLVED` | Appended `\|\| true` to `okf_lint.py` in `cronjob-wiki-audit.secret.yaml`, preventing `set -e` abort when lint findings exist. |
| **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
| **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
| **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
| **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. |
| **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. |
| **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. |
| **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis (Butter vs. Pflanzenaufstriche), decoupled `recherche.md` prompt into Modus A/B, raised startup probe threshold to 360 (30m), and updated `recover_sessions.py` to detect interrupted user turns. |


---

## 🔍 Detailed Incident Records

### Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`)

- **Date**: 2026-09-20
- **Severity**: High (User-facing timeout in Telegram topics `#IT Guy` and `#Recherche`)
- **Symptom**:
  Telegram replies with error message:
  ```text
  Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
  ```
- **Root Cause**:
  The IT Guy agent (`itguy`) performs deep, sequential DevOps investigations (multiple `exec` / `kubectl` calls, file writes, and PR operations) using `deepseek/deepseek-v4.1-flash`. Similarly, `#Recherche` runs deep research chains. The global default was set to `timeoutSeconds: 600` (10 minutes), causing active, non-stuck runs to be killed mid-flight.
- **Resolution**:
  In [`k8s/openclaw/openclaw-instance.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml), configure:
  ```yaml
  spec:
    config:
      raw:
        agents:
          defaults:
            timeoutSeconds: 1800  # 30 minutes
  ```
- **How to Avoid Recurrence**:
  Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.

---

### Issue 2: Schema Crash on Per-Agent Timeout Override

- **Date**: 2026-09-20
- **Severity**: Critical (Deployment blocked with `CrashLoopBackOff`)
- **Symptom**:
  Gateway container fails during startup:
  ```text
  Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
  openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
  ```
- **Root Cause**:
  OpenClaw’s JSON schema strict validator does not permit `timeoutSeconds` inside individual agent entries under `agents.entries.<name>`.
- **Resolution**:
  Remove `timeoutSeconds` from `agents.entries.*`. Configure it strictly at the root of `agents.defaults.timeoutSeconds`.
- **How to Avoid Recurrence**:
  Before adding new keys to agent configurations in `openclaw-instance.secret.yaml`, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.

---

### Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization

- **Date**: 2026-09-24
- **Severity**: High (Intermittent container kills on gateway restart)
- **Symptom**:
  Container `openclaw` fails startup or liveness probes, causing kubelet to restart the container:
  ```text
  Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
  Startup probe failed: context deadline exceeded / connection reset by peer
  ```
  Logs display repeated warnings:
  ```text
  [state/agent-db] slow OpenClaw agent database open
  ```
- **Root Cause**:
  OpenClaw maintains separate SQLite databases for each specialist subagent (`agents/*/agent/openclaw-agent.sqlite`). Over time, databases and uncommitted WAL logs accumulate significant size:
  - `agents/main/agent/openclaw-agent.sqlite`: >220 MB
  - `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files.
  On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases.
- **Resolution**:
  1. Increase `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` to 240 (with 5s period = 1200s / 20 min).
  2. Periodically run `PRAGMA wal_checkpoint(TRUNCATE)` across all SQLite databases to compact WAL journals into main files.
- **How to Avoid Recurrence**:
  Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.

---

### Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary

- **Date**: 2026-09-28
- **Severity**: High (Subagents lose access to user profile and memory context)
- **Symptom**:
  On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports:
  ```text
  [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/USER.md reason=path must not be hardlinked
  [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/MEMORY.md reason=path must not be hardlinked
  ```
- **Root Cause**:
  OpenClaw’s embedded execution harness enforces a strict filesystem security check: `stat.nlink > 1` triggers `path must not be hardlinked`. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files.
- **Resolution**:
  Decouple all hardlinked files so that each workspace has its own standalone copy with `nlink == 1`:
  ```bash
  for dir in /home/node/.openclaw/workspace-*; do
    for file in USER.md MEMORY.md; do
      cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file"
    done
  done
  ```
  Ensure the init container in `openclaw-instance.secret.yaml` maintains this pattern with `cp --remove-destination`.
- **How to Avoid Recurrence**:
  Never create hardlinks (`ln` without `-s`) inside OpenClaw workspace directories. Always use standalone file copies (`cp --remove-destination`).

---

### Issue 5: Cron Trigger Sandbox Scope ReferenceError

- **Date**: 2026-09-28
- **Severity**: Medium (Automated recurring cron job fails to evaluate)
- **Symptom**:
  In `cron_run_receipts`, the job `rennerhof-website-watch` failed with:
  ```text
  cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
      at <anonymous> (openclaw-code-mode:user.js:12:172)
  ```
- **Root Cause**:
  The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access `state && state.hash` directly, causing a `ReferenceError` when `state` is undeclared in the global scope.
- **Resolution**:
  Use `typeof state !== "undefined"` to safely guard against undeclared variables in the trigger script:
  ```javascript
  const s = (typeof state !== 'undefined' && state) ? state : {};
  const prev = s.hash;
  ```
- **How to Avoid Recurrence**:
  In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard.

---

### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)

- **Date**: 2026-09-28
- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
- **Symptom**:
  1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang:
     ```text
     Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
     Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
     ```
  2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`).
- **Root Cause**:
  1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**.
  2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
  3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively.
- **Resolution**:
  1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**.
  2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
  3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`).
  4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation.
- **How to Avoid Recurrence**:
  Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.

---

### Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck

- **Date**: 2026-09-29
- **Severity**: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in `Dl` state)
- **Symptom**:
  1. Telegram bot stopped answering messages entirely on the morning of Sep 29.
  2. `openclaw-0` pod was killed by kubelet with exit code 137 (`Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded`).
  3. Upon restarting, the container remained in `4/5 Running` for over 15 minutes, with the gateway process (`openclaw-gateway`, PID 15930) stuck in Linux uninterruptible sleep state `Dl` (`generic_file_buffered_read`) and logging repeated:
     ```text
     [state/agent-db] slow OpenClaw agent database open
     ```
- **Root Cause**:
  1. **Recurring Hardlink Regression**: The nightly `openclaw-wiki-backup` CronJob runs daily at 03:30 UTC. Lines 41-42 in `cronjob-wiki-backup.secret.yaml` contained:
     ```bash
     [ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true
     [ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true
     ```
     This routinely re-created hardlinks (`nlink == 11`) across all subagent workspaces every night, directly undoing the `nlink == 1` fix from Issue 4 and causing OpenClaw to reject `USER.md` and `MEMORY.md`. When a user message arrived in `#Mila` at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137).
  2. **Spinning Disk Random I/O Bottleneck on Startup**: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs `verifyAndRepairCanonicalSqliteIndexes` on each database. On rotational SATA hard drives (`/dev/sda2`, `rotational=1`), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold.
- **Resolution**:
  1. **Permanent Manifest Fix**: Patched `k8s/openclaw/cronjob-wiki-backup.secret.yaml` to replace `ln -f` with `cp --remove-destination`, ensuring subagent workspace bootstrap files remain strict standalone files (`nlink == 1`). Applied the updated manifest to the cluster and pushed commit `74e32ed` to git.
  2. **Volume Decoupling**: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies.
  3. **Page Cache Pre-warming**: Sequentially streamed each agent SQLite database into the Linux RAM page cache (`cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null`). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent.
  4. The gateway successfully bound `:18789`, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy `5/5 Running` state.
- **How to Avoid Recurrence**:
  - Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`.
  - When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.

---

### Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction `--max-lines` Pitfall

- **Date**: 2026-09-29
- **Severity**: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German)
- **Symptom**:
  1. A user sent a voice message to the `#Einkaufen` (Shopping) topic asking for recommendations for a kitchen scale (`Küchenwaage`).
  2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German:
     ```text
     Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel...
     ```
  3. The user was concerned whether the system had been compromised or hacked.
- **Root Cause**:
  1. **The `--max-lines` Compaction Flaw**:
     The nightly compaction cron script (`setup-compaction-cron.sh` / `compact_sessions.py`) invoked:
     ```bash
     openclaw sessions compact <key> --agent <agent> --max-lines 250
     ```
     According to the OpenClaw CLI specification, `--max-lines <count>` means:
     `Truncate to the last N transcript lines instead of LLM summarization`.
     If a session has $\le 250$ transcript lines, OpenClaw outputs `No compaction needed for session ...` and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached **38,000 to 95,000 tokens** while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (`shopping`, `recherche`, `grabengasse_media`, etc.).
  2. **Split-Turn Context Anchoring**:
     When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a `Turn Context (split turn)` block indicating unfinished work:
     ```markdown
     Turn Context (split turn):
     ### Original Request
     User korrigierte die Supermarkt-Liste der Wiki-Seite...
     ### Next Steps
     1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen...
     3. Änderung gegenüber Nutzer bestätigen.
     ```
     When the new voice note arrived at `seq=231`, the prompt presented to `z-ai/glm-5.3-flash` was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (`seq=232`) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding:
     ```text
     "No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction."
     ```
     It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German).
- **Resolution**:
  1. **Remove `--max-lines` for True LLM Compaction**:
     Updated `setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` to remove `--max-lines`, invoking `openclaw sessions compact <key> --agent <agent>` so OpenClaw performs native LLM summarization.
  2. **Compacted Live Sessions**:
     Ran LLM compaction on `shopping:topic:115`, successfully shrinking it from **39,283 tokens down to 5,343 tokens** (an 86% context reclamation).
  3. **Event Loop Pacing**:
     Added `time.sleep(2)` between consecutive session compactions in `compact_sessions.py` to allow Node.js event loop ticks and prevent Kubelet `/healthz` liveness probe timeouts during batch compactions.
- **How to Avoid Recurrence**:
  - Never use `--max-lines` when the objective is semantic context reduction; omit `--max-lines` to let OpenClaw invoke LLM summarization.
  - Monitor session token sizes (`openclaw sessions --all-agents --json`) rather than line counts.
  - Keep `setup-compaction-cron.sh` aligned across git and live PVCs.

---

### Issue 9: Speech-to-Text / Audio Transcription Failure (`ProviderHttpError`) & Empty Voice Note Transcripts

- **Date**: 2026-09-29
- **Severity**: High (Telegram voice notes fail transcription, passing empty messages to agents and prompting hallucinated/frantic behavior)
- **Symptom**:
  1. Telegram voice notes fail transcription with:
     ```text
     [media-understanding] audio: failed (0/1) reason=ProviderHttpError
     ```
     in ~300–500ms.
  2. The inbound Telegram message is delivered to the agent with `0 chars` text (`(group, audio/ogg, 0 chars)`).
  3. The agent receives an empty prompt or placeholder (`[Audio attachment could not be analyzed]`). It does not know what the user requested, resulting in:
     - No reply,
     - Completely hallucinated or unrelated replies, or
     - Frantic autonomous attempts by the agent to find and transcribe the audio file in `/home/node/.openclaw/media/inbound/` via terminal commands (`ffmpeg`, `python3`, `uv`, free model APIs), producing errors and strange messages that give the appearance the bot was "hacked".
- **Root Cause**:
  1. **OpenRouter Audio Balance Floor ($0.50)**:
     OpenRouter requires at least **$0.50 in credit buffer / remaining key limit** for any audio transcription request (`openai/whisper-large-v3-turbo` or multimodal audio). When the monthly limit was set to $70.00 and usage reached ~$59.80 (leaving <$0.50 of head room under the prior limit), OpenRouter rejected audio requests with HTTP 402:
     ```json
     {"error":{"message":"This request requires at least $0.50 in balance for audio","code":402,"metadata":{"limit_source":"openrouter_key_limit"}}}
     ```
  2. **Chat Completions Succeeded while Audio Failed**:
     Because text completions cost only fractions of a cent ($0.0001–$0.001), regular chat queries continued to work normally. Only audio transcriptions were blocked by the $0.50 threshold.
  3. **Error Truncation in OpenClaw**:
     In `runner.entries-LWAceiTW.js`, the error was processed by `summarizeDecisionReason(reason)`, which executes `normalizeDecisionReason(reason).split(":")[0]`. Because `ProviderHttpError` was not stripped by the regex, the error string `ProviderHttpError: OpenRouter audio transcription failed (HTTP 402)...` was truncated to `ProviderHttpError`. This obscured the HTTP 402 status code and key limit message in normal logs.
  4. **Single Provider without Fallback**:
     `openclaw-instance.secret.yaml` only declared a single audio provider (`openrouter/openai/whisper-large-v3-turbo`) with no fallback provider configured. When OpenRouter returned 402, OpenClaw marked the audio as failed and fell back to empty input.
- **Resolution**:
  1. **Buffer Headroom Verified**:
     Confirmed OpenRouter key monthly limit has ample remaining buffer ($10.19 remaining), and re-tested live audio transcription of the exact failed 52-second Telegram voice files (`b9e0da37...` and `aef0424f...`), confirming fast, accurate transcription (HTTP 200).
  2. **Recherche Agent Upgraded**:
     Switched `recherche` agent from `openrouter/z-ai/glm-5.3-flash` to `openrouter/deepseek/deepseek-v4.1-flash` in `openclaw-instance.secret.yaml` and live `openclaw.json` to ensure robust analytical reasoning and prevent prompt confusion.
- **How to Avoid Recurrence**:
  - Maintain a sufficient credit buffer on OpenRouter (at least $5–$10 remaining monthly headroom).
  - Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
  - Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.

---

### Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure

- **Date**: 2026-10-02
- **Severity**: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status)
- **Symptom**:
  1. Pod restarted at `2026-10-02T12:40:43Z` with `lastState: terminated (exitCode: 137, reason: OOMKilled)`.
  2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (`context deadline exceeded while awaiting headers`).
  3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit).
  4. The nightly session compaction cron job `nightly-session-compaction` completed with `completionStatus: "failed"` due to unrouted announcement delivery.
- **Root Cause**:
  1. **Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting**: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit `--max-old-space-size` ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL.
  2. **Aggressive Liveness Threshold on Single-Threaded Event Loop**: The liveness probe was set to `failureThreshold: 5, periodSeconds: 10` (50s tolerance). When disk I/O on the physical spinning HDD (`/dev/sda2`) or memory sync tasks delayed the Node.js event loop by 46 seconds (`overdue=16192ms elapsed=46192ms`), Kubelet killed the container.
  3. **Unbound Cron Delivery**: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or `--no-deliver`, OpenClaw fails delivery closed (`Refusing implicit isolated cron delivery: delivery.to is required`).
  4. **Uncheckpointed SQLite WAL Journals & Dead Sessions**: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens.
- **Resolution**:
  1. Injected `NODE_OPTIONS="--max-old-space-size=3072"` into `k8s/openclaw/openclaw-instance.secret.yaml` to enforce V8 garbage collection when heap reaches 3 GB.
  2. Increased liveness probe tolerance to `failureThreshold: 12, timeoutSeconds: 15` (120s tolerance).
  3. Lowered native safeguard compaction threshold `maxActiveTranscriptBytes` from `3mb` to `400kb` (~50k tokens).
  4. Updated `k8s/openclaw/setup-compaction-cron.sh` with `--no-deliver` and automated nightly `PRAGMA wal_checkpoint(TRUNCATE)` across all agent DBs.
  5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens.
- **How to Avoid Recurrence**:
  - Keep `NODE_OPTIONS="--max-old-space-size=3072"` configured in the container spec.
  - Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases.
  - Always register isolated automated cron jobs with `--no-deliver`.

---

### Issue 11: Event Loop Starvation and Probe Timeouts from MCP Process Storms & Cold-Start SQLite Verification on HDD Storage

- **Date**: 2026-10-05
- **Severity**: Critical (Pod flapping with startup/liveness probe timeouts, event loop frozen for 48s–70s)
- **Symptom**:
  1. OpenClaw container restarted frequently; logs recorded massive timing gaps:
     ```text
     [health] host timing gap detected: process was frozen ~48830ms (clock jump: 48830ms, interval: 10000ms)
     system-prompt:64391ms
     [state/agent-db] slow OpenClaw agent database open
     ```
  2. Kubelet liveness/startup probes timed out with `context deadline exceeded`.
  3. Cold starts took up to 22 minutes verifying database B-tree integrity across 12 agent SQLite stores.
- **Root Cause**:
  1. **MCP Subprocess Storm**: 16 MCP servers were registered globally in `openclaw-instance.secret.yaml`. On turns or background runs, OpenClaw spawned all 16 processes simultaneously (Python, Node.js, and Go). Concurrent agent turns resulted in 32+ processes competing for CPU and random I/O on the rotational SATA drive (`/dev/sda2`), spiking RAM to 2.43 GB and starving the Node.js event loop.
  2. **Cold-Start Disk Seek Bottleneck**: On startup, OpenClaw runs SQLite integrity and schema assertions across 12 databases (~1.8 GB). On rotational disk, thousands of random 4KB read operations take up to 22 minutes if the OS page cache is cold.
  3. **ext4 Directory Inode Bloat**: `/home/node/.openclaw` directory inode had grown to 17.8 MB from historical file deletions. ext4 does not shrink directory leaf blocks upon unlink, forcing the kernel to scan ~4,300 directory blocks on every stat/lookup.
- **Resolution**:
  1. Implemented sequential page-cache pre-warming in the pod `initContainer` and nightly cron script:
     ```sh
     cat /home/node/.openclaw/state/openclaw.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true
     ```
     Because sequential read throughput on the HDD is ~150 MB/s, the kernel streams all 1.8 GB into RAM in ~11 seconds. When the main container boots, index verification runs in 11 seconds instead of 22 minutes.
  2. Applied manifest to ensure all restarts benefit from page cache warming.
- **How to Avoid Recurrence**:
  - Always maintain database streaming in the pod `initContainer` before OpenClaw starts.
  - Avoid spawning unnecessary MCP servers; ensure heavy processes are lazy or decoupled.

---

### Issue 12: Knowledge Wiki Audit CronJob Failure Loop (`set -e` vs Lint Findings) and Prometheus Alert (`WikiAuditJobFailed`)

- **Date**: 2026-10-05
- **Severity**: High (Daily audit CronJob failed repeatedly; Prometheus alert firing; git repo left uncommitted)
- **Symptom**:
  1. Kubernetes CronJob `openclaw-wiki-audit` failed every night at 00:05 UTC with exit code 1.
  2. Prometheus fired persistent alert `WikiAuditJobFailed`.
  3. Changes detected by the linter were never staged or committed, leaving the repository dirty on the PVC.
- **Root Cause**:
  In [`k8s/openclaw/cronjob-wiki-audit.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/cronjob-wiki-audit.secret.yaml), the script had `set -euo pipefail` at line 25. Line 47 executed:
  ```bash
  python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks
  ```
  `okf_lint.py` deliberately exits with code 1 whenever any lint findings or actionable tasks are added to Obsidian notes. Because of `set -e`, the shell script aborted immediately at line 47, skipping `git commit`, `git push`, and exiting 1.
- **Resolution**:
  Patched line 47 to append `|| true`:
  ```bash
  python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks || true
  ```
  Applied manifest, deleted the stale failed job, and triggered a test job. The test successfully audited 117 files, committed `chore(audit): automated OKF health check [skip ci]`, pushed to Gitea, and exited with code 0, clearing the Prometheus alert.
- **How to Avoid Recurrence**:
  - Be mindful of command exit semantics in scripts using `set -e`. Linters and audit tools frequently return non-zero exit codes to indicate "findings were present", which is expected behavior, not a script crash.

---

### Issue 13: Stale Cron Job Error State on `rennerhof-website-watch` (Regensburg Topic 116)

- **Date**: 2026-10-05
- **Severity**: Medium (Website watch cron job showed red error status in OpenClaw dashboard)
- **Symptom**:
  The cron automation `rennerhof-website-watch` for Regensburg Topic 116 showed `status: error` with:
  ```text
  cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
  ```
- **Root Cause**:
  Although the trigger code was previously patched, the historical error was preserved in `state_json` in `/home/node/.openclaw/state/openclaw.sqlite`, causing OpenClaw to report the job as failed.
- **Resolution**:
  Cleared the error state from `state_json` in SQLite and executed a manual trigger evaluation:
  ```json
  "fire": false, "evals": 12, "last_result": null, "status": "running"
  ```
  Job state returned to healthy `running` with zero errors.
- **How to Avoid Recurrence**:
  - After fixing JavaScript trigger logic in cron jobs, always clear stale error payloads from `openclaw.sqlite` or trigger an explicit run to update state.

---

### Issue 14: Ungraceful Restart Deadlock on Telegram Topic 117 (`#Recherche`): `entry_valid = 0` and Recovery Claim Deadlock

- **Date**: 2026-10-05
- **Severity**: Critical (Topic 117 completely unresponsive in Telegram; every message failed with error and retried endlessly)
- **Symptom**:
  1. Inbound messages to Telegram Topic 117 failed with:
     ```text
     [telegram] dispatch failed: Error: restart recovery claim changed before agent adoption
     [telegram] spooled update 985533189 failed; keeping for retry: restart recovery claim changed before agent adoption
     ```
  2. Running any CLI command against the recherche agent failed with:
     ```text
     SessionCanonicalKeyMigrationRequiredError: invalid persisted session row requires repair for agent:recherche:telegram:group:-1004326921510:topic:117; stop the Gateway and run openclaw doctor --fix
     ```
- **Root Cause**:
  1. **Active Writer Abort**: When the pod was killed or restarted during an active message turn, `session_nodes` and `entry_json` were left with `status: "running"`, `activeWriterRunId`, and `restartRecoveryDeliveryRunId`.
  2. **Admission Deadlock**: In `reply-admission-ticket.js:469`, when new messages arrive while `activeClaimRunId` is set and `entry.status === "running"`, the runtime throws `Error: restart recovery claim changed before agent adoption`. Because it throws before resetting state, Telegram kept spooling and retrying the message every 30 seconds forever.
  3. **Database Invalidation (`entry_valid = 0`)**: During the crash, OpenClaw marked `entry_valid = 0` in `session_nodes`. The runtime validator `scanCanonicalSqliteSessionEntries()` requires `row.entry_valid === 1`, throwing `SessionCanonicalKeyMigrationRequiredError` on any read.
  4. **Column Synchronization Requirement**: `parseSqliteSessionEntryRecord(row)` requires that `row.updated_at === record.updatedAt`. Any update modifying one without the other causes the parser to return `null` and reject the session.
  5. **Dead Cron Session Accumulation**: 20 isolated cron sessions from September were never pruned, holding >498,000 dead tokens across `main` and `itguy`.
- **Resolution**:
  1. Synchronized SQLite state in `/home/node/.openclaw/agents/recherche/agent/openclaw-agent.sqlite`:
     - Set `entry_valid = 1`.
     - Reset `status = "done"`, cleared `restartRecoveryDeliveryRunId`, `restartRecoveryDeliverySourceRunId`, `restartRecoveryDeliveryContext`.
     - Synchronized `updated_at` on row and `updatedAt` in `entry_json`.
     - Updated `session_windows` to `status = "done"`.
  2. Upon repair, OpenClaw immediately admitted the spooled Telegram update and dispatched a clean response via DeepSeek V4.1 Flash.
  3. Purged all 20 abandoned cron sessions older than 48 hours across `main` and `itguy`, reclaiming 498,171 tokens and checkpointing WAL journals to disk.
  4. Upgraded `setup-compaction-cron.sh`:
     - Added automatic pre-warming of SQLite page cache.
     - Added automated pruning of isolated cron sessions older than 48 hours.
     - Raised compaction RPC timeout from 10s to 120s (`--timeout 120000`).
     - Added automatic fallback to `--max-lines 200` if LLM summarization safeguards fail quality checks.
- **How to Avoid Recurrence**:
  - The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache.
  - If a session ever reports `invalid persisted session row requires repair`, check `entry_valid == 1` and ensure `row.updated_at == record.updatedAt`.

---

### Issue 15: Subagents Emitted Raw HTML/XML Comments (`<!-- DOMAIN_STATE: { ... } -->`) in Telegram Messages

- **Date**: 2026-10-06
- **Severity**: Medium (Visual/UX defect: raw HTML comments and JSON leaked into user-facing chat messages)
- **Symptom**:
  Messages sent by specialist subagents in Telegram forum topics (`#Mila`, `#Medizin`, `#Shopping`, `#Travel`, `#Sport`, `#Regensburg`, etc.) consistently concluded with a raw HTML/XML comment block, for example:
  ```html
  <!-- DOMAIN_STATE: {
    "agent": "doktor",
    "topic_category": "symptom_check",
    "evidence_source": "AWMF",
    "triage_urgency": "routine"
  } -->
  ```
- **Root Cause**:
  1. **Prompt Directive**: In `knowledge-wiki/agents/*.md`, each subagent prompt contained a `## 🔄 State Bridge Schema` instructing the LLM: *„At the end of your response, output a compact state comment: `<!-- DOMAIN_STATE: { ... } -->`“*.
  2. **Telegram Formatting Mismatch**: The schema was originally conceived in ADR 0001 under the assumption that `<!-- ... -->` would be invisible in chat rendering. However, Telegram's Bot API does not support hidden HTML comments. OpenClaw's text formatter safely escaped them to `&lt;!-- ... --&gt;`, causing the Telegram client to render the comments literally as visible text.
  3. **Architectural Redundancy**: Subagents operate inside persistent SQLite sessions with full conversation history and vector embeddings. Furthermore, structured domain states (sleep, nutrition, feeding, workouts) were transitioned to DuckDB/JSONL via `skills/datastore` (ADR 0004). No system component or code read `DOMAIN_STATE`.
- **Resolution**:
  1. Removed `## 🔄 State Bridge Schema` from all 9 subagent definition files in `knowledge-wiki/agents/*.md` (`hebamme`, `doktor`, `shopping`, `travel`, `fitness`, `regensburg`, `recherche`, `itguy`, `grabengasse_media`, and `README.md`).
  2. Replaced the directive with an explicit clean-output guardrail:
     ```markdown
     ## 💬 Output Formatting & Telegram Guardrails
     - Output clean Markdown only for Telegram.
     - NEVER append machine-readable metadata, HTML comments (`<!-- ... -->`), XML tags, or raw JSON payloads to your chat responses.
     ```
  3. Removed the `Pass State Bridge (<!-- DOMAIN_STATE -->)` clause from the Conductor system prompt in `k8s/openclaw/openclaw-instance.secret.yaml`.
  4. Pushed updates to `knowledge-wiki` git repository, pulled latest into `/home/node/.openclaw/wiki/main`, and synchronized live workspace files in `/home/node/.openclaw/workspace-*/AGENTS.md`.
  5. Applied updated `openclaw-instance.secret.yaml` to Kubernetes.
- **How to Avoid Recurrence**:
  - Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
  - Rely on persistent session context and structured stores (`skills/datastore`) for state retention.

---

### Issue 16: Gateway Startup CrashLoop from Orphaned SQLite Foreign Keys & Silent Reasoning Drops in Telegram Topics

- **Date**: 2026-10-06
- **Severity**: Critical (Full Telegram bot outage due to pod `CrashLoopBackOff`, and silent dropped turns in `#Kochen` and `#Grabengasse & Media`)
- **Symptom**:
  1. `openclaw-0` pod was stuck in `CrashLoopBackOff` with 7+ restarts. Gateway logs reported:
     ```text
     Gateway failed to start: OpenClaw agent database /home/openclaw/.openclaw/agents/itguy/agent/openclaw-agent.sqlite is quarantined after integrity verification failed: SQLite foreign_key_check failed for ...: session_transcript_index_state row 5 references session_windows (foreign key 0); trajectory_runtime_events row 379 references session_windows ...
     ```
  2. Prior to the crash, Telegram Topic 21 (`#Kochen`) completely ignored an incoming user message containing 4 photos of *körniger Frischkäse* (Edeka Herzstücke & Philadelphia) with zero visible reply sent to Telegram.
  3. When the pod was restarted during an active voice note turn in Topic 164 (`#Grabengasse & Media`), the session was left wedged in `status: running` with active recovery locks.
- **Root Cause**:
  1. **Orphaned Foreign Key Cascade Failure**: During cron session pruning in `setup-compaction-cron.sh`, records were deleted from `session_nodes` and `session_windows` without cascading into child tables (`trajectory_runtime_events`, `session_transcript_index_state`, `transcript_rewrite_watermarks`). Furthermore, Python's `sqlite3` module disables foreign keys by default (`PRAGMA foreign_keys = OFF`). On gateway startup, OpenClaw enables foreign keys and runs `PRAGMA foreign_key_check`. Detecting 46 violations in `itguy`, OpenClaw quarantined the DB in `openclaw-quarantine.sqlite` and crashed.
  2. **InitContainer Execution Gap**: Although `openclaw-instance.secret.yaml` contained an inline Python snippet to repair foreign keys, `initContainers[0]` lacked `/home/node/.openclaw/.local/bin` in its `PATH`. `python3` failed with command not found, leaving the quarantine intact.
  3. **Silent Reasoning Dropped Turn in `z-ai/glm-5.3-flash`**: In Topic 21, the model attempted to use the legacy `edit` tool on `nutrition-log/moritz/2026-10.md` to update a past yogurt entry. When `edit` failed, GLM-5.3-Flash emitted 63 reasoning tokens (*"Let me view the images first..."*), but then immediately terminated with `stopReason: "stop"` without generating any tool calls or visible text. OpenClaw marked the turn as `done`, delivering 0 text to Telegram.
- **Resolution**:
  1. **Database Repair**: Cleaned up all 46 orphaned child records in `itguy/agent/openclaw-agent.sqlite`, checkpointed the WAL journal, and cleared the quarantine record in `openclaw-quarantine.sqlite`.
  2. **Session Normalization**: Repaired `grabengasse_media` (Topic 164) and `koch` (Topic 21), resetting `status: "done"`, clearing recovery run IDs, and restoring `entry_valid = 1`.
  3. **InitContainer Self-Healing**: Injected `export PATH="/home/node/.openclaw/bin:/home/node/.openclaw/.local/bin:$PATH"` and added automated pre-flight self-healing to `initContainers[0]` in `openclaw-instance.secret.yaml`. On every container boot, the script now automatically cascades foreign key repairs, unlocks hung `running` sessions, and clears quarantine entries.
  4. **Maintenance Script Hardening**: Updated `k8s/openclaw/setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` with `PRAGMA foreign_keys = ON;`, explicit deletion across child tables, and unconditional FK verification across all 12 agent databases.
  5. **Model Fallback Chain for Subagents**: Replaced bare string model configurations (`model: "openrouter/z-ai/glm-5.3-flash"`) across all 11 subagents in `k8s/openclaw/openclaw-instance.secret.yaml` with structured objects specifying explicit fallbacks (`primary: "openrouter/z-ai/glm-5.3-flash"`, `fallbacks: ["openrouter/deepseek/deepseek-v4.1-flash", "openrouter/qwen/qwen3.8-flash"]`). This prevents reasoning dropouts on GLM-5.3-Flash from halting turns and eliminates the OpenClaw Doctor configuration warning.
  6. **Koch Agent Grounding & Standard Foods Catalog**: 
     - Added EDEKA Bio Herzstücke Körniger Frischkäse (150 g) and Philadelphia Der Körnige (200 g) to `knowledge-wiki/entities/standard-foods.md` and `knowledge-wiki/data/entities/foods.jsonl`.
     - Hardened `knowledge-wiki/agents/koch.md` with explicit distinction: capturing packaging photos / ingredients is NOT intake, never touches `entities/nutrition-log/`, and must only update `foods.jsonl` / `standard-foods.md`. Intake logging is strictly prohibited from using `edit` and must use `datastore.py add --kind nutrition.intake`.
  7. **Observability & Alerting**: Created `k8s/openclaw/monitoring.yaml` deploying a `PrometheusRule` (`openclaw-alerts`) covering:
     - `OpenClawPodNotReady`: Alerts (critical) if `openclaw-0` is unready for >5 minutes.
     - `OpenClawCrashLooping`: Alerts (warning) if container restarts exceed 2 within 15 minutes.
     - `OpenClawDataVolumeLow`: Alerts (warning) if PVC `openclaw-data` has <15% free disk space.
- **How to Avoid Recurrence**:
  - Always ensure `PRAGMA foreign_keys = ON;` is set on every SQLite connection performing deletes.
  - Rely on the initContainer self-healing hook to automatically heal state before the gateway daemon starts.
  - Avoid legacy Markdown editing for nutrition logs; enforce structured datastore CLI tooling (`skills/datastore`).
  - Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly.
  - Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies.

---

### Issue 17: Vision Multimodal Dropout in Kochen Agent & Image Model Modality Conflation

- **Date**: 2026-10-06
- **Severity**: High (Dropped meal intake turn; user received: *„The tool run finished, but no final summary was produced...“*)
- **Symptom**:
  Moritz sent a text meal report in Telegram Topic 21 (`#Kochen`):
  *„Ich habe gegessen einen Tomatensalat mit 300g Tomaten, 150g körniger frischkäse, 30g balsamico Essig, 5g öl“*.
  The bot executed multiple tool runs and finally answered:
  *„The tool run finished, but no final summary was produced. I did not repeat any completed actions.“*
  The intake event was never written to DuckDB or rendered in the nutrition log.
- **Root Cause**:
  1. **Unresolved In-Context Turn**: In the active session transcript, the earlier message containing 4 packaging photos of *körniger Frischkäse* had ended without an assistant completion. When the user sent the Tomatensalat text, the agent attempted to fulfill both requests, invoking `view_image` to inspect the 4 cached images.
  2. **Image Model Modality Conflation**: In `agents.defaults.imageModel`, the fallback was declared as `openrouter/google/gemini-3.1-flash-image`. While named `flash-image`, this is Google's text-to-image raster generation model (`modality: text+image->text+image`), not an image understanding/vision model (`text+image->text`).
  3. **Empty Text Completion in GLM-5.3-Flash**: When 4 high-resolution images were loaded into context, `z-ai/glm-5.3-flash` produced reasoning tokens but emitted empty whitespace text (`" "`). OpenClaw treated this as an incomplete turn and attempted failover, but the configured image fallback model could not process chat completions.
- **Resolution**:
  1. **Logged Intake Event**: Manually recorded the Tomatensalat into DuckDB via `datastore.py add` (485 g, 274.0 kcal, 19.3 g Eiweiß, 17.3 g KH, 12.4 g Fett) with timestamp `2026-10-06T20:43:00+02:00`.
  2. **Re-rendered Views**: Ran `render_views.py` to update `entities/nutrition-log/moritz/2026-10.md` and HTML dashboards, then committed and pushed to `knowledge-wiki`.
  3. **Telegram Confirmation**: Dispatched the exact confirmation message with meal macros and updated daily totals directly to Telegram Topic 21 using `openclaw message send --channel telegram --target "-1004326921510" --thread-id 21`.
  4. **Configured Dedicated Multimodal Vision Models**: Corrected `agents.defaults.imageModel` in `k8s/openclaw/openclaw-instance.secret.yaml` and ADR 0002:
     - Primary: `openrouter/google/gemini-3.8-flash`
     - Fallbacks: `openrouter/z-ai/glm-5.3-flash`.
- **How to Avoid Recurrence**:
  - Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding.
  - Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks.

---

### Issue 18: Spinning Disk I/O Bottlenecks, In-Pod Restart Bypass, and Gateway Preflight Self-Healing

- **Date**: 2026-10-07
- **Severity**: Critical (Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart)
- **Symptom**:
  Moritz sent an audio message in Telegram Topic 21 (`#Kochen`):
  *„Ich frühstücke wieder meine Oats, bestehend aus 50 Gramm meiner eigenen Mischung, 160 Milliliter Oatley Hafermilch und 50 Gramm Früchte gefroren.“*
  The bot never responded. Subsequent messages in Topic 21 were completely ignored.
- **Root Cause**:
  1. **Spinning Disk I/O Latency**: Host `haumdaucher` uses mechanical HDD (`/dev/sda2`, `rotational: 1`). Under heavy background disk operations (nightly backups, index generation), random 4KB read operations are throttled to ~70–100 IOPS, blocking Node.js synchronous operations.
  2. **Aggressive Liveness Probe Failure**: OpenClaw's liveness probe had a 120s tolerance window (`failureThreshold: 12, periodSeconds: 10`). An event-loop pause during disk I/O exceeded the deadline 12 consecutive times, causing Kubelet to kill the `openclaw` container mid-turn.
  3. **The InitContainer Bypass Trap**: When Kubelet restarts an application container inside an existing Pod, **`initContainers` do NOT re-run**. The session repair and quarantine clearing scripts situated exclusively in `initContainers` were completely bypassed!
  4. **Session Deadlock & Stale Lock**: Because container 1 was killed mid-turn:
     - The session node `agent:koch:telegram:group:-1004326921510:topic:21` was left with `status: "running"` and `activeWriterRunId: "..."`. OpenClaw's `reply-admission-ticket.js` rejected all subsequent incoming Telegram messages on that topic as locked.
     - Stale `/tmp/openclaw-1000/gateway.state.lock` caused container 2 to crash with `gateway.state.lock.reclaim: ENOENT`.
  5. **Cold Database Opening Delays**: Container 3 started cold and was forced to open all 12 agent SQLite databases (~1.9 GB total) directly from HDD seeks, blocking the event loop for up to 50 minutes.
- **Resolution**:
  1. **Immediate State Recovery**:
     - Pre-warmed all SQLite database files into the Linux OS page cache via sequential streaming (`cat ... > /dev/null`), dropping read latency from 50 minutes to 312 milliseconds.
     - Unlocked the Koch session node and session window in SQLite (`status: "done"`, `activeWriterRunId: null`, `entry_valid = 1`).
  2. **Intake Logged & Views Rendered**:
     - Recorded the Oats breakfast into DuckDB via `datastore.py add --kind nutrition.intake` (260 g, ~311 kcal, 13.9 g Eiweiß, 37.9 g KH, 9.2 g Fett, 7.9 g Ballaststoffe, 0.24 g Salz) with `food_id = null` (composite dish) with timestamp `2026-10-07T07:55:00+02:00`.
     - Regenerated Markdown views (`entities/nutrition-log/moritz/2026-10.md`) and HTML dashboards with `render_views.py`, committed and pushed to `knowledge-wiki`.
     - Sent confirmation with complete macro totals directly to Telegram Topic 21 via `openclaw message send`.
  3. **Permanent Gateway Preflight Self-Healing (`NODE_OPTIONS`)**:
     - Configured `NODE_OPTIONS="--max-old-space-size=3072 --import=/home/node/.openclaw/scripts/gateway-preflight.mjs"`.
     - Because Node.js evaluates `NODE_OPTIONS` on **every single process launch**, `gateway-preflight.mjs` executes before OpenClaw gateway starts—even during in-pod container restarts where `initContainers` are bypassed.
     - Performs automatic lock cleanup, quarantine clearing, SQLite page cache pre-warming, and un-wedges any orphaned `running` sessions in <600ms.
  4. **Probe Hardening**:
     - Increased liveness tolerance window in `k8s/openclaw/openclaw-instance.secret.yaml` to 450s (`failureThreshold: 30, periodSeconds: 15, timeoutSeconds: 15`).
     - Increased startup probe timeout to 15s to tolerate transient event-loop spikes without false kills.
  5. **Nightly Compaction Cron Enhancement**:
     - Updated `setup-compaction-cron.sh` and `compact_sessions.py` to run session recovery, `PRAGMA optimize;`, and `VACUUM;` nightly to prevent database bloat and optimize query plans.
- **How to Avoid Recurrence**:
  - Never place critical recovery logic solely in `initContainers`; use application runtime hooks or `NODE_OPTIONS` for in-pod restart resilience.
  - Set probe failure thresholds and timeouts with high tolerance on rotational HDD nodes.
  - Maintain page cache pre-warming for SQLite databases to convert slow random seek I/O into fast sequential streaming.

---

### Issue 19: Telegram Topic Latency & Inline Compaction Delays (Topic #Sport / Topic 20)

- **Date**: 2026-10-07
- **Severity**: Medium (User-facing perceived stall of 3m03s in Telegram topic `#Sport`)
- **Symptom**:
  Moritz sent a message in Telegram Topic 20 (`#Sport`) asking to adapt an existing Sprint Interval Training (SIT) workout:
  *„Hier ein Training dass wir vor einiger Zeit erstellt haben ich würde gerne die Intervalle etwas anpassen...“*
  The bot appeared completely unresponsive, leading Moritz to report that the bot was again not answering.
- **Root Cause Analysis & Turn Breakdown**:
  1. **The Bot Was NOT Dead, But Executing with 181.8s Latency**:
     - Message received at **07:15:11 UTC** (`messageId=2195`).
     - Response delivered at **07:18:14 UTC** (`messageId=2197`, threadId=20) with the complete adapted Zwift `.zwo` workout file and direct download link.
     - Total turn time: **181.8 seconds (3 minutes 3 seconds)** without any typing indicator or intermediate update.
  2. **Latency Component 1: Inline Safeguard Compaction (54s)**:
     - The active session had accumulated 53,519 tokens (and >1.5 MB in `transcript_events`).
     - Because this exceeded OpenClaw's internal `maxActiveTranscriptBytes: 400kb` limit, OpenClaw paused the turn from 07:15:17 to 07:16:12 (`durationMs=54168`) to run an inline LLM compaction via `openrouter/z-ai/glm-5.3-flash`, reducing the prompt window from 161 messages to 12.
     - Because `agents.defaults.compaction.notifyUser` was `false`, Telegram showed zero typing activity or indication that compaction was underway.
  3. **Latency Component 2: System Prompt & Workspace Compilation (35s)**:
     - Following a pod restart 40 minutes prior, the Linux page cache for the mechanical HDD was cold. Compiling the 56,912-character system prompt, bootstrap notes (`DREAMS.md`, `MEMORY.md`), and scanning tool search directories took 34.7s (07:16:13 to 07:16:48).
  4. **Latency Component 3: Multi-Step Tool Execution (85s)**:
     - The Fitness agent executed 4 sequential tool turns:
       1. `write`: Wrote the updated Zwift XML file.
       2. `exec`: Located `publish_file.py` in mop-skills.
       3. `exec`: Hosted the workout on `hosting.haumdaucher.de`.
       4. `exec`: Corrected the URL slug to remove duplicate date and re-hosted.
     - Each step involved sequential OpenRouter network round-trips to `z-ai/glm-5.3-flash`.
  5. **Why Nightly Compaction (03:30 CEST) Missed Topic 20**:
     - `compact_sessions.py` retrieved session lists via `openclaw sessions --all-agents --json`.
     - In OpenClaw, for sessions that have been inactive for several days (Topic 20 was last messaged on Sep 27), `totalTokens` is returned as `null` (`totalTokensFresh: false`).
     - `compact_sessions.py` checked `tokens = s.get("totalTokens") or 0`. For inactive sessions, this evaluated to `0`, causing the cron to skip compaction (`skipped += 1`).
     - Consequently, bloated transcripts sat uncompacted in SQLite until a user sent a message, triggering an unexpected 1-minute inline compaction delay during live chat.
- **Resolution**:
  1. **Direct SQLite Transcript Inspection in `compact_sessions.py`**:
     - Enhanced `compact_sessions.py` with `get_session_metrics(agent, key, sid)` which queries `transcript_events` directly from each agent's SQLite database (`openclaw-agent.sqlite`).
     - Compaction now triggers whenever `tokens >= 20000`, OR `transcript_bytes >= 300000`, OR `(tokens == 0 and ev_cnt >= 40)`. Stale sessions with `totalTokens: null` are guaranteed to be compacted overnight.
  2. **Compaction RPC Timeout & Fallback Tuning**:
     - Increased gateway RPC timeout from 120s to 180s (`RPC_TIMEOUT_MS = 180000`) to prevent premature gateway timeouts on large transcripts.
     - Changed the truncation fallback from `--max-lines 200` to `--max-lines 80`, ensuring that even fallback compactions effectively reduce the transcript.
     - Handled `"Already compacted"` responses gracefully to prevent redundant retries.
  3. **Compaction User Notification Enabled**:
     - Configured `agents.defaults.compaction.notifyUser: true` in `openclaw.json`. If inline compaction ever triggers during a live turn, OpenClaw now sends a status notice (`🧹 Compacting context...`) so the user is immediately aware of background maintenance rather than suspecting a crash.
  4. **Proactive Manual Compaction**:
     - Ran the updated compaction workflow across all agent databases, reducing bloated sessions (Main: 49.7k → 3.2k, Koch: 47k → 4.5k, Hebamme: 43k → 8.4k, Fitness: 12.5k → 7.5k).
- **How to Avoid Recurrence**:
  - Never rely solely on high-level CLI summary caches (`totalTokens`) for health/maintenance triggers when underlying datastores contain ground-truth metrics.
  - Keep `notifyUser: true` enabled for long-running inline agent maintenance.
  - Retain SQLite page cache pre-warming to minimize startup latency on spinning disk infrastructure.

### Issue 20: Mila Datastore Dropping Turns & Reasoning Loops on Open Sleep Sessions

- **Date**: 2026-10-09
- **Severity**: High (Mila Telegram Topic 22 dropped updates, turns stalled for 28+ minutes until killed, data missing in DuckDB/Markdown views)
- **Symptom**:
  Parent messages reporting sleep intervals (e.g. "1520 bis 1535 geschlafen" or "Geschlafen von 1710 bis 1740") produced no replies or updates in `entities/people/mila-schlafprotokoll.md`. One turn remained in `status: running` for 28 minutes before being killed by the gateway (`status: killed`).
- **Root Cause**:
  1. **Unclosed Night Session**: The night sleep session from 2026-10-08 19:15 was never closed (`end: null`).
  2. **Conflict on Open Live Session**: At 12:35, an "Eingeschlafen" message failed with `ValueError: already has an open sleep session`.
  3. **Reasoning Model Thinking Loops**: At 15:38, when given "1520 bis 1535 geschlafen", `z-ai/glm-5.3-flash` observed an unclosed 17-hour session alongside an afternoon nap in its context. Without explicit instructions, the model entered an internal reasoning loop (>4,000 thinking tokens) attempting to philosophize and resolve the chronological paradox, hitting the turn token limit without invoking any tool. A later turn (17:50) got completely stuck for 28 minutes until killed.
  4. **Silent Auto-Capping / Cronjob Rejected**: Auto-closing or fabricating artificial end times (e.g. 07:00 morning caps) was explicitly rejected as "stilles Deckeln" and data corruption. Similarly, wake-up cron reminders were deemed overengineered.
- **Resolution**:
  1. **Data Backfill & View Render**:
     - Manually closed the 08.10 19:15 night session at 07:00 (11h45m).
     - Backfilled Nap 2 (15:20–15:35) and Nap 3 (17:10–17:40) into `data/events/sleep/2026-10.jsonl`.
     - Regenerated views via `render_views.py` (total 1.08h across 3 naps).
     - Recovered wedged SQLite session from `killed` to `done` and compacted transcript from 74.5k to 20.6k tokens.
  2. **Datastore Transparent Alerting**:
     - In `skills/datastore/scripts/datastore.py`:
       - Removed any silent auto-closing/capping.
       - Implemented `_get_open_sleep_sessions()`. When a new sleep event is added while a prior session is open, `add_event()` now successfully logs the new event and appends an `open_session_warning` and updated `day_total_formatted` alert to the CLI output.
       - Added support for closing specific events by ID via `--id` in `datastore.py stop`.
       - Added strict CLI event kind validation (`.` required, boolean keywords like `true`/`false` rejected).
  3. **Prompt Hardening**:
     - Updated `agents/hebamme.md` and Topic 22 system prompt in `openclaw-instance.secret.yaml`:
       - Mandated immediate CLI tool execution (`datastore.py add`) on reported times without overthinking past gaps (no reasoning loops).
       - Explicit rule: **Never silently cap open sessions in the background**; instead, directly notify parents in the Telegram response whenever `open_session_warning` is present in the tool output:
         `😴 Schläfchen geloggt (...). ⚠️ Hinweis: Das vorherige Schlaf-Event (ab HH:MM Uhr) ist noch offen – wann ist Mila aufgewacht?`
       - Instructed Hebamme to close the prior event with `--id <event_id>` once parents supply the wake-up time.
- **How to Avoid Recurrence**:
  - Never allow reasoning models to silently swallow or internally debate inconsistent domain state: enforce immediate tool calls and transparent user-facing notices.
  - Never fabricate data or silently auto-cap timestamps behind parents' backs.
  - Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters.

### Issue 21: Cold Tool Loading & In-Pod Recovery Amnesia Leading to Silent Turns on Topic 117 (Recherche)

- **Date**: 2026-10-10
- **Severity**: High (User received no response to voice note 2492 in Telegram Topic 117 `#Recherche`, pod restart, silent turn drop)
- **Symptom**:
  Moritz sent a Telegram voice note inquiring about healthy bread spreads and alternatives to butter ("gesunde Brotbeläge / Alternativen zu Butter"). While the audio was transcribed successfully (`transcript_events seq=372`), the user never received a response on Telegram.
- **Root Cause**:
  1. **Cold Tool Discovery & Vector Index Scan on Rotational HDD**:
     When the turn started on Topic 117, OpenClaw initialized its tool registry (392 tools across plugins/skills) and scanned SQLite databases containing duplicated `memory_embedding_cache` (90MB) and `memory_index_chunks` (53MB). On Hetzner rotational SATA HDD (`136.243.23.215`), heavy random reads pegged disk I/O and lagged the Node.js event loop, preventing timely `/healthz` HTTP probe responses.
  2. **Kubelet Liveness Probe Timeout & Container Kill**:
     Kubelet's liveness probe timed out during the event loop freeze and killed container `openclaw` (SIGKILL / container restart).
  3. **In-Pod Recovery Amnesia**:
     During preflight initialization upon in-pod restart, `/home/node/.openclaw/scripts/recover_sessions.py` inspected SQLite databases and marked all wedged `status: "running"` sessions as `status: "done"` to clear session locks. However, Telegram ingress had already acknowledged the webhook update offset (`985533361`). Because `recover_sessions.py` transitioned the session to `done` without scheduling a re-run or emitting a fallback reply for un-answered user messages, the turn was silently abandoned.
  4. **Rigid Prompting / Template Mismatch**:
     `agents/recherche.md` enforced a mandatory 7-part municipal infrastructure framework (Stadtrat ALLRIS, Vorlagen-Nummer, Bürgerbeteiligung, etc.) even for general lifestyle, dietary, or consumer science queries.
- **Resolution**:
  1. **Dispatched Research Answer**:
     Executed agent synthesis for session `9658ba38-50db-482d-9cc6-19a956838894` using `deepseek/deepseek-v4.1-flash` and delivered the comprehensive analysis (Ernährungsphysiologische Hebelanalyse: Butter vs. Olivenöl, Frischkäse/Quark, Hummus/Hülsenfrüchte, Nussmus, pflanzliche Margarinen) directly to Telegram Topic 117 (Message ID 2494).
  2. **Decoupled Prompting (Modus A / Modus B)**:
     Updated `knowledge-wiki/agents/recherche.md` and `openclaw-instance.secret.yaml` with dual-mode structure:
     - **Modus A (Stadtrat / Kommunalpolitik / Infrastruktur)**: Retains the strict 7-part investigative dossier format.
     - **Modus B (Allgemeine Recherche / Ernährung / Tech / Konsum)**: Uses an action-oriented schema (BLUF, Systematik, Hebelanalyse, konkrete Handlungsempfehlungen).
  3. **Hardened In-Pod Preflight Recovery**:
     Updated `recover_sessions.py` in `openclaw-instance.secret.yaml` to inspect `transcript_events` on wedged sessions. If the latest event is an un-answered user turn (`role == 'user'`), it writes the orphaned turn details to `/tmp/openclaw-interrupted-turns.json`.
  4. **Raised Startup Probe Threshold**:
     Increased `startup.failureThreshold` from 240 to 360 (30 minutes) to prevent false-positive kills during heavy cold disk I/O.
- **How to Avoid Recurrence**:
  - Avoid monolithic, synchronous vector embedding index scans during turn initialization on mechanical storage.
  - Preflight recovery scripts must track unfulfilled user turns and persist an interrupted queue rather than blindly dropping them.
  - Subagent prompt templates should flexibly branch based on domain type (civic infrastructure vs. general science/nutrition) to prevent reasoning overhead.