fix(openclaw): handle interrupted turns in preflight recovery and decouple recherche prompts

k8s/openclaw/TROUBLESHOOTING.md

@@ -30,6 +30,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. |
 | **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. |
 | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
+| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis (Butter vs. Pflanzenaufstriche), decoupled `recherche.md` prompt into Modus A/B, raised startup probe threshold to 360 (30m), and updated `recover_sessions.py` to detect interrupted user turns. |
 
 
 ---
@@ -665,3 +666,34 @@ This document captures fixed issues, operational quirks, and incident root-cause
   - Never fabricate data or silently auto-cap timestamps behind parents' backs.
   - Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters.
 
+### Issue 21: Cold Tool Loading & In-Pod Recovery Amnesia Leading to Silent Turns on Topic 117 (Recherche)
+
+- **Date**: 2026-10-10
+- **Severity**: High (User received no response to voice note 2492 in Telegram Topic 117 `#Recherche`, pod restart, silent turn drop)
+- **Symptom**:
+  Moritz sent a Telegram voice note inquiring about healthy bread spreads and alternatives to butter ("gesunde Brotbeläge / Alternativen zu Butter"). While the audio was transcribed successfully (`transcript_events seq=372`), the user never received a response on Telegram.
+- **Root Cause**:
+  1. **Cold Tool Discovery & Vector Index Scan on Rotational HDD**:
+     When the turn started on Topic 117, OpenClaw initialized its tool registry (392 tools across plugins/skills) and scanned SQLite databases containing duplicated `memory_embedding_cache` (90MB) and `memory_index_chunks` (53MB). On Hetzner rotational SATA HDD (`136.243.23.215`), heavy random reads pegged disk I/O and lagged the Node.js event loop, preventing timely `/healthz` HTTP probe responses.
+  2. **Kubelet Liveness Probe Timeout & Container Kill**:
+     Kubelet's liveness probe timed out during the event loop freeze and killed container `openclaw` (SIGKILL / container restart).
+  3. **In-Pod Recovery Amnesia**:
+     During preflight initialization upon in-pod restart, `/home/node/.openclaw/scripts/recover_sessions.py` inspected SQLite databases and marked all wedged `status: "running"` sessions as `status: "done"` to clear session locks. However, Telegram ingress had already acknowledged the webhook update offset (`985533361`). Because `recover_sessions.py` transitioned the session to `done` without scheduling a re-run or emitting a fallback reply for un-answered user messages, the turn was silently abandoned.
+  4. **Rigid Prompting / Template Mismatch**:
+     `agents/recherche.md` enforced a mandatory 7-part municipal infrastructure framework (Stadtrat ALLRIS, Vorlagen-Nummer, Bürgerbeteiligung, etc.) even for general lifestyle, dietary, or consumer science queries.
+- **Resolution**:
+  1. **Dispatched Research Answer**:
+     Executed agent synthesis for session `9658ba38-50db-482d-9cc6-19a956838894` using `deepseek/deepseek-v4.1-flash` and delivered the comprehensive analysis (Ernährungsphysiologische Hebelanalyse: Butter vs. Olivenöl, Frischkäse/Quark, Hummus/Hülsenfrüchte, Nussmus, pflanzliche Margarinen) directly to Telegram Topic 117 (Message ID 2494).
+  2. **Decoupled Prompting (Modus A / Modus B)**:
+     Updated `knowledge-wiki/agents/recherche.md` and `openclaw-instance.secret.yaml` with dual-mode structure:
+     - **Modus A (Stadtrat / Kommunalpolitik / Infrastruktur)**: Retains the strict 7-part investigative dossier format.
+     - **Modus B (Allgemeine Recherche / Ernährung / Tech / Konsum)**: Uses an action-oriented schema (BLUF, Systematik, Hebelanalyse, konkrete Handlungsempfehlungen).
+  3. **Hardened In-Pod Preflight Recovery**:
+     Updated `recover_sessions.py` in `openclaw-instance.secret.yaml` to inspect `transcript_events` on wedged sessions. If the latest event is an un-answered user turn (`role == 'user'`), it writes the orphaned turn details to `/tmp/openclaw-interrupted-turns.json`.
+  4. **Raised Startup Probe Threshold**:
+     Increased `startup.failureThreshold` from 240 to 360 (30 minutes) to prevent false-positive kills during heavy cold disk I/O.
+- **How to Avoid Recurrence**:
+  - Avoid monolithic, synchronous vector embedding index scans during turn initialization on mechanical storage.
+  - Preflight recovery scripts must track unfulfilled user turns and persist an interrupted queue rather than blindly dropping them.
+  - Subagent prompt templates should flexibly branch based on domain type (civic infrastructure vs. general science/nutrition) to prevent reasoning overhead.
+