diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index 69f1bda..59b3eed 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -30,6 +30,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. | | **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. | | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. | +| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis (Butter vs. Pflanzenaufstriche), decoupled `recherche.md` prompt into Modus A/B, raised startup probe threshold to 360 (30m), and updated `recover_sessions.py` to detect interrupted user turns. | --- @@ -665,3 +666,34 @@ This document captures fixed issues, operational quirks, and incident root-cause - Never fabricate data or silently auto-cap timestamps behind parents' backs. - Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters. +### Issue 21: Cold Tool Loading & In-Pod Recovery Amnesia Leading to Silent Turns on Topic 117 (Recherche) + +- **Date**: 2026-10-10 +- **Severity**: High (User received no response to voice note 2492 in Telegram Topic 117 `#Recherche`, pod restart, silent turn drop) +- **Symptom**: + Moritz sent a Telegram voice note inquiring about healthy bread spreads and alternatives to butter ("gesunde Brotbeläge / Alternativen zu Butter"). While the audio was transcribed successfully (`transcript_events seq=372`), the user never received a response on Telegram. +- **Root Cause**: + 1. **Cold Tool Discovery & Vector Index Scan on Rotational HDD**: + When the turn started on Topic 117, OpenClaw initialized its tool registry (392 tools across plugins/skills) and scanned SQLite databases containing duplicated `memory_embedding_cache` (90MB) and `memory_index_chunks` (53MB). On Hetzner rotational SATA HDD (`136.243.23.215`), heavy random reads pegged disk I/O and lagged the Node.js event loop, preventing timely `/healthz` HTTP probe responses. + 2. **Kubelet Liveness Probe Timeout & Container Kill**: + Kubelet's liveness probe timed out during the event loop freeze and killed container `openclaw` (SIGKILL / container restart). + 3. **In-Pod Recovery Amnesia**: + During preflight initialization upon in-pod restart, `/home/node/.openclaw/scripts/recover_sessions.py` inspected SQLite databases and marked all wedged `status: "running"` sessions as `status: "done"` to clear session locks. However, Telegram ingress had already acknowledged the webhook update offset (`985533361`). Because `recover_sessions.py` transitioned the session to `done` without scheduling a re-run or emitting a fallback reply for un-answered user messages, the turn was silently abandoned. + 4. **Rigid Prompting / Template Mismatch**: + `agents/recherche.md` enforced a mandatory 7-part municipal infrastructure framework (Stadtrat ALLRIS, Vorlagen-Nummer, Bürgerbeteiligung, etc.) even for general lifestyle, dietary, or consumer science queries. +- **Resolution**: + 1. **Dispatched Research Answer**: + Executed agent synthesis for session `9658ba38-50db-482d-9cc6-19a956838894` using `deepseek/deepseek-v4.1-flash` and delivered the comprehensive analysis (Ernährungsphysiologische Hebelanalyse: Butter vs. Olivenöl, Frischkäse/Quark, Hummus/Hülsenfrüchte, Nussmus, pflanzliche Margarinen) directly to Telegram Topic 117 (Message ID 2494). + 2. **Decoupled Prompting (Modus A / Modus B)**: + Updated `knowledge-wiki/agents/recherche.md` and `openclaw-instance.secret.yaml` with dual-mode structure: + - **Modus A (Stadtrat / Kommunalpolitik / Infrastruktur)**: Retains the strict 7-part investigative dossier format. + - **Modus B (Allgemeine Recherche / Ernährung / Tech / Konsum)**: Uses an action-oriented schema (BLUF, Systematik, Hebelanalyse, konkrete Handlungsempfehlungen). + 3. **Hardened In-Pod Preflight Recovery**: + Updated `recover_sessions.py` in `openclaw-instance.secret.yaml` to inspect `transcript_events` on wedged sessions. If the latest event is an un-answered user turn (`role == 'user'`), it writes the orphaned turn details to `/tmp/openclaw-interrupted-turns.json`. + 4. **Raised Startup Probe Threshold**: + Increased `startup.failureThreshold` from 240 to 360 (30 minutes) to prevent false-positive kills during heavy cold disk I/O. +- **How to Avoid Recurrence**: + - Avoid monolithic, synchronous vector embedding index scans during turn initialization on mechanical storage. + - Preflight recovery scripts must track unfulfilled user turns and persist an interrupted queue rather than blindly dropping them. + - Subagent prompt templates should flexibly branch based on domain type (civic infrastructure vs. general science/nutrition) to prevent reasoning overhead. + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index 818f9e1..e2a8590 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/knowledge-wiki b/knowledge-wiki index 24081eb..83b51d3 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 24081eb7f447f7d7bd540dd1a3e57ea531edc939 +Subproject commit 83b51d36402f1e4d0affe11bab226fc26ff022de