fix(openclaw): implement turn-replay daemon, vector dedup, 6Gi memory limit, and 25s probe timeouts

k8s/openclaw/TROUBLESHOOTING.md

@@ -30,7 +30,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. |
 | **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. |
 | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
-| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis (Butter vs. Pflanzenaufstriche), decoupled `recherche.md` prompt into Modus A/B, raised startup probe threshold to 360 (30m), and updated `recover_sessions.py` to detect interrupted user turns. |
+| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. |
 
 
 ---
@@ -688,12 +688,19 @@ This document captures fixed issues, operational quirks, and incident root-cause
      Updated `knowledge-wiki/agents/recherche.md` and `openclaw-instance.secret.yaml` with dual-mode structure:
      - **Modus A (Stadtrat / Kommunalpolitik / Infrastruktur)**: Retains the strict 7-part investigative dossier format.
      - **Modus B (Allgemeine Recherche / Ernährung / Tech / Konsum)**: Uses an action-oriented schema (BLUF, Systematik, Hebelanalyse, konkrete Handlungsempfehlungen).
-  3. **Hardened In-Pod Preflight Recovery**:
-     Updated `recover_sessions.py` in `openclaw-instance.secret.yaml` to inspect `transcript_events` on wedged sessions. If the latest event is an un-answered user turn (`role == 'user'`), it writes the orphaned turn details to `/tmp/openclaw-interrupted-turns.json`.
-  4. **Raised Startup Probe Threshold**:
-     Increased `startup.failureThreshold` from 240 to 360 (30 minutes) to prevent false-positive kills during heavy cold disk I/O.
+  3. **Automated Turn-Replay & Telegram Recovery Notice**:
+     - Hardened `/home/node/.openclaw/scripts/recover_sessions.py` to inspect `transcript_events` on wedged sessions. If the last event is an un-answered user prompt (`role == 'user'`), it extracts the prompt text and routing context, cascades deletion of the dangling user event, marks the session `done`, and queues the turn to `/tmp/openclaw-interrupted-turns.json`.
+     - Implemented `/home/node/.openclaw/scripts/replay_interrupted_turns.py`: waits for `/healthz`, dispatches a Telegram notification (`⚠️ OpenClaw wurde während der Bearbeitung neu gestartet – deine Anfrage wird jetzt nachgeholt...`), and replays the agent turn via `openclaw agent --session-key "<sk>" -m "<prompt>" --deliver`.
+     - Integrated into `gateway-preflight.mjs` as a background detached daemon (`child.unref()`).
+  4. **Vector Database Scoping & 94% Deduplication**:
+     - Scoped `memory.search.extraPaths` strictly to `agents.entries.main` (the personal concierge), removing it from global `memory.search`.
+     - Created `/home/node/.openclaw/scripts/dedup_subagent_vectors.py` in `gateway-preflight.mjs` to purge duplicate wiki vector tables (`memory_index_chunks_vec`, `memory_embedding_cache`, `memory_index_chunks`) and vacuum subagents.
+     - Reclaimed ~1.8 GB disk space across 10 subagent databases (shrinking them from ~185 MB each down to 5–15 MB), slashing random I/O and cold start times.
+  5. **Memory Limits & Probe Resilience**:
+     - Increased container memory limits in `openclaw-instance.secret.yaml` from 4Gi to 6Gi (requests 2Gi) and `NODE_OPTIONS` heap to `--max-old-space-size=4096`, eliminating Linux kernel cgroup OOMKills.
+     - Increased probe timeouts from 15s to 25s across `startup`, `liveness`, and `readiness` probes to accommodate mechanical HDD latency bursts without premature kills.
 - **How to Avoid Recurrence**:
-  - Avoid monolithic, synchronous vector embedding index scans during turn initialization on mechanical storage.
-  - Preflight recovery scripts must track unfulfilled user turns and persist an interrupted queue rather than blindly dropping them.
-  - Subagent prompt templates should flexibly branch based on domain type (civic infrastructure vs. general science/nutrition) to prevent reasoning overhead.
+  - Never configure broad vector embedding sync paths (`extraPaths`) at the global level in multi-agent environments; scope them strictly to the agents requiring them.
+  - Never let recovery scripts silently discard unfulfilled user turns without re-queuing and notifying the user.
+  - Generously tune probe timeouts and memory cgroup ceilings on nodes with ample headroom (32GB RAM) to survive temporary multi-process activity.