diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index fb4cb7a..69f1bda 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -29,6 +29,8 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. | | **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. | | **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. | +| **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. | + --- @@ -629,3 +631,37 @@ This document captures fixed issues, operational quirks, and incident root-cause - Keep `notifyUser: true` enabled for long-running inline agent maintenance. - Retain SQLite page cache pre-warming to minimize startup latency on spinning disk infrastructure. +### Issue 20: Mila Datastore Dropping Turns & Reasoning Loops on Open Sleep Sessions + +- **Date**: 2026-10-09 +- **Severity**: High (Mila Telegram Topic 22 dropped updates, turns stalled for 28+ minutes until killed, data missing in DuckDB/Markdown views) +- **Symptom**: + Parent messages reporting sleep intervals (e.g. "1520 bis 1535 geschlafen" or "Geschlafen von 1710 bis 1740") produced no replies or updates in `entities/people/mila-schlafprotokoll.md`. One turn remained in `status: running` for 28 minutes before being killed by the gateway (`status: killed`). +- **Root Cause**: + 1. **Unclosed Night Session**: The night sleep session from 2026-10-08 19:15 was never closed (`end: null`). + 2. **Conflict on Open Live Session**: At 12:35, an "Eingeschlafen" message failed with `ValueError: already has an open sleep session`. + 3. **Reasoning Model Thinking Loops**: At 15:38, when given "1520 bis 1535 geschlafen", `z-ai/glm-5.3-flash` observed an unclosed 17-hour session alongside an afternoon nap in its context. Without explicit instructions, the model entered an internal reasoning loop (>4,000 thinking tokens) attempting to philosophize and resolve the chronological paradox, hitting the turn token limit without invoking any tool. A later turn (17:50) got completely stuck for 28 minutes until killed. + 4. **Silent Auto-Capping / Cronjob Rejected**: Auto-closing or fabricating artificial end times (e.g. 07:00 morning caps) was explicitly rejected as "stilles Deckeln" and data corruption. Similarly, wake-up cron reminders were deemed overengineered. +- **Resolution**: + 1. **Data Backfill & View Render**: + - Manually closed the 08.10 19:15 night session at 07:00 (11h45m). + - Backfilled Nap 2 (15:20–15:35) and Nap 3 (17:10–17:40) into `data/events/sleep/2026-10.jsonl`. + - Regenerated views via `render_views.py` (total 1.08h across 3 naps). + - Recovered wedged SQLite session from `killed` to `done` and compacted transcript from 74.5k to 20.6k tokens. + 2. **Datastore Transparent Alerting**: + - In `skills/datastore/scripts/datastore.py`: + - Removed any silent auto-closing/capping. + - Implemented `_get_open_sleep_sessions()`. When a new sleep event is added while a prior session is open, `add_event()` now successfully logs the new event and appends an `open_session_warning` and updated `day_total_formatted` alert to the CLI output. + - Added support for closing specific events by ID via `--id` in `datastore.py stop`. + - Added strict CLI event kind validation (`.` required, boolean keywords like `true`/`false` rejected). + 3. **Prompt Hardening**: + - Updated `agents/hebamme.md` and Topic 22 system prompt in `openclaw-instance.secret.yaml`: + - Mandated immediate CLI tool execution (`datastore.py add`) on reported times without overthinking past gaps (no reasoning loops). + - Explicit rule: **Never silently cap open sessions in the background**; instead, directly notify parents in the Telegram response whenever `open_session_warning` is present in the tool output: + `😴 Schläfchen geloggt (...). ⚠️ Hinweis: Das vorherige Schlaf-Event (ab HH:MM Uhr) ist noch offen – wann ist Mila aufgewacht?` + - Instructed Hebamme to close the prior event with `--id ` once parents supply the wake-up time. +- **How to Avoid Recurrence**: + - Never allow reasoning models to silently swallow or internally debate inconsistent domain state: enforce immediate tool calls and transparent user-facing notices. + - Never fabricate data or silently auto-cap timestamps behind parents' backs. + - Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters. +