docs(troubleshooting): document issue 20 on mila datastore reasoning loops
This commit is contained in:
@@ -29,6 +29,8 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
| **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. |
|
||||
| **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. |
|
||||
| **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. |
|
||||
| **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
|
||||
|
||||
|
||||
---
|
||||
|
||||
@@ -629,3 +631,37 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- Keep `notifyUser: true` enabled for long-running inline agent maintenance.
|
||||
- Retain SQLite page cache pre-warming to minimize startup latency on spinning disk infrastructure.
|
||||
|
||||
### Issue 20: Mila Datastore Dropping Turns & Reasoning Loops on Open Sleep Sessions
|
||||
|
||||
- **Date**: 2026-10-09
|
||||
- **Severity**: High (Mila Telegram Topic 22 dropped updates, turns stalled for 28+ minutes until killed, data missing in DuckDB/Markdown views)
|
||||
- **Symptom**:
|
||||
Parent messages reporting sleep intervals (e.g. "1520 bis 1535 geschlafen" or "Geschlafen von 1710 bis 1740") produced no replies or updates in `entities/people/mila-schlafprotokoll.md`. One turn remained in `status: running` for 28 minutes before being killed by the gateway (`status: killed`).
|
||||
- **Root Cause**:
|
||||
1. **Unclosed Night Session**: The night sleep session from 2026-10-08 19:15 was never closed (`end: null`).
|
||||
2. **Conflict on Open Live Session**: At 12:35, an "Eingeschlafen" message failed with `ValueError: already has an open sleep session`.
|
||||
3. **Reasoning Model Thinking Loops**: At 15:38, when given "1520 bis 1535 geschlafen", `z-ai/glm-5.3-flash` observed an unclosed 17-hour session alongside an afternoon nap in its context. Without explicit instructions, the model entered an internal reasoning loop (>4,000 thinking tokens) attempting to philosophize and resolve the chronological paradox, hitting the turn token limit without invoking any tool. A later turn (17:50) got completely stuck for 28 minutes until killed.
|
||||
4. **Silent Auto-Capping / Cronjob Rejected**: Auto-closing or fabricating artificial end times (e.g. 07:00 morning caps) was explicitly rejected as "stilles Deckeln" and data corruption. Similarly, wake-up cron reminders were deemed overengineered.
|
||||
- **Resolution**:
|
||||
1. **Data Backfill & View Render**:
|
||||
- Manually closed the 08.10 19:15 night session at 07:00 (11h45m).
|
||||
- Backfilled Nap 2 (15:20–15:35) and Nap 3 (17:10–17:40) into `data/events/sleep/2026-10.jsonl`.
|
||||
- Regenerated views via `render_views.py` (total 1.08h across 3 naps).
|
||||
- Recovered wedged SQLite session from `killed` to `done` and compacted transcript from 74.5k to 20.6k tokens.
|
||||
2. **Datastore Transparent Alerting**:
|
||||
- In `skills/datastore/scripts/datastore.py`:
|
||||
- Removed any silent auto-closing/capping.
|
||||
- Implemented `_get_open_sleep_sessions()`. When a new sleep event is added while a prior session is open, `add_event()` now successfully logs the new event and appends an `open_session_warning` and updated `day_total_formatted` alert to the CLI output.
|
||||
- Added support for closing specific events by ID via `--id` in `datastore.py stop`.
|
||||
- Added strict CLI event kind validation (`.` required, boolean keywords like `true`/`false` rejected).
|
||||
3. **Prompt Hardening**:
|
||||
- Updated `agents/hebamme.md` and Topic 22 system prompt in `openclaw-instance.secret.yaml`:
|
||||
- Mandated immediate CLI tool execution (`datastore.py add`) on reported times without overthinking past gaps (no reasoning loops).
|
||||
- Explicit rule: **Never silently cap open sessions in the background**; instead, directly notify parents in the Telegram response whenever `open_session_warning` is present in the tool output:
|
||||
`😴 Schläfchen geloggt (...). ⚠️ Hinweis: Das vorherige Schlaf-Event (ab HH:MM Uhr) ist noch offen – wann ist Mila aufgewacht?`
|
||||
- Instructed Hebamme to close the prior event with `--id <event_id>` once parents supply the wake-up time.
|
||||
- **How to Avoid Recurrence**:
|
||||
- Never allow reasoning models to silently swallow or internally debate inconsistent domain state: enforce immediate tool calls and transparent user-facing notices.
|
||||
- Never fabricate data or silently auto-cap timestamps behind parents' backs.
|
||||
- Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user