feat(openclaw): automate nightly multi-agent session compaction and document incident

2026-09-28 20:18:02 +02:00
parent 613a19bcca
commit bdcf6a8d18
4 changed files with 118 additions and 3 deletions
@@ -16,6 +16,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. |
 | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
 | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
+| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |

 ---

@@ -140,3 +141,29 @@ This document captures fixed issues, operational quirks, and incident root-cause
  ```
 - **How to Avoid Recurrence**:
  In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard.
+
+---
+
+### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)
+
+- **Date**: 2026-09-28
+- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
+- **Symptom**:
+  1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang:
+     ```text
+     Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
+     Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
+     ```
+  2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`).
+- **Root Cause**:
+  1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**.
+  2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
+  3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively.
+- **Resolution**:
+  1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**.
+  2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
+  3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`).
+  4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation.
+- **How to Avoid Recurrence**:
+  Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.
+