feat(openclaw): automate nightly multi-agent session compaction and document incident

2026-09-28 20:18:02 +02:00
parent 613a19bcca
commit bdcf6a8d18
4 changed files with 118 additions and 3 deletions
@@ -55,9 +55,10 @@ To perform modifications or rotate api keys:
   kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml
   kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml
   ```
-4. **Register / Sync Automations**: Run `setup-itguy-cron.sh` to ensure the weekly health audit automation is registered in OpenClaw:
+4. **Register / Sync Automations**: Run the automation setup scripts to register OpenClaw Cron tasks:
   ```bash
-   ./k8s/openclaw/setup-itguy-cron.sh
+   ./k8s/openclaw/setup-itguy-cron.sh        # Weekly health audit (Sun 03:00)
+   ./k8s/openclaw/setup-compaction-cron.sh   # Nightly multi-agent session compaction (Daily 03:30)
   ```

 ---
@@ -16,6 +16,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. |
 | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
 | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
+| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |

 ---

@@ -140,3 +141,29 @@ This document captures fixed issues, operational quirks, and incident root-cause
  ```
 - **How to Avoid Recurrence**:
  In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard.
+
+---
+
+### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)
+
+- **Date**: 2026-09-28
+- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
+- **Symptom**:
+  1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang:
+     ```text
+     Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
+     Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
+     ```
+  2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`).
+- **Root Cause**:
+  1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**.
+  2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
+  3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively.
+- **Resolution**:
+  1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**.
+  2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
+  3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`).
+  4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation.
+- **How to Avoid Recurrence**:
+  Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.
+
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# ==============================================================================
+# setup-compaction-cron.sh
+# Declaratively registers the Multi-Agent Nightly Session Compaction in OpenClaw.
+# Schedule: Every night at 03:30 Europe/Berlin (cron: 30 3 * * *)
+# ==============================================================================
+
+set -euo pipefail
+
+NAMESPACE="${NAMESPACE:-openclaw}"
+POD_NAME="${POD_NAME:-openclaw-0}"
+CONTAINER_NAME="${CONTAINER_NAME:-openclaw}"
+
+JOB_NAME="nightly-session-compaction"
+CRON_SCHEDULE="30 3 * * *"
+TIMEZONE="Europe/Berlin"
+
+echo "=== Registering Multi-Agent Nightly Session Compaction in OpenClaw ($POD_NAME) ==="
+
+# 1. Install or ensure python compaction script is present on PVC
+echo "Ensuring /home/node/.openclaw/scripts/compact_sessions.py exists..."
+kubectl exec -i -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- /bin/sh -c 'cat << "EOF" > /home/node/.openclaw/scripts/compact_sessions.py
+#!/usr/bin/env python3
+"""
+Nightly multi-agent session compaction script for OpenClaw.
+Compacts all active forum topics and main sessions exceeding the token threshold.
+"""
+import subprocess
+import json
+import sys
+
+THRESHOLD_TOKENS = 25000
+MAX_LINES = 250
+
+def main():
+    print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, maxLines: {MAX_LINES}) ===")
+    res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
+    if res.returncode != 0:
+        print(f"Failed to list sessions: {res.stderr}")
+        sys.exit(1)
+        
+    sessions = json.loads(res.stdout).get("sessions", [])
+    compacted = 0
+    skipped = 0
+    
+    for s in sessions:
+        key = s.get("key", "")
+        agent = s.get("agentId") or "main"
+        tokens = s.get("totalTokens") or 0
+        
+        # Target main session and all telegram group topic sessions
+        if (":telegram:group" in key or key == "agent:main:main") and not key.endswith(":heartbeat"):
+            if tokens >= THRESHOLD_TOKENS:
+                print(f"Compacting {agent} ({tokens} tokens) -> {key}...")
+                c_res = subprocess.run(
+                    ["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", str(MAX_LINES)],
+                    capture_output=True, text=True
+                )
+                if c_res.returncode == 0:
+                    print(f"  ✓ {c_res.stdout.strip()}")
+                    compacted += 1
+                else:
+                    print(f"  ✗ {c_res.stderr.strip()}")
+            else:
+                skipped += 1
+                
+    print(f"Compaction complete: {compacted} compacted, {skipped} skipped.")
+
+if __name__ == "__main__":
+    main()
+EOF
+chmod +x /home/node/.openclaw/scripts/compact_sessions.py
+'
+
+# 2. Register / Update the OpenClaw cron job
+echo "Registering OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZONE)..."
+kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
+  --name "Nightly Session Compaction" \
+  --declaration-key "$JOB_NAME" \
+  --description "Nightly multi-agent session compaction for all active topics and main sessions" \
+  --cron "$CRON_SCHEDULE" \
+  --tz "$TIMEZONE" \
+  --session "isolated" \
+  --command "python3 /home/node/.openclaw/scripts/compact_sessions.py"
+
+echo "=== Compaction Cron automation successfully registered! ==="
+kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron show "$JOB_NAME" 2>/dev/null || true