From 61ac5baa6f25b49618ce049eb89cb828ab0e9ce3 Mon Sep 17 00:00:00 2001 From: Moritz Graf Date: Tue, 29 Sep 2026 08:48:15 +0200 Subject: [PATCH] fix(openclaw): switch multi-agent compaction to true LLM summarization and document split-turn drift --- k8s/openclaw/TROUBLESHOOTING.md | 50 +++++++++++++++++++++++++++ k8s/openclaw/setup-compaction-cron.sh | 16 ++++++--- 2 files changed, 61 insertions(+), 5 deletions(-) diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index edb8050..e369818 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -18,6 +18,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. | | **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. | | **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. | +| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. | --- @@ -198,4 +199,53 @@ This document captures fixed issues, operational quirks, and incident root-cause - Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`. - When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage. +--- + +### Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction `--max-lines` Pitfall + +- **Date**: 2026-09-29 +- **Severity**: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German) +- **Symptom**: + 1. A user sent a voice message to the `#Einkaufen` (Shopping) topic asking for recommendations for a kitchen scale (`Küchenwaage`). + 2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German: + ```text + Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel... + ``` + 3. The user was concerned whether the system had been compromised or hacked. +- **Root Cause**: + 1. **The `--max-lines` Compaction Flaw**: + The nightly compaction cron script (`setup-compaction-cron.sh` / `compact_sessions.py`) invoked: + ```bash + openclaw sessions compact --agent --max-lines 250 + ``` + According to the OpenClaw CLI specification, `--max-lines ` means: + `Truncate to the last N transcript lines instead of LLM summarization`. + If a session has $\le 250$ transcript lines, OpenClaw outputs `No compaction needed for session ...` and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached **38,000 to 95,000 tokens** while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (`shopping`, `recherche`, `grabengasse_media`, etc.). + 2. **Split-Turn Context Anchoring**: + When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a `Turn Context (split turn)` block indicating unfinished work: + ```markdown + Turn Context (split turn): + ### Original Request + User korrigierte die Supermarkt-Liste der Wiki-Seite... + ### Next Steps + 1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen... + 3. Änderung gegenüber Nutzer bestätigen. + ``` + When the new voice note arrived at `seq=231`, the prompt presented to `z-ai/glm-5.3-flash` was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (`seq=232`) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding: + ```text + "No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction." + ``` + It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German). +- **Resolution**: + 1. **Remove `--max-lines` for True LLM Compaction**: + Updated `setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` to remove `--max-lines`, invoking `openclaw sessions compact --agent ` so OpenClaw performs native LLM summarization. + 2. **Compacted Live Sessions**: + Ran LLM compaction on `shopping:topic:115`, successfully shrinking it from **39,283 tokens down to 5,343 tokens** (an 86% context reclamation). + 3. **Event Loop Pacing**: + Added `time.sleep(2)` between consecutive session compactions in `compact_sessions.py` to allow Node.js event loop ticks and prevent Kubelet `/healthz` liveness probe timeouts during batch compactions. +- **How to Avoid Recurrence**: + - Never use `--max-lines` when the objective is semantic context reduction; omit `--max-lines` to let OpenClaw invoke LLM summarization. + - Monitor session token sizes (`openclaw sessions --all-agents --json`) rather than line counts. + - Keep `setup-compaction-cron.sh` aligned across git and live PVCs. + diff --git a/k8s/openclaw/setup-compaction-cron.sh b/k8s/openclaw/setup-compaction-cron.sh index e71c21e..98020ad 100755 --- a/k8s/openclaw/setup-compaction-cron.sh +++ b/k8s/openclaw/setup-compaction-cron.sh @@ -28,12 +28,12 @@ Compacts all active forum topics and main sessions exceeding the token threshold import subprocess import json import sys +import time THRESHOLD_TOKENS = 25000 -MAX_LINES = 250 def main(): - print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, maxLines: {MAX_LINES}) ===") + print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, LLM summarization) ===") res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True) if res.returncode != 0: print(f"Failed to list sessions: {res.stderr}") @@ -48,12 +48,17 @@ def main(): agent = s.get("agentId") or "main" tokens = s.get("totalTokens") or 0 - # Target main session and all telegram group topic sessions - if (":telegram:group" in key or key == "agent:main:main") and not key.endswith(":heartbeat"): + # Target main sessions and all telegram sessions (topics and direct) + is_target = ( + key == "agent:main:main" or + ":telegram:" in key or + (key.startswith("agent:") and key.endswith(":main")) + ) + if is_target and not key.endswith(":heartbeat"): if tokens >= THRESHOLD_TOKENS: print(f"Compacting {agent} ({tokens} tokens) -> {key}...") c_res = subprocess.run( - ["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", str(MAX_LINES)], + ["openclaw", "sessions", "compact", key, "--agent", agent], capture_output=True, text=True ) if c_res.returncode == 0: @@ -61,6 +66,7 @@ def main(): compacted += 1 else: print(f" ✗ {c_res.stderr.strip()}") + time.sleep(2) else: skipped += 1