fix(openclaw): switch multi-agent compaction to true LLM summarization and document split-turn drift
This commit is contained in:
Moritz Graf
@@ -18,6 +18,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
| **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
|
||||
| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |
|
||||
| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
|
||||
| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
|
||||
|
||||
---
|
||||
|
||||
@@ -198,4 +199,53 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`.
|
||||
- When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.
|
||||
|
||||
---
|
||||
|
||||
### Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction `--max-lines` Pitfall
|
||||
|
||||
- **Date**: 2026-09-29
|
||||
- **Severity**: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German)
|
||||
- **Symptom**:
|
||||
1. A user sent a voice message to the `#Einkaufen` (Shopping) topic asking for recommendations for a kitchen scale (`Küchenwaage`).
|
||||
2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German:
|
||||
```text
|
||||
Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel...
|
||||
```
|
||||
3. The user was concerned whether the system had been compromised or hacked.
|
||||
- **Root Cause**:
|
||||
1. **The `--max-lines` Compaction Flaw**:
|
||||
The nightly compaction cron script (`setup-compaction-cron.sh` / `compact_sessions.py`) invoked:
|
||||
```bash
|
||||
openclaw sessions compact <key> --agent <agent> --max-lines 250
|
||||
```
|
||||
According to the OpenClaw CLI specification, `--max-lines <count>` means:
|
||||
`Truncate to the last N transcript lines instead of LLM summarization`.
|
||||
If a session has $\le 250$ transcript lines, OpenClaw outputs `No compaction needed for session ...` and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached **38,000 to 95,000 tokens** while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (`shopping`, `recherche`, `grabengasse_media`, etc.).
|
||||
2. **Split-Turn Context Anchoring**:
|
||||
When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a `Turn Context (split turn)` block indicating unfinished work:
|
||||
```markdown
|
||||
Turn Context (split turn):
|
||||
### Original Request
|
||||
User korrigierte die Supermarkt-Liste der Wiki-Seite...
|
||||
### Next Steps
|
||||
1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen...
|
||||
3. Änderung gegenüber Nutzer bestätigen.
|
||||
```
|
||||
When the new voice note arrived at `seq=231`, the prompt presented to `z-ai/glm-5.3-flash` was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (`seq=232`) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding:
|
||||
```text
|
||||
"No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction."
|
||||
```
|
||||
It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German).
|
||||
- **Resolution**:
|
||||
1. **Remove `--max-lines` for True LLM Compaction**:
|
||||
Updated `setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` to remove `--max-lines`, invoking `openclaw sessions compact <key> --agent <agent>` so OpenClaw performs native LLM summarization.
|
||||
2. **Compacted Live Sessions**:
|
||||
Ran LLM compaction on `shopping:topic:115`, successfully shrinking it from **39,283 tokens down to 5,343 tokens** (an 86% context reclamation).
|
||||
3. **Event Loop Pacing**:
|
||||
Added `time.sleep(2)` between consecutive session compactions in `compact_sessions.py` to allow Node.js event loop ticks and prevent Kubelet `/healthz` liveness probe timeouts during batch compactions.
|
||||
- **How to Avoid Recurrence**:
|
||||
- Never use `--max-lines` when the objective is semantic context reduction; omit `--max-lines` to let OpenClaw invoke LLM summarization.
|
||||
- Monitor session token sizes (`openclaw sessions --all-agents --json`) rather than line counts.
|
||||
- Keep `setup-compaction-cron.sh` aligned across git and live PVCs.
|
||||
|
||||
|
||||
|
||||
@@ -28,12 +28,12 @@ Compacts all active forum topics and main sessions exceeding the token threshold
|
||||
import subprocess
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
|
||||
THRESHOLD_TOKENS = 25000
|
||||
MAX_LINES = 250
|
||||
|
||||
def main():
|
||||
print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, maxLines: {MAX_LINES}) ===")
|
||||
print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, LLM summarization) ===")
|
||||
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
|
||||
if res.returncode != 0:
|
||||
print(f"Failed to list sessions: {res.stderr}")
|
||||
@@ -48,12 +48,17 @@ def main():
|
||||
agent = s.get("agentId") or "main"
|
||||
tokens = s.get("totalTokens") or 0
|
||||
|
||||
# Target main session and all telegram group topic sessions
|
||||
if (":telegram:group" in key or key == "agent:main:main") and not key.endswith(":heartbeat"):
|
||||
# Target main sessions and all telegram sessions (topics and direct)
|
||||
is_target = (
|
||||
key == "agent:main:main" or
|
||||
":telegram:" in key or
|
||||
(key.startswith("agent:") and key.endswith(":main"))
|
||||
)
|
||||
if is_target and not key.endswith(":heartbeat"):
|
||||
if tokens >= THRESHOLD_TOKENS:
|
||||
print(f"Compacting {agent} ({tokens} tokens) -> {key}...")
|
||||
c_res = subprocess.run(
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", str(MAX_LINES)],
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if c_res.returncode == 0:
|
||||
@@ -61,6 +66,7 @@ def main():
|
||||
compacted += 1
|
||||
else:
|
||||
print(f" ✗ {c_res.stderr.strip()}")
|
||||
time.sleep(2)
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
|
||||
Reference in New Issue
Block a user