From bdcf6a8d187b452498012dbc7004d9ad4e7dc193 Mon Sep 17 00:00:00 2001 From: Moritz Graf Date: Mon, 28 Sep 2026 20:18:02 +0200 Subject: [PATCH] feat(openclaw): automate nightly multi-agent session compaction and document incident --- k8s/openclaw/AGENTS.md | 5 +- k8s/openclaw/TROUBLESHOOTING.md | 27 +++++++++ k8s/openclaw/setup-compaction-cron.sh | 87 +++++++++++++++++++++++++++ knowledge-wiki | 2 +- 4 files changed, 118 insertions(+), 3 deletions(-) create mode 100755 k8s/openclaw/setup-compaction-cron.sh diff --git a/k8s/openclaw/AGENTS.md b/k8s/openclaw/AGENTS.md index d819ad1..a2c9ec4 100644 --- a/k8s/openclaw/AGENTS.md +++ b/k8s/openclaw/AGENTS.md @@ -55,9 +55,10 @@ To perform modifications or rotate api keys: kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml ``` -4. **Register / Sync Automations**: Run `setup-itguy-cron.sh` to ensure the weekly health audit automation is registered in OpenClaw: +4. **Register / Sync Automations**: Run the automation setup scripts to register OpenClaw Cron tasks: ```bash - ./k8s/openclaw/setup-itguy-cron.sh + ./k8s/openclaw/setup-itguy-cron.sh # Weekly health audit (Sun 03:00) + ./k8s/openclaw/setup-compaction-cron.sh # Nightly multi-agent session compaction (Daily 03:30) ``` --- diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index a453dbb..1c0e20f 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -16,6 +16,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. | | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. | | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. | +| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. | --- @@ -140,3 +141,29 @@ This document captures fixed issues, operational quirks, and incident root-cause ``` - **How to Avoid Recurrence**: In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard. + +--- + +### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens) + +- **Date**: 2026-09-28 +- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB) +- **Symptom**: + 1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang: + ```text + Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal). + Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉 + ``` + 2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`). +- **Root Cause**: + 1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**. + 2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands. + 3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively. +- **Resolution**: + 1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**. + 2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens. + 3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`). + 4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation. +- **How to Avoid Recurrence**: + Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance. + diff --git a/k8s/openclaw/setup-compaction-cron.sh b/k8s/openclaw/setup-compaction-cron.sh new file mode 100755 index 0000000..e71c21e --- /dev/null +++ b/k8s/openclaw/setup-compaction-cron.sh @@ -0,0 +1,87 @@ +#!/usr/bin/env bash +# ============================================================================== +# setup-compaction-cron.sh +# Declaratively registers the Multi-Agent Nightly Session Compaction in OpenClaw. +# Schedule: Every night at 03:30 Europe/Berlin (cron: 30 3 * * *) +# ============================================================================== + +set -euo pipefail + +NAMESPACE="${NAMESPACE:-openclaw}" +POD_NAME="${POD_NAME:-openclaw-0}" +CONTAINER_NAME="${CONTAINER_NAME:-openclaw}" + +JOB_NAME="nightly-session-compaction" +CRON_SCHEDULE="30 3 * * *" +TIMEZONE="Europe/Berlin" + +echo "=== Registering Multi-Agent Nightly Session Compaction in OpenClaw ($POD_NAME) ===" + +# 1. Install or ensure python compaction script is present on PVC +echo "Ensuring /home/node/.openclaw/scripts/compact_sessions.py exists..." +kubectl exec -i -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- /bin/sh -c 'cat << "EOF" > /home/node/.openclaw/scripts/compact_sessions.py +#!/usr/bin/env python3 +""" +Nightly multi-agent session compaction script for OpenClaw. +Compacts all active forum topics and main sessions exceeding the token threshold. +""" +import subprocess +import json +import sys + +THRESHOLD_TOKENS = 25000 +MAX_LINES = 250 + +def main(): + print(f"=== Running OpenClaw Nightly Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens, maxLines: {MAX_LINES}) ===") + res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True) + if res.returncode != 0: + print(f"Failed to list sessions: {res.stderr}") + sys.exit(1) + + sessions = json.loads(res.stdout).get("sessions", []) + compacted = 0 + skipped = 0 + + for s in sessions: + key = s.get("key", "") + agent = s.get("agentId") or "main" + tokens = s.get("totalTokens") or 0 + + # Target main session and all telegram group topic sessions + if (":telegram:group" in key or key == "agent:main:main") and not key.endswith(":heartbeat"): + if tokens >= THRESHOLD_TOKENS: + print(f"Compacting {agent} ({tokens} tokens) -> {key}...") + c_res = subprocess.run( + ["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", str(MAX_LINES)], + capture_output=True, text=True + ) + if c_res.returncode == 0: + print(f" ✓ {c_res.stdout.strip()}") + compacted += 1 + else: + print(f" ✗ {c_res.stderr.strip()}") + else: + skipped += 1 + + print(f"Compaction complete: {compacted} compacted, {skipped} skipped.") + +if __name__ == "__main__": + main() +EOF +chmod +x /home/node/.openclaw/scripts/compact_sessions.py +' + +# 2. Register / Update the OpenClaw cron job +echo "Registering OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZONE)..." +kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \ + --name "Nightly Session Compaction" \ + --declaration-key "$JOB_NAME" \ + --description "Nightly multi-agent session compaction for all active topics and main sessions" \ + --cron "$CRON_SCHEDULE" \ + --tz "$TIMEZONE" \ + --session "isolated" \ + --command "python3 /home/node/.openclaw/scripts/compact_sessions.py" + +echo "=== Compaction Cron automation successfully registered! ===" +kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron show "$JOB_NAME" 2>/dev/null || true diff --git a/knowledge-wiki b/knowledge-wiki index 2772e1b..fac1e59 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 2772e1b4b864149f29952f11ce119804ae574029 +Subproject commit fac1e592235989fa46ef64ff2ae0964699bb1f37