diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index b27bc17..a1e9144 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -32,6 +32,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. | | **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. | | **2026-10-11** | **Automation & Garmin Sync** | Garmin Connect sync stopped updating overnight; cron job missing from OpenClaw; untracked activities folder | `RESOLVED` | Identified `garmin-nightly-mirror` was never registered in OpenClaw; created missing `activities` directory, hardened `setup-garmin-cron.sh` with JSON declaration lookup and `--announce`, registered automation (Daily 03:30 Berlin), and verified full sync through 2026-10-11. | +| **2026-10-12** | **Compaction Tuning & History Retention** | Session bloat during the day (~58k tokens) in Topic 21; tuning compaction strength to avoid binary choice between 0-token reset and bloat | `RESOLVED` | Configured `keepRecentTokens: 3000`, `recentTurnsPreserve: 2`, `maxActiveTranscriptBytes: 180kb`. Converted Tier A to deep-compaction with fallback, preserving conversational history while shrinking sessions from 30k-127k down to 1k-4k tokens. | --- @@ -833,3 +834,40 @@ This document captures fixed issues, operational quirks, and incident root-cause - Agents must enforce strict focus on the very last user message at the tail of the transcript. - Reset sessions periodically when context grows beyond 30k tokens for simple transactional logging tasks. +### Issue 26: Compaction Parameter Tuning & Shift to Deep-Compaction with History Retention + +- **Date**: 2026-10-12 +- **Severity**: Medium (Evaluating why sessions grew to ~58k tokens despite nightly maintenance; eliminating the binary choice between complete 0-token reset and bloated 58k-token sessions) +- **Symptom**: + Moritz inquired why Topic 21 had reached ~58k tokens and asked whether nightly maintenance was functioning properly. He requested investigating if compaction could be tuned to aggressively compact while preserving a few recent tokens/history for all agents instead of performing hard 0-token resets. +- **Root Cause Isolated**: + 1. **Nightly Job Execution Verified**: The nightly maintenance cron (`ce39bbf8-b83f-453e-be53-04e27db27b78`, schedule `45 4 * * * @ Europe/Berlin`) had indeed run successfully at 04:45:00 Berlin (`1791773100019 ms`). + 2. **Intraday Context Inflation**: The 58k tokens were NOT carried over from previous days; they were accumulated within the same day between 07:20 and 11:55 due to base prompt/tools (~18.2k baseline), 10 queries in Turn 1, multiple Perplexity web searches (~16 kB each), and `cat foods.jsonl`. + 3. **Overly Lenient Auto-Compaction Settings**: + - `keepRecentTokens: 15000`: Left 15k tokens verbatim in memory after compaction. + - `maxActiveTranscriptBytes: "400kb"`: Did not trigger inline safeguard compaction during the day until files reached 400 kB (~50k+ tokens). + - `recentTurnsPreserve: 3` (default): Preserved 3 full turns with potentially huge tool outputs verbatim. + 4. **Binary Tier A Trade-off**: Tier A agents (`koch`, `hebamme`, `shopping`) were previously hard-reset to 0 (`openclaw sessions delete`) because state is externalized in DuckDB. While preventing hallucinations, this discarded recent conversational continuity. +- **Resolution**: + 1. **Tuned Compaction Parameters**: + - Updated both live configuration (`/home/node/.openclaw/openclaw.json`) and declarative manifest (`k8s/openclaw/openclaw-instance.secret.yaml`): + - `keepRecentTokens: 3000` (down from 15,000; forces dense LLM summarization). + - `recentTurnsPreserve: 2` (preserves only the last 2 turns verbatim). + - `maxActiveTranscriptBytes: "180kb"` (down from 400kb; triggers proactive compaction before intraday sessions exceed ~25k tokens). + 2. **Switched Tier A from Hard-Reset to Deep-Compaction**: + - Updated `k8s/openclaw/setup-compaction-cron.sh` and `k8s/openclaw/scripts/compact_sessions.py`. + - Tier A now calls `openclaw sessions compact --agent ` with fallback to `--max-lines 40` if built-in safeguard summarization quality checks fail. + - Lowered Tier B thresholds: `THRESHOLD_TOKENS = 15000`, `THRESHOLD_TRANSCRIPT_BYTES = 200000` (200 KB), `THRESHOLD_EVENT_COUNT = 30`. + - Re-registered cron job in OpenClaw (`setup-compaction-cron.sh`). + 3. **Verified Live Reduction**: + - `koch`: 29,783 $\rightarrow$ **2,098 tokens** (Tier A deep-compacted). + - `hebamme`: Truncated to 40 lines (Tier A fallback). + - `fitness`: 101,950 $\rightarrow$ **5,318 tokens**. + - `recherche`: 127,356 $\rightarrow$ **4,323 tokens**. + - `grabengasse_media`: 49,447 $\rightarrow$ **3,356 tokens**. + - `doktor`: 20,259 $\rightarrow$ **1,079 tokens**. +- **How to Avoid Recurrence**: + - Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests. + - Rely on deep-compaction with fallback rather than hard-resets across all topic agents. + + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index 2742658..c562e21 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/k8s/openclaw/scripts/compact_sessions.py b/k8s/openclaw/scripts/compact_sessions.py index 2d93260..544bed5 100644 --- a/k8s/openclaw/scripts/compact_sessions.py +++ b/k8s/openclaw/scripts/compact_sessions.py @@ -17,15 +17,15 @@ import subprocess import sys import time -THRESHOLD_TOKENS = 20000 -THRESHOLD_TRANSCRIPT_BYTES = 300000 -THRESHOLD_EVENT_COUNT = 40 +THRESHOLD_TOKENS = 15000 +THRESHOLD_TRANSCRIPT_BYTES = 200000 +THRESHOLD_EVENT_COUNT = 30 CRON_MAX_AGE_HOURS = 48 RPC_TIMEOUT_MS = 180000 -# Tier A: Event-logging agents whose state is 100% externalized (DuckDB / JSONL / Groceries DB). -# Nightly reset to 0 prevents cross-day hallucination and eliminates LLM compaction token costs. -TIER_A_RESET_TARGETS = [ +# Tier A: High-frequency interaction topics (koch, hebamme, shopping). +# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory. +TIER_A_TARGETS = [ ("koch", "agent:koch:telegram:group:-1004326921510:topic:21"), ("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"), ("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"), @@ -117,9 +117,9 @@ def prune_stale_cron_sessions(): print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.") -def reset_tier_a_sessions(): - print("=== Tier A: Daily Hard-Reset to 0 (koch, hebamme, shopping) ===") - for agent, key in TIER_A_RESET_TARGETS: +def compact_tier_a_sessions(): + print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===") + for agent, key in TIER_A_TARGETS: db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite" if not os.path.exists(db_path): continue @@ -130,18 +130,32 @@ def reset_tier_a_sessions(): if not row or not row[0]: print(f" [Tier A] {agent} ({key}) already at 0 (no active session).") continue - print(f" [Tier A] Resetting {agent} ({key})...") + ev_cnt, ev_bytes = get_session_metrics(agent, key, row[0]) + if ev_cnt <= 2 and ev_bytes < 20000: + print(f" [Tier A] {agent} ({key}) has only {ev_cnt} events ({ev_bytes} bytes), skipping.") + continue + print(f" [Tier A] Deep-compacting {agent} ({key}) [{ev_cnt} events, {ev_bytes//1024}KB]...") r = subprocess.run( - ["openclaw", "sessions", "delete", key, "--agent", agent, "--yes"], - capture_output=True, text=True, timeout=60.0 + ["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)], + capture_output=True, text=True ) if r.returncode == 0: - print(f" ✓ Reset success: {r.stdout.strip()}") + print(f" ✓ Deep-compacted: {r.stdout.strip()}") + elif "Already compacted" in r.stderr or "Already compacted" in r.stdout: + print(f" ✓ Already compacted.") else: - print(f" ✗ Reset error: {r.stderr.strip()}") + print(f" ✗ Standard compact error: {r.stderr.strip()}; retrying with --max-lines 40...") + fb = subprocess.run( + ["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "40", "--timeout", str(RPC_TIMEOUT_MS)], + capture_output=True, text=True + ) + if fb.returncode == 0: + print(f" ✓ Truncation fallback success: {fb.stdout.strip()}") + else: + print(f" ✗ Truncation fallback failed: {fb.stderr.strip()}") except Exception as e: - print(f" ✗ Exception resetting {agent}: {e}") - time.sleep(1) + print(f" ✗ Exception deep-compacting {agent}: {e}") + time.sleep(2) def get_session_metrics(agent, key, sid=None): db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite" @@ -166,7 +180,7 @@ def get_session_metrics(agent, key, sid=None): return 0, 0 def compact_active_sessions(): - tier_a_keys = set(t[1] for t in TIER_A_RESET_TARGETS) + tier_a_keys = set(t[1] for t in TIER_A_TARGETS) print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===") res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True) if res.returncode != 0: @@ -266,7 +280,7 @@ def main(): except Exception: pass prune_stale_cron_sessions() - reset_tier_a_sessions() + compact_tier_a_sessions() compact_active_sessions() checkpoint_wal_journals() vacuum_and_optimize() diff --git a/k8s/openclaw/setup-compaction-cron.sh b/k8s/openclaw/setup-compaction-cron.sh index 04e2914..5e63ba6 100755 --- a/k8s/openclaw/setup-compaction-cron.sh +++ b/k8s/openclaw/setup-compaction-cron.sh @@ -5,8 +5,8 @@ # Schedule: Every night at 04:45 Europe/Berlin (02:45 UTC) (cron: 45 4 * * *) # # Architecture: -# - Tier A: Daily Hard-Reset to 0 for koch (Topic 21), hebamme (Topic 22), shopping (Topic 115) -# (100% externalized state in DuckDB/JSONL; resets prevent cross-day hallucination) +# - Tier A: Deep-Compaction for high-frequency topics (koch, hebamme, shopping) +# (maintains conversational continuity while keeping token count very low) # - Tier B: Smart Compaction for long-running project agents (itguy, recherche, regensburg, fitness, doktor) # - WAL checkpoints and SQLite optimization without blocking full VACUUM # ============================================================================== @@ -45,15 +45,15 @@ import subprocess import sys import time -THRESHOLD_TOKENS = 20000 -THRESHOLD_TRANSCRIPT_BYTES = 300000 -THRESHOLD_EVENT_COUNT = 40 +THRESHOLD_TOKENS = 15000 +THRESHOLD_TRANSCRIPT_BYTES = 200000 +THRESHOLD_EVENT_COUNT = 30 CRON_MAX_AGE_HOURS = 48 RPC_TIMEOUT_MS = 180000 -# Tier A: Event-logging agents whose state is 100% externalized (DuckDB / JSONL / Groceries DB). -# Nightly reset to 0 prevents cross-day hallucination and eliminates LLM compaction token costs. -TIER_A_RESET_TARGETS = [ +# Tier A: High-frequency interaction topics (koch, hebamme, shopping). +# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory. +TIER_A_TARGETS = [ ("koch", "agent:koch:telegram:group:-1004326921510:topic:21"), ("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"), ("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"), @@ -145,9 +145,9 @@ def prune_stale_cron_sessions(): print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.") -def reset_tier_a_sessions(): - print("=== Tier A: Daily Hard-Reset to 0 (koch, hebamme, shopping) ===") - for agent, key in TIER_A_RESET_TARGETS: +def compact_tier_a_sessions(): + print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===") + for agent, key in TIER_A_TARGETS: db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite" if not os.path.exists(db_path): continue @@ -158,18 +158,32 @@ def reset_tier_a_sessions(): if not row or not row[0]: print(f" [Tier A] {agent} ({key}) already at 0 (no active session).") continue - print(f" [Tier A] Resetting {agent} ({key})...") + ev_cnt, ev_bytes = get_session_metrics(agent, key, row[0]) + if ev_cnt <= 2 and ev_bytes < 20000: + print(f" [Tier A] {agent} ({key}) has only {ev_cnt} events ({ev_bytes} bytes), skipping.") + continue + print(f" [Tier A] Deep-compacting {agent} ({key}) [{ev_cnt} events, {ev_bytes//1024}KB]...") r = subprocess.run( - ["openclaw", "sessions", "delete", key, "--agent", agent, "--yes"], - capture_output=True, text=True, timeout=60.0 + ["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)], + capture_output=True, text=True ) if r.returncode == 0: - print(f" ✓ Reset success: {r.stdout.strip()}") + print(f" ✓ Deep-compacted: {r.stdout.strip()}") + elif "Already compacted" in r.stderr or "Already compacted" in r.stdout: + print(f" ✓ Already compacted.") else: - print(f" ✗ Reset error: {r.stderr.strip()}") + print(f" ✗ Standard compact error: {r.stderr.strip()}; retrying with --max-lines 40...") + fb = subprocess.run( + ["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "40", "--timeout", str(RPC_TIMEOUT_MS)], + capture_output=True, text=True + ) + if fb.returncode == 0: + print(f" ✓ Truncation fallback success: {fb.stdout.strip()}") + else: + print(f" ✗ Truncation fallback failed: {fb.stderr.strip()}") except Exception as e: - print(f" ✗ Exception resetting {agent}: {e}") - time.sleep(1) + print(f" ✗ Exception deep-compacting {agent}: {e}") + time.sleep(2) def get_session_metrics(agent, key, sid=None): db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite" @@ -194,7 +208,7 @@ def get_session_metrics(agent, key, sid=None): return 0, 0 def compact_active_sessions(): - tier_a_keys = set(t[1] for t in TIER_A_RESET_TARGETS) + tier_a_keys = set(t[1] for t in TIER_A_TARGETS) print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===") res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True) if res.returncode != 0: @@ -294,7 +308,7 @@ def main(): except Exception: pass prune_stale_cron_sessions() - reset_tier_a_sessions() + compact_tier_a_sessions() compact_active_sessions() checkpoint_wal_journals() vacuum_and_optimize()