feat(openclaw): tune compaction to keepRecentTokens 3k and convert Tier A to deep-compaction
This commit is contained in:
Moritz Graf
@@ -32,6 +32,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
| **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
|
||||
| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. |
|
||||
| **2026-10-11** | **Automation & Garmin Sync** | Garmin Connect sync stopped updating overnight; cron job missing from OpenClaw; untracked activities folder | `RESOLVED` | Identified `garmin-nightly-mirror` was never registered in OpenClaw; created missing `activities` directory, hardened `setup-garmin-cron.sh` with JSON declaration lookup and `--announce`, registered automation (Daily 03:30 Berlin), and verified full sync through 2026-10-11. |
|
||||
| **2026-10-12** | **Compaction Tuning & History Retention** | Session bloat during the day (~58k tokens) in Topic 21; tuning compaction strength to avoid binary choice between 0-token reset and bloat | `RESOLVED` | Configured `keepRecentTokens: 3000`, `recentTurnsPreserve: 2`, `maxActiveTranscriptBytes: 180kb`. Converted Tier A to deep-compaction with fallback, preserving conversational history while shrinking sessions from 30k-127k down to 1k-4k tokens. |
|
||||
|
||||
|
||||
---
|
||||
@@ -833,3 +834,40 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- Agents must enforce strict focus on the very last user message at the tail of the transcript.
|
||||
- Reset sessions periodically when context grows beyond 30k tokens for simple transactional logging tasks.
|
||||
|
||||
### Issue 26: Compaction Parameter Tuning & Shift to Deep-Compaction with History Retention
|
||||
|
||||
- **Date**: 2026-10-12
|
||||
- **Severity**: Medium (Evaluating why sessions grew to ~58k tokens despite nightly maintenance; eliminating the binary choice between complete 0-token reset and bloated 58k-token sessions)
|
||||
- **Symptom**:
|
||||
Moritz inquired why Topic 21 had reached ~58k tokens and asked whether nightly maintenance was functioning properly. He requested investigating if compaction could be tuned to aggressively compact while preserving a few recent tokens/history for all agents instead of performing hard 0-token resets.
|
||||
- **Root Cause Isolated**:
|
||||
1. **Nightly Job Execution Verified**: The nightly maintenance cron (`ce39bbf8-b83f-453e-be53-04e27db27b78`, schedule `45 4 * * * @ Europe/Berlin`) had indeed run successfully at 04:45:00 Berlin (`1791773100019 ms`).
|
||||
2. **Intraday Context Inflation**: The 58k tokens were NOT carried over from previous days; they were accumulated within the same day between 07:20 and 11:55 due to base prompt/tools (~18.2k baseline), 10 queries in Turn 1, multiple Perplexity web searches (~16 kB each), and `cat foods.jsonl`.
|
||||
3. **Overly Lenient Auto-Compaction Settings**:
|
||||
- `keepRecentTokens: 15000`: Left 15k tokens verbatim in memory after compaction.
|
||||
- `maxActiveTranscriptBytes: "400kb"`: Did not trigger inline safeguard compaction during the day until files reached 400 kB (~50k+ tokens).
|
||||
- `recentTurnsPreserve: 3` (default): Preserved 3 full turns with potentially huge tool outputs verbatim.
|
||||
4. **Binary Tier A Trade-off**: Tier A agents (`koch`, `hebamme`, `shopping`) were previously hard-reset to 0 (`openclaw sessions delete`) because state is externalized in DuckDB. While preventing hallucinations, this discarded recent conversational continuity.
|
||||
- **Resolution**:
|
||||
1. **Tuned Compaction Parameters**:
|
||||
- Updated both live configuration (`/home/node/.openclaw/openclaw.json`) and declarative manifest (`k8s/openclaw/openclaw-instance.secret.yaml`):
|
||||
- `keepRecentTokens: 3000` (down from 15,000; forces dense LLM summarization).
|
||||
- `recentTurnsPreserve: 2` (preserves only the last 2 turns verbatim).
|
||||
- `maxActiveTranscriptBytes: "180kb"` (down from 400kb; triggers proactive compaction before intraday sessions exceed ~25k tokens).
|
||||
2. **Switched Tier A from Hard-Reset to Deep-Compaction**:
|
||||
- Updated `k8s/openclaw/setup-compaction-cron.sh` and `k8s/openclaw/scripts/compact_sessions.py`.
|
||||
- Tier A now calls `openclaw sessions compact <key> --agent <agent>` with fallback to `--max-lines 40` if built-in safeguard summarization quality checks fail.
|
||||
- Lowered Tier B thresholds: `THRESHOLD_TOKENS = 15000`, `THRESHOLD_TRANSCRIPT_BYTES = 200000` (200 KB), `THRESHOLD_EVENT_COUNT = 30`.
|
||||
- Re-registered cron job in OpenClaw (`setup-compaction-cron.sh`).
|
||||
3. **Verified Live Reduction**:
|
||||
- `koch`: 29,783 $\rightarrow$ **2,098 tokens** (Tier A deep-compacted).
|
||||
- `hebamme`: Truncated to 40 lines (Tier A fallback).
|
||||
- `fitness`: 101,950 $\rightarrow$ **5,318 tokens**.
|
||||
- `recherche`: 127,356 $\rightarrow$ **4,323 tokens**.
|
||||
- `grabengasse_media`: 49,447 $\rightarrow$ **3,356 tokens**.
|
||||
- `doktor`: 20,259 $\rightarrow$ **1,079 tokens**.
|
||||
- **How to Avoid Recurrence**:
|
||||
- Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests.
|
||||
- Rely on deep-compaction with fallback rather than hard-resets across all topic agents.
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
@@ -17,15 +17,15 @@ import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
THRESHOLD_TOKENS = 20000
|
||||
THRESHOLD_TRANSCRIPT_BYTES = 300000
|
||||
THRESHOLD_EVENT_COUNT = 40
|
||||
THRESHOLD_TOKENS = 15000
|
||||
THRESHOLD_TRANSCRIPT_BYTES = 200000
|
||||
THRESHOLD_EVENT_COUNT = 30
|
||||
CRON_MAX_AGE_HOURS = 48
|
||||
RPC_TIMEOUT_MS = 180000
|
||||
|
||||
# Tier A: Event-logging agents whose state is 100% externalized (DuckDB / JSONL / Groceries DB).
|
||||
# Nightly reset to 0 prevents cross-day hallucination and eliminates LLM compaction token costs.
|
||||
TIER_A_RESET_TARGETS = [
|
||||
# Tier A: High-frequency interaction topics (koch, hebamme, shopping).
|
||||
# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory.
|
||||
TIER_A_TARGETS = [
|
||||
("koch", "agent:koch:telegram:group:-1004326921510:topic:21"),
|
||||
("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"),
|
||||
("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"),
|
||||
@@ -117,9 +117,9 @@ def prune_stale_cron_sessions():
|
||||
|
||||
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
|
||||
|
||||
def reset_tier_a_sessions():
|
||||
print("=== Tier A: Daily Hard-Reset to 0 (koch, hebamme, shopping) ===")
|
||||
for agent, key in TIER_A_RESET_TARGETS:
|
||||
def compact_tier_a_sessions():
|
||||
print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===")
|
||||
for agent, key in TIER_A_TARGETS:
|
||||
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
|
||||
if not os.path.exists(db_path):
|
||||
continue
|
||||
@@ -130,18 +130,32 @@ def reset_tier_a_sessions():
|
||||
if not row or not row[0]:
|
||||
print(f" [Tier A] {agent} ({key}) already at 0 (no active session).")
|
||||
continue
|
||||
print(f" [Tier A] Resetting {agent} ({key})...")
|
||||
ev_cnt, ev_bytes = get_session_metrics(agent, key, row[0])
|
||||
if ev_cnt <= 2 and ev_bytes < 20000:
|
||||
print(f" [Tier A] {agent} ({key}) has only {ev_cnt} events ({ev_bytes} bytes), skipping.")
|
||||
continue
|
||||
print(f" [Tier A] Deep-compacting {agent} ({key}) [{ev_cnt} events, {ev_bytes//1024}KB]...")
|
||||
r = subprocess.run(
|
||||
["openclaw", "sessions", "delete", key, "--agent", agent, "--yes"],
|
||||
capture_output=True, text=True, timeout=60.0
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if r.returncode == 0:
|
||||
print(f" ✓ Reset success: {r.stdout.strip()}")
|
||||
print(f" ✓ Deep-compacted: {r.stdout.strip()}")
|
||||
elif "Already compacted" in r.stderr or "Already compacted" in r.stdout:
|
||||
print(f" ✓ Already compacted.")
|
||||
else:
|
||||
print(f" ✗ Reset error: {r.stderr.strip()}")
|
||||
print(f" ✗ Standard compact error: {r.stderr.strip()}; retrying with --max-lines 40...")
|
||||
fb = subprocess.run(
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "40", "--timeout", str(RPC_TIMEOUT_MS)],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if fb.returncode == 0:
|
||||
print(f" ✓ Truncation fallback success: {fb.stdout.strip()}")
|
||||
else:
|
||||
print(f" ✗ Truncation fallback failed: {fb.stderr.strip()}")
|
||||
except Exception as e:
|
||||
print(f" ✗ Exception resetting {agent}: {e}")
|
||||
time.sleep(1)
|
||||
print(f" ✗ Exception deep-compacting {agent}: {e}")
|
||||
time.sleep(2)
|
||||
|
||||
def get_session_metrics(agent, key, sid=None):
|
||||
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
|
||||
@@ -166,7 +180,7 @@ def get_session_metrics(agent, key, sid=None):
|
||||
return 0, 0
|
||||
|
||||
def compact_active_sessions():
|
||||
tier_a_keys = set(t[1] for t in TIER_A_RESET_TARGETS)
|
||||
tier_a_keys = set(t[1] for t in TIER_A_TARGETS)
|
||||
print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===")
|
||||
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
|
||||
if res.returncode != 0:
|
||||
@@ -266,7 +280,7 @@ def main():
|
||||
except Exception:
|
||||
pass
|
||||
prune_stale_cron_sessions()
|
||||
reset_tier_a_sessions()
|
||||
compact_tier_a_sessions()
|
||||
compact_active_sessions()
|
||||
checkpoint_wal_journals()
|
||||
vacuum_and_optimize()
|
||||
|
||||
@@ -5,8 +5,8 @@
|
||||
# Schedule: Every night at 04:45 Europe/Berlin (02:45 UTC) (cron: 45 4 * * *)
|
||||
#
|
||||
# Architecture:
|
||||
# - Tier A: Daily Hard-Reset to 0 for koch (Topic 21), hebamme (Topic 22), shopping (Topic 115)
|
||||
# (100% externalized state in DuckDB/JSONL; resets prevent cross-day hallucination)
|
||||
# - Tier A: Deep-Compaction for high-frequency topics (koch, hebamme, shopping)
|
||||
# (maintains conversational continuity while keeping token count very low)
|
||||
# - Tier B: Smart Compaction for long-running project agents (itguy, recherche, regensburg, fitness, doktor)
|
||||
# - WAL checkpoints and SQLite optimization without blocking full VACUUM
|
||||
# ==============================================================================
|
||||
@@ -45,15 +45,15 @@ import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
THRESHOLD_TOKENS = 20000
|
||||
THRESHOLD_TRANSCRIPT_BYTES = 300000
|
||||
THRESHOLD_EVENT_COUNT = 40
|
||||
THRESHOLD_TOKENS = 15000
|
||||
THRESHOLD_TRANSCRIPT_BYTES = 200000
|
||||
THRESHOLD_EVENT_COUNT = 30
|
||||
CRON_MAX_AGE_HOURS = 48
|
||||
RPC_TIMEOUT_MS = 180000
|
||||
|
||||
# Tier A: Event-logging agents whose state is 100% externalized (DuckDB / JSONL / Groceries DB).
|
||||
# Nightly reset to 0 prevents cross-day hallucination and eliminates LLM compaction token costs.
|
||||
TIER_A_RESET_TARGETS = [
|
||||
# Tier A: High-frequency interaction topics (koch, hebamme, shopping).
|
||||
# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory.
|
||||
TIER_A_TARGETS = [
|
||||
("koch", "agent:koch:telegram:group:-1004326921510:topic:21"),
|
||||
("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"),
|
||||
("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"),
|
||||
@@ -145,9 +145,9 @@ def prune_stale_cron_sessions():
|
||||
|
||||
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
|
||||
|
||||
def reset_tier_a_sessions():
|
||||
print("=== Tier A: Daily Hard-Reset to 0 (koch, hebamme, shopping) ===")
|
||||
for agent, key in TIER_A_RESET_TARGETS:
|
||||
def compact_tier_a_sessions():
|
||||
print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===")
|
||||
for agent, key in TIER_A_TARGETS:
|
||||
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
|
||||
if not os.path.exists(db_path):
|
||||
continue
|
||||
@@ -158,18 +158,32 @@ def reset_tier_a_sessions():
|
||||
if not row or not row[0]:
|
||||
print(f" [Tier A] {agent} ({key}) already at 0 (no active session).")
|
||||
continue
|
||||
print(f" [Tier A] Resetting {agent} ({key})...")
|
||||
ev_cnt, ev_bytes = get_session_metrics(agent, key, row[0])
|
||||
if ev_cnt <= 2 and ev_bytes < 20000:
|
||||
print(f" [Tier A] {agent} ({key}) has only {ev_cnt} events ({ev_bytes} bytes), skipping.")
|
||||
continue
|
||||
print(f" [Tier A] Deep-compacting {agent} ({key}) [{ev_cnt} events, {ev_bytes//1024}KB]...")
|
||||
r = subprocess.run(
|
||||
["openclaw", "sessions", "delete", key, "--agent", agent, "--yes"],
|
||||
capture_output=True, text=True, timeout=60.0
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if r.returncode == 0:
|
||||
print(f" ✓ Reset success: {r.stdout.strip()}")
|
||||
print(f" ✓ Deep-compacted: {r.stdout.strip()}")
|
||||
elif "Already compacted" in r.stderr or "Already compacted" in r.stdout:
|
||||
print(f" ✓ Already compacted.")
|
||||
else:
|
||||
print(f" ✗ Reset error: {r.stderr.strip()}")
|
||||
print(f" ✗ Standard compact error: {r.stderr.strip()}; retrying with --max-lines 40...")
|
||||
fb = subprocess.run(
|
||||
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "40", "--timeout", str(RPC_TIMEOUT_MS)],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
if fb.returncode == 0:
|
||||
print(f" ✓ Truncation fallback success: {fb.stdout.strip()}")
|
||||
else:
|
||||
print(f" ✗ Truncation fallback failed: {fb.stderr.strip()}")
|
||||
except Exception as e:
|
||||
print(f" ✗ Exception resetting {agent}: {e}")
|
||||
time.sleep(1)
|
||||
print(f" ✗ Exception deep-compacting {agent}: {e}")
|
||||
time.sleep(2)
|
||||
|
||||
def get_session_metrics(agent, key, sid=None):
|
||||
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
|
||||
@@ -194,7 +208,7 @@ def get_session_metrics(agent, key, sid=None):
|
||||
return 0, 0
|
||||
|
||||
def compact_active_sessions():
|
||||
tier_a_keys = set(t[1] for t in TIER_A_RESET_TARGETS)
|
||||
tier_a_keys = set(t[1] for t in TIER_A_TARGETS)
|
||||
print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===")
|
||||
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
|
||||
if res.returncode != 0:
|
||||
@@ -294,7 +308,7 @@ def main():
|
||||
except Exception:
|
||||
pass
|
||||
prune_stale_cron_sessions()
|
||||
reset_tier_a_sessions()
|
||||
compact_tier_a_sessions()
|
||||
compact_active_sessions()
|
||||
checkpoint_wal_journals()
|
||||
vacuum_and_optimize()
|
||||
|
||||
Reference in New Issue
Block a user