feat(openclaw): tune compaction to keepRecentTokens 3k and convert Tier A to deep-compaction

2026-10-12 17:10:00 +02:00
parent 8f01b75bd6
commit 393d928b19
4 changed files with 104 additions and 38 deletions
@@ -32,6 +32,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
 | **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. |
 | **2026-10-11** | **Automation & Garmin Sync** | Garmin Connect sync stopped updating overnight; cron job missing from OpenClaw; untracked activities folder | `RESOLVED` | Identified `garmin-nightly-mirror` was never registered in OpenClaw; created missing `activities` directory, hardened `setup-garmin-cron.sh` with JSON declaration lookup and `--announce`, registered automation (Daily 03:30 Berlin), and verified full sync through 2026-10-11. |
+| **2026-10-12** | **Compaction Tuning & History Retention** | Session bloat during the day (~58k tokens) in Topic 21; tuning compaction strength to avoid binary choice between 0-token reset and bloat | `RESOLVED` | Configured `keepRecentTokens: 3000`, `recentTurnsPreserve: 2`, `maxActiveTranscriptBytes: 180kb`. Converted Tier A to deep-compaction with fallback, preserving conversational history while shrinking sessions from 30k-127k down to 1k-4k tokens. |


 ---
@@ -833,3 +834,40 @@ This document captures fixed issues, operational quirks, and incident root-cause
  - Agents must enforce strict focus on the very last user message at the tail of the transcript.
  - Reset sessions periodically when context grows beyond 30k tokens for simple transactional logging tasks.

+### Issue 26: Compaction Parameter Tuning & Shift to Deep-Compaction with History Retention
+
+- **Date**: 2026-10-12
+- **Severity**: Medium (Evaluating why sessions grew to ~58k tokens despite nightly maintenance; eliminating the binary choice between complete 0-token reset and bloated 58k-token sessions)
+- **Symptom**:
+  Moritz inquired why Topic 21 had reached ~58k tokens and asked whether nightly maintenance was functioning properly. He requested investigating if compaction could be tuned to aggressively compact while preserving a few recent tokens/history for all agents instead of performing hard 0-token resets.
+- **Root Cause Isolated**:
+  1. **Nightly Job Execution Verified**: The nightly maintenance cron (`ce39bbf8-b83f-453e-be53-04e27db27b78`, schedule `45 4 * * * @ Europe/Berlin`) had indeed run successfully at 04:45:00 Berlin (`1791773100019 ms`).
+  2. **Intraday Context Inflation**: The 58k tokens were NOT carried over from previous days; they were accumulated within the same day between 07:20 and 11:55 due to base prompt/tools (~18.2k baseline), 10 queries in Turn 1, multiple Perplexity web searches (~16 kB each), and `cat foods.jsonl`.
+  3. **Overly Lenient Auto-Compaction Settings**:
+     - `keepRecentTokens: 15000`: Left 15k tokens verbatim in memory after compaction.
+     - `maxActiveTranscriptBytes: "400kb"`: Did not trigger inline safeguard compaction during the day until files reached 400 kB (~50k+ tokens).
+     - `recentTurnsPreserve: 3` (default): Preserved 3 full turns with potentially huge tool outputs verbatim.
+  4. **Binary Tier A Trade-off**: Tier A agents (`koch`, `hebamme`, `shopping`) were previously hard-reset to 0 (`openclaw sessions delete`) because state is externalized in DuckDB. While preventing hallucinations, this discarded recent conversational continuity.
+- **Resolution**:
+  1. **Tuned Compaction Parameters**:
+     - Updated both live configuration (`/home/node/.openclaw/openclaw.json`) and declarative manifest (`k8s/openclaw/openclaw-instance.secret.yaml`):
+       - `keepRecentTokens: 3000` (down from 15,000; forces dense LLM summarization).
+       - `recentTurnsPreserve: 2` (preserves only the last 2 turns verbatim).
+       - `maxActiveTranscriptBytes: "180kb"` (down from 400kb; triggers proactive compaction before intraday sessions exceed ~25k tokens).
+  2. **Switched Tier A from Hard-Reset to Deep-Compaction**:
+     - Updated `k8s/openclaw/setup-compaction-cron.sh` and `k8s/openclaw/scripts/compact_sessions.py`.
+     - Tier A now calls `openclaw sessions compact <key> --agent <agent>` with fallback to `--max-lines 40` if built-in safeguard summarization quality checks fail.
+     - Lowered Tier B thresholds: `THRESHOLD_TOKENS = 15000`, `THRESHOLD_TRANSCRIPT_BYTES = 200000` (200 KB), `THRESHOLD_EVENT_COUNT = 30`.
+     - Re-registered cron job in OpenClaw (`setup-compaction-cron.sh`).
+  3. **Verified Live Reduction**:
+     - `koch`: 29,783 $\rightarrow$ **2,098 tokens** (Tier A deep-compacted).
+     - `hebamme`: Truncated to 40 lines (Tier A fallback).
+     - `fitness`: 101,950 $\rightarrow$ **5,318 tokens**.
+     - `recherche`: 127,356 $\rightarrow$ **4,323 tokens**.
+     - `grabengasse_media`: 49,447 $\rightarrow$ **3,356 tokens**.
+     - `doktor`: 20,259 $\rightarrow$ **1,079 tokens**.
+- **How to Avoid Recurrence**:
+  - Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests.
+  - Rely on deep-compaction with fallback rather than hard-resets across all topic agents.
+
+