feat(openclaw): strip tracking agent prompt bloat and add 15m deterministic sliding-window compaction

- Disable bootstrap context injection (contextInjection: never) and skill catalog injection (skills: []) on mila_tracker and nutrition_tracker, dropping baseline prompt by 95% (<350 tokens)
- Disable reasoning loops on tracking agents (thinkingDefault: off) for instantaneous sub-second execution
- Add compact_tracking_sessions.py with deterministic 8-line sliding window (--max-lines 8) requiring 0 LLM tokens
- Register automated 15-minute cron job (tracking-session-sliding-window) in OpenClaw gateway
- Streamline setup-compaction-cron.sh to sync python scripts directly and manage both 15m and nightly maintenance
This commit is contained in:
Moritz Graf
2026-10-14 14:14:35 +02:00
parent a0c8c3828c
commit 02fcbd3174
5 changed files with 190 additions and 326 deletions
+20 -2
View File
@@ -94,12 +94,30 @@ This repository is divided into distinct domains, each with its own `AGENTS.md`
* **Key Tools**: Interactive `uv run flash.py` wizard and `./send_message.sh` CLI push messaging. * **Key Tools**: Interactive `uv run flash.py` wizard and `./send_message.sh` CLI push messaging.
* **Read**: [projects/esphome_display_kitchen/AGENTS.md](./projects/esphome_display_kitchen/AGENTS.md) * **Read**: [projects/esphome_display_kitchen/AGENTS.md](./projects/esphome_display_kitchen/AGENTS.md)
### 12. Utility Scripts (`scripts/`) ### 12. [DuckDB Backend](./projects/duckdb-backend/AGENTS.md) (`projects/duckdb-backend/`)
* **Purpose**: High-performance HTTP analytics and event tracking data service executing in-memory DuckDB queries against Knowledge Wiki JSONL datasets.
* **Manifests**: `k8s/family/` (`duckdb-backend.yaml`, `pvc.yaml`, `ingress.yaml` in namespace `family`).
* **Endpoints**: Curated dashboards (`/duckdb-api/v1/mila/dashboard`, `/duckdb-api/v1/moritz/dashboard`), generic SQL queries (`/duckdb-api/v1/query`), and event tracking writes with Git sync.
* **Read**: [projects/duckdb-backend/AGENTS.md](./projects/duckdb-backend/AGENTS.md)
### 13. [Mila Tracking Web App](./projects/mila/AGENTS.md) (`projects/mila/`)
* **Purpose**: Reactive mobile-first PWA for Mila's baby sleep, feeding, diaper tracking, and growth percentiles (`https://family.haumdaucher.de/tracking/mila`).
* **Manifest**: `k8s/family/mila.yaml` (Namespace: `family`).
* **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; live wake timer; interactive 1-tap and custom-time tracking modal.
* **Read**: [projects/mila/AGENTS.md](./projects/mila/AGENTS.md)
### 14. [Moritz Health & Nutrition Web App](./projects/moritz/AGENTS.md) (`projects/moritz/`)
* **Purpose**: Apple Health-inspired nutrition and metabolic balance dashboard (`https://family.haumdaucher.de/tracking/moritz`).
* **Manifest**: `k8s/family/moritz.yaml` (Namespace: `family`).
* **Key Rules**: Vue 3, Vite, TypeScript, and Tailwind CSS; net calorie balance hero, Garmin health breakdown, and quick meal logger.
* **Read**: [projects/moritz/AGENTS.md](./projects/moritz/AGENTS.md)
### 15. Utility Scripts (`scripts/`)
* **Purpose**: Cross-cutting operational and administration CLI scripts that do not belong inside Kubernetes namespace directories. * **Purpose**: Cross-cutting operational and administration CLI scripts that do not belong inside Kubernetes namespace directories.
* **Key Tools**: * **Key Tools**:
* `scripts/htpasswd.py`: Generates bcrypt-hashed htpasswd credentials for ingress basic auth secrets. * `scripts/htpasswd.py`: Generates bcrypt-hashed htpasswd credentials for ingress basic auth secrets.
### 13. Network & Router Integration (`fritzbox/`) ### 16. Network & Router Integration (`fritzbox/`)
* **Purpose**: Site-to-Site WireGuard LAN coupling and DynDNS OVH synchronization for the home FritzBox router. * **Purpose**: Site-to-Site WireGuard LAN coupling and DynDNS OVH synchronization for the home FritzBox router.
* **Cluster Endpoint**: `k8s/wireguard/` (`vpn.haumdaucher.de`). * **Cluster Endpoint**: `k8s/wireguard/` (`vpn.haumdaucher.de`).
* **Read**: [fritzbox/README.md](./fritzbox/README.md) * **Read**: [fritzbox/README.md](./fritzbox/README.md)
Binary file not shown.
+48 -3
View File
@@ -23,6 +23,15 @@ THRESHOLD_EVENT_COUNT = 30
CRON_MAX_AGE_HOURS = 48 CRON_MAX_AGE_HOURS = 48
RPC_TIMEOUT_MS = 180000 RPC_TIMEOUT_MS = 180000
# Tracking Channels: Deterministic sliding-window (mila_tracker, nutrition_tracker).
# Strict sliding window of ~5-8 events (2-3 turns max), 0 LLM tokens, instant truncation.
TRACKING_TARGETS = [
("mila_tracker", "agent:mila_tracker:telegram:group:-1004326921510:topic:2796"),
("nutrition_tracker", "agent:nutrition_tracker:telegram:group:-1004326921510:topic:2797"),
]
MAX_TRACKING_EVENTS = 8
MAX_TRACKING_LINES = 8
# Tier A: High-frequency interaction topics (koch, hebamme, shopping). # Tier A: High-frequency interaction topics (koch, hebamme, shopping).
# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory. # Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory.
TIER_A_TARGETS = [ TIER_A_TARGETS = [
@@ -117,6 +126,41 @@ def prune_stale_cron_sessions():
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.") print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
def compact_tracking_sessions():
print("=== Tracking Channels Sliding-Window Maintenance (mila_tracker, nutrition_tracker) ===")
for agent, key in TRACKING_TARGETS:
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
continue
try:
con = sqlite3.connect(db_path, timeout=5.0)
row = con.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
if not row or not row[0]:
con.close()
print(f" [{agent}] No active session node found for {key}.")
continue
sid = row[0]
cnt = con.execute("SELECT count(*) FROM transcript_events WHERE session_id=?", (sid,)).fetchone()[0]
con.close()
if cnt > MAX_TRACKING_EVENTS:
print(f" [{agent}] Session {key} has {cnt} events (> {MAX_TRACKING_EVENTS}). Truncating to last {MAX_TRACKING_LINES} lines...")
cmd = [
"openclaw", "sessions", "compact", key,
"--agent", agent,
"--max-lines", str(MAX_TRACKING_LINES),
"--timeout", "30000"
]
r = subprocess.run(cmd, capture_output=True, text=True)
if r.returncode == 0:
print(f" ✓ Truncated successfully: {r.stdout.strip()}")
else:
print(f" ✗ Truncation failed: {r.stderr.strip()}")
else:
print(f" [{agent}] Session {key} has {cnt} events (<= {MAX_TRACKING_EVENTS}). Clean.")
except Exception as e:
print(f" ✗ [{agent}] Error during tracking compaction: {e}")
def compact_tier_a_sessions(): def compact_tier_a_sessions():
print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===") print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===")
for agent, key in TIER_A_TARGETS: for agent, key in TIER_A_TARGETS:
@@ -180,7 +224,7 @@ def get_session_metrics(agent, key, sid=None):
return 0, 0 return 0, 0
def compact_active_sessions(): def compact_active_sessions():
tier_a_keys = set(t[1] for t in TIER_A_TARGETS) excluded_keys = set(t[1] for t in TIER_A_TARGETS) | set(t[1] for t in TRACKING_TARGETS)
print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===") print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===")
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True) res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
if res.returncode != 0: if res.returncode != 0:
@@ -197,8 +241,8 @@ def compact_active_sessions():
tokens = s.get("totalTokens") or 0 tokens = s.get("totalTokens") or 0
sid = s.get("sessionId") sid = s.get("sessionId")
# Skip Tier A agents as they are handled via hard-reset # Skip Tier A and Tracking agents as they have dedicated routines
if key in tier_a_keys: if key in excluded_keys:
continue continue
# Target main sessions and all telegram sessions (topics and direct) # Target main sessions and all telegram sessions (topics and direct)
@@ -280,6 +324,7 @@ def main():
except Exception: except Exception:
pass pass
prune_stale_cron_sessions() prune_stale_cron_sessions()
compact_tracking_sessions()
compact_tier_a_sessions() compact_tier_a_sessions()
compact_active_sessions() compact_active_sessions()
checkpoint_wal_journals() checkpoint_wal_journals()
@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""
Lightweight sliding-window compaction for OpenClaw tracking channels:
- mila_tracker (topic 2796)
- nutrition_tracker (topic 2797)
Maintains a strict sliding window of ~5-8 transcript events (max 2-3 turns).
Uses native gateway truncation (--max-lines 8) requiring 0 LLM tokens and 0 external API calls.
"""
import os
import sqlite3
import subprocess
import sys
import time
MAX_TRACKING_EVENTS = 8
MAX_TRACKING_LINES = 8
TIMEOUT_MS = 30000
TRACKING_TARGETS = [
("mila_tracker", "agent:mila_tracker:telegram:group:-1004326921510:topic:2796"),
("nutrition_tracker", "agent:nutrition_tracker:telegram:group:-1004326921510:topic:2797"),
]
def check_and_compact_tracking():
compacted_count = 0
skipped_count = 0
for agent, key in TRACKING_TARGETS:
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
continue
try:
con = sqlite3.connect(db_path, timeout=5.0)
row = con.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
if not row or not row[0]:
con.close()
continue
sid = row[0]
cnt = con.execute("SELECT count(*) FROM transcript_events WHERE session_id=?", (sid,)).fetchone()[0]
con.close()
if cnt > MAX_TRACKING_EVENTS:
print(f"[{agent}] Session {key} has {cnt} events (threshold {MAX_TRACKING_EVENTS}). Truncating to last {MAX_TRACKING_LINES} lines...")
cmd = [
"openclaw", "sessions", "compact", key,
"--agent", agent,
"--max-lines", str(MAX_TRACKING_LINES),
"--timeout", str(TIMEOUT_MS)
]
res = subprocess.run(cmd, capture_output=True, text=True)
if res.returncode == 0:
print(f" ✓ [{agent}] Truncated successfully: {res.stdout.strip()}")
compacted_count += 1
else:
print(f" ✗ [{agent}] Truncation failed: {res.stderr.strip()}", file=sys.stderr)
else:
skipped_count += 1
except Exception as e:
print(f"[{agent}] Error checking session: {e}", file=sys.stderr)
if compacted_count > 0:
print(f"Tracking maintenance: {compacted_count} sessions truncated, {skipped_count} within window.")
if __name__ == "__main__":
check_and_compact_tracking()
+56 -321
View File
@@ -1,14 +1,16 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# ============================================================================== # ==============================================================================
# setup-compaction-cron.sh # setup-compaction-cron.sh
# Declaratively registers the Multi-Agent Nightly Session Compaction in OpenClaw. # Declaratively registers OpenClaw session maintenance automations:
# Schedule: Every night at 04:45 Europe/Berlin (02:45 UTC) (cron: 45 4 * * *) # 1. Tracking Sliding Window (every 15m): Truncates mila_tracker & nutrition_tracker
# # to <= 8 events (~5 messages / 2 turns) via instant native gateway truncation.
# Architecture: # 2. Nightly Deep Compaction (every night at 04:45 Europe/Berlin):
# - Tier A: Deep-Compaction for high-frequency topics (koch, hebamme, shopping) # - Pre-warms SQLite page cache on rotational storage
# (maintains conversational continuity while keeping token count very low) # - Recovers wedged/quarantined sessions
# - Tier B: Smart Compaction for long-running project agents (itguy, recherche, regensburg, fitness, doktor) # - Prunes abandoned isolated cron sessions (>48h)
# - WAL checkpoints and SQLite optimization without blocking full VACUUM # - Deep-compacts conversational topics (koch, hebamme, shopping)
# - Smart-compacts active long-running sessions (itguy, recherche, etc.)
# - Flushes SQLite WAL journals and executes PRAGMA optimize
# ============================================================================== # ==============================================================================
set -euo pipefail set -euo pipefail
@@ -16,326 +18,62 @@ set -euo pipefail
NAMESPACE="${NAMESPACE:-openclaw}" NAMESPACE="${NAMESPACE:-openclaw}"
POD_NAME="${POD_NAME:-openclaw-0}" POD_NAME="${POD_NAME:-openclaw-0}"
CONTAINER_NAME="${CONTAINER_NAME:-openclaw}" CONTAINER_NAME="${CONTAINER_NAME:-openclaw}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JOB_NAME="nightly-session-compaction" echo "=== Registering Multi-Agent Session Maintenance in OpenClaw ($POD_NAME) ==="
# 1. Install maintenance scripts from repo onto PVC
echo "Syncing compact_tracking_sessions.py to pod..."
kubectl exec -i -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- /bin/sh -c \
'cat > /home/node/.openclaw/scripts/compact_tracking_sessions.py && chmod +x /home/node/.openclaw/scripts/compact_tracking_sessions.py' \
< "$SCRIPT_DIR/scripts/compact_tracking_sessions.py"
echo "Syncing compact_sessions.py to pod..."
kubectl exec -i -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- /bin/sh -c \
'cat > /home/node/.openclaw/scripts/compact_sessions.py && chmod +x /home/node/.openclaw/scripts/compact_sessions.py' \
< "$SCRIPT_DIR/scripts/compact_sessions.py"
# 2. Register / Update 15-Minute Tracking Sliding Window Automation
TRACKING_JOB="tracking-session-sliding-window"
echo "Checking existing OpenClaw Cron automation '$TRACKING_JOB'..."
EXISTING_TRACKING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list --json 2>/dev/null | python3 -c 'import sys, json; data=json.load(sys.stdin); print(next((j["id"] for j in data.get("jobs", []) if j.get("declarationKey") == "'"$TRACKING_JOB"'"), ""))' 2>/dev/null || true)
if [ -n "$EXISTING_TRACKING_ID" ]; then
echo "Updating existing tracking sliding window automation '$EXISTING_TRACKING_ID' (every 15m)..."
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron edit "$EXISTING_TRACKING_ID" \
--every "15m" \
--command "python3 /home/node/.openclaw/scripts/compact_tracking_sessions.py"
else
echo "Registering new tracking sliding window automation '$TRACKING_JOB' (every 15m)..."
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
--name "Tracking Session Sliding Window" \
--declaration-key "$TRACKING_JOB" \
--description "Lightweight sliding-window truncation (max 8 events) for mila_tracker and nutrition_tracker" \
--every "15m" \
--session "isolated" \
--no-deliver \
--command "python3 /home/node/.openclaw/scripts/compact_tracking_sessions.py"
fi
# 3. Register / Update Nightly Deep Compaction Automation
NIGHTLY_JOB="nightly-session-compaction"
CRON_SCHEDULE="45 4 * * *" CRON_SCHEDULE="45 4 * * *"
TIMEZONE="Europe/Berlin" TIMEZONE="Europe/Berlin"
echo "=== Registering Multi-Agent Nightly Session Maintenance in OpenClaw ($POD_NAME) ===" echo "Checking existing OpenClaw Cron automation '$NIGHTLY_JOB'..."
EXISTING_NIGHTLY_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list --json 2>/dev/null | python3 -c 'import sys, json; data=json.load(sys.stdin); print(next((j["id"] for j in data.get("jobs", []) if j.get("declarationKey") == "'"$NIGHTLY_JOB"'"), ""))' 2>/dev/null || true)
# 1. Install or ensure python maintenance script is present on PVC if [ -n "$EXISTING_NIGHTLY_ID" ]; then
echo "Ensuring /home/node/.openclaw/scripts/compact_sessions.py exists..." echo "Updating existing nightly compaction automation '$EXISTING_NIGHTLY_ID' ($CRON_SCHEDULE @ $TIMEZONE)..."
kubectl exec -i -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- /bin/sh -c 'cat << "EOF" > /home/node/.openclaw/scripts/compact_sessions.py kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron edit "$EXISTING_NIGHTLY_ID" \
#!/usr/bin/env python3
"""
Nightly multi-agent session maintenance and compaction script for OpenClaw.
1. Pre-warms OS page cache for SQLite databases on spinning disks.
2. Recovers any wedged or locked sessions via recover_sessions.py.
3. Prunes abandoned isolated cron sessions (>48h old) to prevent token/state bloat.
4. Tier A Hard-Reset: Resets koch, hebamme, and shopping topics to 0 tokens (clean archive).
5. Tier B Smart-Compaction: Compacts active sessions exceeding token thresholds (5s pacing).
6. Checkpoints SQLite WAL journals across all agent databases to disk.
7. Optimizes SQLite query planner without blocking full VACUUM.
"""
import glob
import json
import os
import sqlite3
import subprocess
import sys
import time
THRESHOLD_TOKENS = 15000
THRESHOLD_TRANSCRIPT_BYTES = 200000
THRESHOLD_EVENT_COUNT = 30
CRON_MAX_AGE_HOURS = 48
RPC_TIMEOUT_MS = 180000
# Tier A: High-frequency interaction and tracking topics (koch, hebamme, shopping, mila_tracker, nutrition_tracker).
# Deep-compaction keeps a compact summary + latest 2 turns without completely wiping conversational memory.
TIER_A_TARGETS = [
("koch", "agent:koch:telegram:group:-1004326921510:topic:21"),
("hebamme", "agent:hebamme:telegram:group:-1004326921510:topic:22"),
("shopping", "agent:shopping:telegram:group:-1004326921510:topic:115"),
("mila_tracker", "agent:mila_tracker:telegram:group:-1004326921510:topic:2796"),
("nutrition_tracker", "agent:nutrition_tracker:telegram:group:-1004326921510:topic:2797"),
]
def prewarm_page_cache():
print("=== Pre-warming SQLite databases into OS page cache ===")
t0 = time.time()
try:
subprocess.run(
"cat /home/node/.openclaw/state/openclaw.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true",
shell=True,
timeout=90.0
)
print(f"Page cache pre-warmed in {time.time() - t0:.2f}s")
except Exception as e:
print(f"Page cache pre-warming warning: {e}")
def prune_stale_cron_sessions():
print(f"=== Pruning isolated cron sessions older than {CRON_MAX_AGE_HOURS}h ===")
cutoff_ms = int(time.time() * 1000) - (CRON_MAX_AGE_HOURS * 3600 * 1000)
total_pruned = 0
total_tokens_reclaimed = 0
for db in sorted(glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")):
agent = db.split("/")[5]
try:
conn = sqlite3.connect(db, timeout=30.0)
conn.execute("PRAGMA foreign_keys = ON;")
cursor = conn.cursor()
cursor.execute("SELECT name FROM sqlite_master WHERE type=\"table\"")
all_tables = set(r[0] for r in cursor.fetchall())
cursor.execute("SELECT session_key, current_session_id, updated_at, entry_json FROM session_nodes WHERE session_key LIKE \"%:cron:%\"")
stale = []
for k, sid, up, ej in cursor.fetchall():
if up and up < cutoff_ms:
tok = 0
if ej:
try:
tok = json.loads(ej).get("totalTokens", 0) or 0
except Exception:
pass
stale.append((k, sid, tok))
if stale:
tokens_reclaimed = sum(t[2] for t in stale)
for k, sid, _ in stale:
for t in ["trajectory_runtime_events", "transcript_events", "transcript_event_identities", "session_transcript_active_events", "transcript_rewrite_watermarks", "session_transcript_index_state", "session_transcript_fts", "session_conversations"]:
if t in all_tables and sid:
cursor.execute(f"DELETE FROM {t} WHERE session_id = ?", (sid,))
for t in ["session_participants", "heartbeat_outcomes", "session_progress_cards", "board_widgets", "board_tabs"]:
if t in all_tables:
cursor.execute(f"DELETE FROM {t} WHERE session_key = ?", (k,))
if "memory_entry_origins" in all_tables:
cursor.execute("DELETE FROM memory_entry_origins WHERE session_key = ? OR session_id = ?", (k, sid or ""))
if "session_windows" in all_tables:
cursor.execute("DELETE FROM session_windows WHERE session_key = ? OR session_id = ?", (k, sid or ""))
cursor.execute("DELETE FROM session_nodes WHERE session_key = ?", (k,))
conn.commit()
total_pruned += len(stale)
total_tokens_reclaimed += tokens_reclaimed
print(f" [{agent}] Pruned {len(stale)} dead cron sessions ({tokens_reclaimed} tokens reclaimed)")
# Unconditionally resolve any dangling foreign key references across all tables
while True:
cursor.execute("PRAGMA foreign_key_check;")
violations = cursor.fetchall()
if not violations:
break
for table, rowid, parent, fkid in violations:
cursor.execute(f"DELETE FROM {table} WHERE rowid = ?", (rowid,))
conn.commit()
conn.close()
except Exception as e:
print(f" [{agent}] Pruning error: {e}")
# Clear resolved quarantine entries if all databases are healthy
try:
q_path = "/home/node/.openclaw/state/openclaw-quarantine.sqlite"
if os.path.exists(q_path):
q_conn = sqlite3.connect(q_path)
q_cur = q_conn.cursor()
q_cur.execute("DELETE FROM quarantined_databases")
q_conn.commit()
q_conn.close()
except Exception as e:
print(f"Quarantine cleanup error: {e}")
print(f"Cron pruning complete: {total_pruned} sessions pruned, {total_tokens_reclaimed} tokens reclaimed.")
def compact_tier_a_sessions():
print("=== Tier A: Deep-Compaction (koch, hebamme, shopping) ===")
for agent, key in TIER_A_TARGETS:
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
continue
try:
con = sqlite3.connect(db_path, timeout=10.0)
row = con.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
con.close()
if not row or not row[0]:
print(f" [Tier A] {agent} ({key}) already at 0 (no active session).")
continue
ev_cnt, ev_bytes = get_session_metrics(agent, key, row[0])
if ev_cnt <= 2 and ev_bytes < 20000:
print(f" [Tier A] {agent} ({key}) has only {ev_cnt} events ({ev_bytes} bytes), skipping.")
continue
print(f" [Tier A] Deep-compacting {agent} ({key}) [{ev_cnt} events, {ev_bytes//1024}KB]...")
r = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if r.returncode == 0:
print(f" ✓ Deep-compacted: {r.stdout.strip()}")
elif "Already compacted" in r.stderr or "Already compacted" in r.stdout:
print(f" ✓ Already compacted.")
else:
print(f" ✗ Standard compact error: {r.stderr.strip()}; retrying with --max-lines 40...")
fb = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "40", "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if fb.returncode == 0:
print(f" ✓ Truncation fallback success: {fb.stdout.strip()}")
else:
print(f" ✗ Truncation fallback failed: {fb.stderr.strip()}")
except Exception as e:
print(f" ✗ Exception deep-compacting {agent}: {e}")
time.sleep(2)
def get_session_metrics(agent, key, sid=None):
db_path = f"/home/node/.openclaw/agents/{agent}/agent/openclaw-agent.sqlite"
if not os.path.exists(db_path):
return 0, 0
try:
con = sqlite3.connect(db_path, timeout=5.0)
cur = con.cursor()
if not sid:
row = cur.execute("SELECT current_session_id FROM session_nodes WHERE session_key=?", (key,)).fetchone()
sid = row[0] if row else None
if not sid:
con.close()
return 0, 0
ev_cnt, ev_bytes = cur.execute(
"SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?",
(sid,)
).fetchone()
con.close()
return ev_cnt, ev_bytes
except Exception:
return 0, 0
def compact_active_sessions():
tier_a_keys = set(t[1] for t in TIER_A_TARGETS)
print(f"=== Tier B: Running Multi-Agent Compaction (threshold: {THRESHOLD_TOKENS} tokens / {THRESHOLD_TRANSCRIPT_BYTES//1024}KB, LLM summarization) ===")
res = subprocess.run(["openclaw", "sessions", "--all-agents", "--json"], capture_output=True, text=True)
if res.returncode != 0:
print(f"Failed to list sessions: {res.stderr}")
return
sessions = json.loads(res.stdout).get("sessions", [])
compacted = 0
skipped = 0
for s in sessions:
key = s.get("key", "")
agent = s.get("agentId") or "main"
tokens = s.get("totalTokens") or 0
sid = s.get("sessionId")
# Skip Tier A agents as they are handled via hard-reset
if key in tier_a_keys:
continue
# Target main sessions and all telegram sessions (topics and direct)
is_target = (
key == "agent:main:main" or
":telegram:" in key or
(key.startswith("agent:") and key.endswith(":main"))
)
if is_target and not key.endswith(":heartbeat"):
ev_cnt, ev_bytes = get_session_metrics(agent, key, sid)
needs_compaction = (
tokens >= THRESHOLD_TOKENS or
ev_bytes >= THRESHOLD_TRANSCRIPT_BYTES or
(tokens == 0 and ev_cnt >= THRESHOLD_EVENT_COUNT)
)
if needs_compaction:
stat_str = f"{tokens} tokens" if tokens else f"{ev_bytes//1024}KB / {ev_cnt} events"
print(f"Compacting {agent} ({stat_str}) -> {key}...")
c_res = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if c_res.returncode == 0:
print(f" ✓ {c_res.stdout.strip()}")
compacted += 1
elif "Already compacted" in c_res.stderr or "Already compacted" in c_res.stdout:
print(f" ✓ Already compacted.")
skipped += 1
else:
print(f" ✗ {c_res.stderr.strip()}; retrying with --max-lines 80 fallback...")
fb_res = subprocess.run(
["openclaw", "sessions", "compact", key, "--agent", agent, "--max-lines", "80", "--timeout", str(RPC_TIMEOUT_MS)],
capture_output=True, text=True
)
if fb_res.returncode == 0:
print(f" ✓ (fallback) {fb_res.stdout.strip()}")
compacted += 1
else:
print(f" ✗ Fallback failed: {fb_res.stderr.strip()}")
time.sleep(5) # 5s pause to avoid rotational HDD I/O saturation
else:
skipped += 1
print(f"Compaction complete: {compacted} compacted, {skipped} skipped.")
def checkpoint_wal_journals():
print("=== Checkpointing SQLite WAL journals across all agent databases ===")
dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
wal_freed = 0
for db in dbs:
wal = db + "-wal"
if os.path.exists(wal):
sz = os.path.getsize(wal)
try:
con = sqlite3.connect(db, timeout=30.0)
con.execute("PRAGMA wal_checkpoint(TRUNCATE);")
con.close()
wal_freed += sz
except Exception as e:
print(f" ✗ Error checkpointing {db}: {e}")
print(f"WAL Checkpoint complete: {wal_freed / (1024*1024):.2f} MB merged to disk.")
def vacuum_and_optimize():
print("=== Running SQLite OPTIMIZE across all databases ===")
dbs = ["/home/node/.openclaw/state/openclaw.sqlite"] + glob.glob("/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite")
for db in dbs:
try:
con = sqlite3.connect(db, timeout=30.0)
con.execute("PRAGMA optimize;")
con.close()
except Exception as e:
print(f" ✗ Error optimizing {db}: {e}")
print("SQLite optimization complete.")
def main():
prewarm_page_cache()
try:
subprocess.run(["python3", "/home/node/.openclaw/scripts/recover_sessions.py"], check=False)
except Exception:
pass
prune_stale_cron_sessions()
compact_tier_a_sessions()
compact_active_sessions()
checkpoint_wal_journals()
vacuum_and_optimize()
if __name__ == "__main__":
main()
EOF
chmod +x /home/node/.openclaw/scripts/compact_sessions.py
'
# 2. Register / Update the OpenClaw cron job
echo "Checking existing OpenClaw Cron automation '$JOB_NAME'..."
EXISTING_ID=$(kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron list --json 2>/dev/null | python3 -c 'import sys, json; data=json.load(sys.stdin); print(next((j["id"] for j in data.get("jobs", []) if j.get("declarationKey") == "'"$JOB_NAME"'"), ""))' 2>/dev/null || true)
if [ -n "$EXISTING_ID" ]; then
echo "Updating existing OpenClaw Cron automation '$EXISTING_ID' ($CRON_SCHEDULE @ $TIMEZONE)..."
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron edit "$EXISTING_ID" \
--cron "$CRON_SCHEDULE" \ --cron "$CRON_SCHEDULE" \
--tz "$TIMEZONE" \ --tz "$TIMEZONE" \
--command "python3 /home/node/.openclaw/scripts/compact_sessions.py" --command "python3 /home/node/.openclaw/scripts/compact_sessions.py"
else else
echo "Registering new OpenClaw Cron automation '$JOB_NAME' ($CRON_SCHEDULE @ $TIMEZONE)..." echo "Registering new nightly compaction automation '$NIGHTLY_JOB' ($CRON_SCHEDULE @ $TIMEZONE)..."
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \ kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron add \
--name "Nightly Session Compaction" \ --name "Nightly Session Compaction" \
--declaration-key "$JOB_NAME" \ --declaration-key "$NIGHTLY_JOB" \
--description "Nightly multi-agent session compaction and SQLite WAL checkpointing" \ --description "Nightly multi-agent session compaction and SQLite WAL checkpointing" \
--cron "$CRON_SCHEDULE" \ --cron "$CRON_SCHEDULE" \
--tz "$TIMEZONE" \ --tz "$TIMEZONE" \
@@ -344,7 +82,4 @@ else
--command "python3 /home/node/.openclaw/scripts/compact_sessions.py" --command "python3 /home/node/.openclaw/scripts/compact_sessions.py"
fi fi
echo "=== Compaction Cron automation successfully registered/updated! ===" echo "=== All session maintenance automations successfully registered/updated! ==="
if [ -n "$EXISTING_ID" ]; then
kubectl exec -n "$NAMESPACE" "$POD_NAME" -c "$CONTAINER_NAME" -- openclaw cron get "$EXISTING_ID" 2>/dev/null || true
fi