From 36e0887c735dc765094f4639131ef0c26674a19d Mon Sep 17 00:00:00 2001 From: Moritz Graf Date: Fri, 9 Oct 2026 07:54:20 +0200 Subject: [PATCH] feat(openclaw): persist operational invariants in AGENTS.md and add openclaw-ops skill runbook --- .agents/skills/openclaw-ops/SKILL.md | 175 +++++++++++++++++++++++++++ k8s/openclaw/AGENTS.md | 24 ++++ 2 files changed, 199 insertions(+) create mode 100644 .agents/skills/openclaw-ops/SKILL.md diff --git a/.agents/skills/openclaw-ops/SKILL.md b/.agents/skills/openclaw-ops/SKILL.md new file mode 100644 index 0000000..055d3af --- /dev/null +++ b/.agents/skills/openclaw-ops/SKILL.md @@ -0,0 +1,175 @@ +--- +name: openclaw-ops +license: Apache-2.0 +description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram. +--- + +# OpenClaw Operations & Troubleshooting Runbook + +> **Target Environment**: Kubernetes cluster `haumdaucher`, namespace `openclaw`, pod `openclaw-0`. +> **Key References**: [`k8s/openclaw/AGENTS.md`](../../../k8s/openclaw/AGENTS.md) and [`k8s/openclaw/TROUBLESHOOTING.md`](../../../k8s/openclaw/TROUBLESHOOTING.md). + +--- + +## 1. Diagnostic Playbook: "Bot is Not Answering / Silent" + +When a user reports that a Telegram forum topic or direct chat is unresponsive: + +### ⚠️ Golden Rule: Do NOT Assume Container Crash +Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (`5/5 Running`). + +### Step 1: Check Live Logs for In-Flight Turns +```bash +kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100 +``` +Look for: +- `compact:start reason=auto-threshold`: Inline safeguard compaction in progress (takes 30–60s). +- `embedded run tool start ... tool=write|exec`: Agent is actively executing tool steps. +- `[model-fetch] start ... status=200`: Waiting on LLM token streaming. +- `heartbeat: webhooks=... active=X`: Active turn count in gateway. + +### Step 2: Query Ground-Truth Session Status in SQLite +Each agent maintains its own SQLite database at `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite`. +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import sqlite3, glob +for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')): + agent = db.split('/')[5] + con = sqlite3.connect(db) + rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall() + if rows: + print(f'[{agent}]', rows) +" +``` +- If `status == "processing"`: Turn is actively executing. +- If `status == "done"`: Turn completed. Check outbound Telegram dispatch. +- If `status == "running"` with old `updated_at`: Session is wedged (see Recovery below). + +### Step 3: Inspect Recent Transcript Events +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import sqlite3, json +con = sqlite3.connect('/home/node/.openclaw/agents//agent/openclaw-agent.sqlite') +sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('',)).fetchone()[0] +for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall(): + print(seq, ca, ej) +" +``` + +--- + +## 2. Session Recovery & Lock Clearing + +### Un-wedge a Stuck Session Node +If a turn was killed mid-execution (e.g. OOMKill or node reboot): +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import sqlite3 +con = sqlite3.connect('/home/node/.openclaw/agents//agent/openclaw-agent.sqlite') +con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('',)) +con.commit() +print('Session un-wedged successfully.') +" +``` + +### Clean Stale State Locks +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock +``` + +### Check Foreign Key Violations & Clear Quarantine +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import sqlite3, glob +for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'): + con = sqlite3.connect(db) + fk = con.execute('PRAGMA foreign_key_check;').fetchall() + if fk: + print(f'FK violations in {db}:', fk) +" +# Clear quarantine table if present +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import sqlite3, os +p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite' +if os.path.exists(p): + c = sqlite3.connect(p) + c.execute('DELETE FROM quarantined_databases') + c.commit() + print('Quarantine cleared.') +" +``` + +--- + +## 3. Multi-Agent Compaction & Transcript Bloat Control + +### Pitfall: Stale Sessions Report `totalTokens: null` +`openclaw sessions --all-agents --json` returns `totalTokens: null` (`totalTokensFresh: false`) for sessions that haven't received a message in several days. **Always inspect SQLite directly**: +```bash +# Check size of active sessions across all agents +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import glob, sqlite3, json +for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')): + agent = db.split('/')[5] + con = sqlite3.connect(db) + for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall(): + cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone() + if bsz > 300000 or cnt > 40: + print(f'{agent:12} {bsz//1024:6} KB {cnt:4} events {skey}') +" +``` + +### Manually Compact a Stale or Bloated Session +```bash +# LLM summarization with 3-minute timeout +kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "" --agent --timeout 180000 + +# Truncation fallback (if LLM summarization times out) +kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "" --agent --max-lines 80 --timeout 180000 +``` + +### Ensure User Compaction Notification is Enabled +To prevent silent user-perceived freezes during daytime turns: +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json +``` + +--- + +## 4. Mechanical HDD / Page Cache Pre-Warming + +Node `haumdaucher` uses spinning HDDs (`rotational: 1`). Cold database opens take up to 20–50 minutes without page cache warming. + +### Warm Page Cache Instantly (<1s) +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \ + "cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true" +``` + +### WAL Checkpoints & SQLite Optimization +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " +import glob, sqlite3 +dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite') +for db in dbs: + con = sqlite3.connect(db) + con.execute('PRAGMA wal_checkpoint(TRUNCATE);') + con.execute('PRAGMA optimize;') + con.close() +print('WAL checkpoints and optimizations complete.') +" +``` + +--- + +## 5. Emergency Telegram Direct Message Bypass + +If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn: +```bash +kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \ + --channel telegram \ + --target "-1004326921510" \ + --thread-id \ + --message "Dein Nachrichtentext hier..." +``` +*(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).* diff --git a/k8s/openclaw/AGENTS.md b/k8s/openclaw/AGENTS.md index def862e..f552b59 100644 --- a/k8s/openclaw/AGENTS.md +++ b/k8s/openclaw/AGENTS.md @@ -88,6 +88,30 @@ OpenClaw connects to three foundational Git repositories: > [!IMPORTANT] > **Mandatory Bug & Incident Documentation**: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, **always document the incident in [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)** with symptom, root cause, resolution, and how to avoid recurrence. +### 🛡️ Operational & Diagnostic Invariants (Learned Guidelines) + +Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules: + +1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**: + - When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**. + - Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first. + - Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (`compact:start`) can introduce multi-minute turn latency without any pod restart. + +2. **Datastore Ground-Truth for Maintenance**: + - High-level CLI summaries (`openclaw sessions --all-agents --json`) report `totalTokens: null` (`totalTokensFresh: false`) for stale sessions or after restarts. + - Automated health checks and compaction scripts MUST query the SQLite database directly (`transcript_events` event count and byte size), triggering compaction when `bytes >= 300KB` or `events >= 40`, regardless of cached token values. + +3. **Compaction User Visibility**: + - Always ensure `agents.defaults.compaction.notifyUser: true` remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (`🧹 Compacting context...`) to avoid perceived bot stalls. + +4. **In-Pod Container Restart Self-Healing**: + - Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (`NODE_OPTIONS="--import=..."`) or application startup hooks. + - Never place critical self-healing logic exclusively in `initContainers`, as Kubelet container restarts inside an existing Pod completely bypass `initContainers`. + +5. **Rotational Storage (HDD) Page-Cache Pre-Warming**: + - Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`). + - Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks. + For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md). ### 1. Verify Deployment Health