--- name: openclaw-ops license: Apache-2.0 description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram. --- # OpenClaw Operations & Troubleshooting Runbook > **Target Environment**: Kubernetes cluster `haumdaucher`, namespace `openclaw`, pod `openclaw-0`. > **Key References**: [`k8s/openclaw/AGENTS.md`](../../../k8s/openclaw/AGENTS.md) and [`k8s/openclaw/TROUBLESHOOTING.md`](../../../k8s/openclaw/TROUBLESHOOTING.md). --- ## 0. Instant One-Shot Triage (< 2 Seconds) Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**. Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem. ### Primary Command: ```bash # By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research) kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic # Or by agent name kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent ``` The script automatically reports: 1. Ground-truth session status (`done`, `processing`, `running`) and token size. 2. Inbound user prompt (including speech-to-text audio transcripts). 3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`❌`), and final text output. 4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`. ### 🚫 Anti-Patterns & Operational Guardrails: - ❌ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks. - ❌ **NEVER guess SQLite table or column names.** - **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)` - **`transcript_events`**: `(session_id, seq, event_json, created_at)` - 📍 **Known Path Anchors (Use Directly):** - Knowledge Wiki: `/home/node/.openclaw/wiki/main` - Agent SQLite DBs: `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite` - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite` - Agent Workspace: `/home/openclaw/.openclaw/workspace-` - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`) --- ## 1. Diagnostic Playbook: "Bot is Not Answering / Silent" When a user reports that a Telegram forum topic or direct chat is unresponsive: ### ⚠️ Golden Rule: Do NOT Assume Container Crash Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (`5/5 Running`). ### Step 1: Check Live Logs for In-Flight Turns ```bash kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100 ``` Look for: - `compact:start reason=auto-threshold`: Inline safeguard compaction in progress (takes 30–60s). - `embedded run tool start ... tool=write|exec`: Agent is actively executing tool steps. - `[model-fetch] start ... status=200`: Waiting on LLM token streaming. - `heartbeat: webhooks=... active=X`: Active turn count in gateway. ### Step 2: Query Ground-Truth Session Status in SQLite Each agent maintains its own SQLite database at `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite`. ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import sqlite3, glob for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')): agent = db.split('/')[5] con = sqlite3.connect(db) rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall() if rows: print(f'[{agent}]', rows) " ``` - If `status == "processing"`: Turn is actively executing. - If `status == "done"`: Turn completed. Check outbound Telegram dispatch. - If `status == "running"` with old `updated_at`: Session is wedged (see Recovery below). ### Step 3: Inspect Recent Transcript Events ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import sqlite3, json con = sqlite3.connect('/home/node/.openclaw/agents//agent/openclaw-agent.sqlite') sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('',)).fetchone()[0] for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall(): print(seq, ca, ej) " ``` --- ## 2. Session Recovery & Lock Clearing ### Un-wedge a Stuck Session Node If a turn was killed mid-execution (e.g. OOMKill or node reboot): ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import sqlite3 con = sqlite3.connect('/home/node/.openclaw/agents//agent/openclaw-agent.sqlite') con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('',)) con.commit() print('Session un-wedged successfully.') " ``` ### Clean Stale State Locks ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock ``` ### Check Foreign Key Violations & Clear Quarantine ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import sqlite3, glob for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'): con = sqlite3.connect(db) fk = con.execute('PRAGMA foreign_key_check;').fetchall() if fk: print(f'FK violations in {db}:', fk) " # Clear quarantine table if present kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import sqlite3, os p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite' if os.path.exists(p): c = sqlite3.connect(p) c.execute('DELETE FROM quarantined_databases') c.commit() print('Quarantine cleared.') " ``` --- ## 3. Multi-Agent Compaction & Transcript Bloat Control ### Pitfall: Stale Sessions Report `totalTokens: null` `openclaw sessions --all-agents --json` returns `totalTokens: null` (`totalTokensFresh: false`) for sessions that haven't received a message in several days. **Always inspect SQLite directly**: ```bash # Check size of active sessions across all agents kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import glob, sqlite3, json for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')): agent = db.split('/')[5] con = sqlite3.connect(db) for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall(): cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone() if bsz > 300000 or cnt > 40: print(f'{agent:12} {bsz//1024:6} KB {cnt:4} events {skey}') " ``` ### Manually Compact a Stale or Bloated Session ```bash # LLM summarization with 3-minute timeout kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "" --agent --timeout 180000 # Truncation fallback (if LLM summarization times out) kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "" --agent --max-lines 80 --timeout 180000 ``` ### Ensure User Compaction Notification is Enabled To prevent silent user-perceived freezes during daytime turns: ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json ``` --- ## 4. Mechanical HDD / Page Cache Pre-Warming Node `haumdaucher` uses spinning HDDs (`rotational: 1`). Cold database opens take up to 20–50 minutes without page cache warming. ### Warm Page Cache Instantly (<1s) ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \ "cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true" ``` ### WAL Checkpoints & SQLite Optimization ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c " import glob, sqlite3 dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite') for db in dbs: con = sqlite3.connect(db) con.execute('PRAGMA wal_checkpoint(TRUNCATE);') con.execute('PRAGMA optimize;') con.close() print('WAL checkpoints and optimizations complete.') " ``` --- ## 5. Emergency Telegram Direct Message Bypass If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn: ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \ --channel telegram \ --target "-1004326921510" \ --thread-id \ --message "Dein Nachrichtentext hier..." ``` *(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).* --- ## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant At the end of **every single troubleshooting task, incident investigation, or bugfix**: ### 🔄 Mandatory Dual-Reflection The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding: 1. **Troubleshooting Process Optimization**: - Did gathering information or isolating the root cause take too long? - Did any query fail due to syntax, guessing, or slow filesystem sweeps? - What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds? 2. **System Maturity & Long-Term Hardening**: - What fragile assumption or missing guardrail allowed this error to surface? - How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs? ### ⚖️ The Simplicity Invariant (The 98% Rule) - **Simplicity and clean, transparent architecture are the central design principles.** - Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%. - Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.