feat(openclaw): persist operational invariants in AGENTS.md and add openclaw-ops skill runbook
This commit is contained in:
Moritz Graf
@@ -0,0 +1,175 @@
|
||||
---
|
||||
name: openclaw-ops
|
||||
license: Apache-2.0
|
||||
description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram.
|
||||
---
|
||||
|
||||
# OpenClaw Operations & Troubleshooting Runbook
|
||||
|
||||
> **Target Environment**: Kubernetes cluster `haumdaucher`, namespace `openclaw`, pod `openclaw-0`.
|
||||
> **Key References**: [`k8s/openclaw/AGENTS.md`](../../../k8s/openclaw/AGENTS.md) and [`k8s/openclaw/TROUBLESHOOTING.md`](../../../k8s/openclaw/TROUBLESHOOTING.md).
|
||||
|
||||
---
|
||||
|
||||
## 1. Diagnostic Playbook: "Bot is Not Answering / Silent"
|
||||
|
||||
When a user reports that a Telegram forum topic or direct chat is unresponsive:
|
||||
|
||||
### ⚠️ Golden Rule: Do NOT Assume Container Crash
|
||||
Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (`5/5 Running`).
|
||||
|
||||
### Step 1: Check Live Logs for In-Flight Turns
|
||||
```bash
|
||||
kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100
|
||||
```
|
||||
Look for:
|
||||
- `compact:start reason=auto-threshold`: Inline safeguard compaction in progress (takes 30–60s).
|
||||
- `embedded run tool start ... tool=write|exec`: Agent is actively executing tool steps.
|
||||
- `[model-fetch] start ... status=200`: Waiting on LLM token streaming.
|
||||
- `heartbeat: webhooks=... active=X`: Active turn count in gateway.
|
||||
|
||||
### Step 2: Query Ground-Truth Session Status in SQLite
|
||||
Each agent maintains its own SQLite database at `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`.
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import sqlite3, glob
|
||||
for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
|
||||
agent = db.split('/')[5]
|
||||
con = sqlite3.connect(db)
|
||||
rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall()
|
||||
if rows:
|
||||
print(f'[{agent}]', rows)
|
||||
"
|
||||
```
|
||||
- If `status == "processing"`: Turn is actively executing.
|
||||
- If `status == "done"`: Turn completed. Check outbound Telegram dispatch.
|
||||
- If `status == "running"` with old `updated_at`: Session is wedged (see Recovery below).
|
||||
|
||||
### Step 3: Inspect Recent Transcript Events
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import sqlite3, json
|
||||
con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
|
||||
sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('<sessionKey>',)).fetchone()[0]
|
||||
for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall():
|
||||
print(seq, ca, ej)
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Session Recovery & Lock Clearing
|
||||
|
||||
### Un-wedge a Stuck Session Node
|
||||
If a turn was killed mid-execution (e.g. OOMKill or node reboot):
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import sqlite3
|
||||
con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
|
||||
con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('<sessionKey>',))
|
||||
con.commit()
|
||||
print('Session un-wedged successfully.')
|
||||
"
|
||||
```
|
||||
|
||||
### Clean Stale State Locks
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock
|
||||
```
|
||||
|
||||
### Check Foreign Key Violations & Clear Quarantine
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import sqlite3, glob
|
||||
for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'):
|
||||
con = sqlite3.connect(db)
|
||||
fk = con.execute('PRAGMA foreign_key_check;').fetchall()
|
||||
if fk:
|
||||
print(f'FK violations in {db}:', fk)
|
||||
"
|
||||
# Clear quarantine table if present
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import sqlite3, os
|
||||
p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite'
|
||||
if os.path.exists(p):
|
||||
c = sqlite3.connect(p)
|
||||
c.execute('DELETE FROM quarantined_databases')
|
||||
c.commit()
|
||||
print('Quarantine cleared.')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Multi-Agent Compaction & Transcript Bloat Control
|
||||
|
||||
### Pitfall: Stale Sessions Report `totalTokens: null`
|
||||
`openclaw sessions --all-agents --json` returns `totalTokens: null` (`totalTokensFresh: false`) for sessions that haven't received a message in several days. **Always inspect SQLite directly**:
|
||||
```bash
|
||||
# Check size of active sessions across all agents
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import glob, sqlite3, json
|
||||
for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
|
||||
agent = db.split('/')[5]
|
||||
con = sqlite3.connect(db)
|
||||
for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall():
|
||||
cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone()
|
||||
if bsz > 300000 or cnt > 40:
|
||||
print(f'{agent:12} {bsz//1024:6} KB {cnt:4} events {skey}')
|
||||
"
|
||||
```
|
||||
|
||||
### Manually Compact a Stale or Bloated Session
|
||||
```bash
|
||||
# LLM summarization with 3-minute timeout
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --timeout 180000
|
||||
|
||||
# Truncation fallback (if LLM summarization times out)
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --max-lines 80 --timeout 180000
|
||||
```
|
||||
|
||||
### Ensure User Compaction Notification is Enabled
|
||||
To prevent silent user-perceived freezes during daytime turns:
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Mechanical HDD / Page Cache Pre-Warming
|
||||
|
||||
Node `haumdaucher` uses spinning HDDs (`rotational: 1`). Cold database opens take up to 20–50 minutes without page cache warming.
|
||||
|
||||
### Warm Page Cache Instantly (<1s)
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \
|
||||
"cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true"
|
||||
```
|
||||
|
||||
### WAL Checkpoints & SQLite Optimization
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
|
||||
import glob, sqlite3
|
||||
dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')
|
||||
for db in dbs:
|
||||
con = sqlite3.connect(db)
|
||||
con.execute('PRAGMA wal_checkpoint(TRUNCATE);')
|
||||
con.execute('PRAGMA optimize;')
|
||||
con.close()
|
||||
print('WAL checkpoints and optimizations complete.')
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Emergency Telegram Direct Message Bypass
|
||||
|
||||
If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn:
|
||||
```bash
|
||||
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
|
||||
--channel telegram \
|
||||
--target "-1004326921510" \
|
||||
--thread-id <topicNumber> \
|
||||
--message "Dein Nachrichtentext hier..."
|
||||
```
|
||||
*(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).*
|
||||
@@ -88,6 +88,30 @@ OpenClaw connects to three foundational Git repositories:
|
||||
> [!IMPORTANT]
|
||||
> **Mandatory Bug & Incident Documentation**: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, **always document the incident in [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)** with symptom, root cause, resolution, and how to avoid recurrence.
|
||||
|
||||
### 🛡️ Operational & Diagnostic Invariants (Learned Guidelines)
|
||||
|
||||
Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules:
|
||||
|
||||
1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**:
|
||||
- When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**.
|
||||
- Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first.
|
||||
- Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (`compact:start`) can introduce multi-minute turn latency without any pod restart.
|
||||
|
||||
2. **Datastore Ground-Truth for Maintenance**:
|
||||
- High-level CLI summaries (`openclaw sessions --all-agents --json`) report `totalTokens: null` (`totalTokensFresh: false`) for stale sessions or after restarts.
|
||||
- Automated health checks and compaction scripts MUST query the SQLite database directly (`transcript_events` event count and byte size), triggering compaction when `bytes >= 300KB` or `events >= 40`, regardless of cached token values.
|
||||
|
||||
3. **Compaction User Visibility**:
|
||||
- Always ensure `agents.defaults.compaction.notifyUser: true` remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (`🧹 Compacting context...`) to avoid perceived bot stalls.
|
||||
|
||||
4. **In-Pod Container Restart Self-Healing**:
|
||||
- Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (`NODE_OPTIONS="--import=..."`) or application startup hooks.
|
||||
- Never place critical self-healing logic exclusively in `initContainers`, as Kubelet container restarts inside an existing Pod completely bypass `initContainers`.
|
||||
|
||||
5. **Rotational Storage (HDD) Page-Cache Pre-Warming**:
|
||||
- Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`).
|
||||
- Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks.
|
||||
|
||||
For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md).
|
||||
|
||||
### 1. Verify Deployment Health
|
||||
|
||||
Reference in New Issue
Block a user