infrapuzzle/.agents/skills/openclaw-ops/SKILL.md

name, license, description

name	license	description
openclaw-ops	Apache-2.0	Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent sessions, SQLite state, and Telegram topic channels. Use when investigating why a Telegram topic is silent or not answering, inspecting session nodes and transcript events in SQLite, recovering wedged or locked sessions, running multi-agent transcript compactions, checking and clearing database quarantines or lockfiles, pre-warming Linux page cache on mechanical HDDs, or sending out-of-band direct messages to Telegram.

OpenClaw Operations & Troubleshooting Runbook

Target Environment: Kubernetes cluster haumdaucher, namespace openclaw, pod openclaw-0.
Key References: k8s/openclaw/AGENTS.md and k8s/openclaw/TROUBLESHOOTING.md.

---

0. Instant One-Shot Triage (< 2 Seconds)

Whenever investigating an issue in a Telegram forum topic or agent, run the one-shot triage tool FIRST. Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem.

Primary Command:

# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research)
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicNumber>

# Or by agent name
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent <agentId>

The script automatically reports:

1. Ground-truth session status (done, processing, running) and token size.
2. Inbound user prompt (including speech-to-text audio transcripts).
3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (❌), and final text output.
4. Instant anomaly flags: [CRITICAL_CONTEXT_BLOAT >60k], [HIGH_CONTEXT_TOKENS >20k], [POSSIBLY_WEDGED_RUNNING], [UNANSWERED USER MESSAGE].

🚫 Anti-Patterns & Operational Guardrails:

• ❌ NEVER run find / or broad grep -r on the root container filesystem. Host haumdaucher uses spinning mechanical HDDs (rotational: 1). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks.
• ❌ NEVER guess SQLite table or column names.
  • session_nodes: (session_key, current_session_id, status, updated_at, entry_json)
  • transcript_events: (session_id, seq, event_json, created_at)
• 📍 Known Path Anchors (Use Directly):
  • Knowledge Wiki: /home/node/.openclaw/wiki/main
  • Agent SQLite DBs: /home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite
  • Gateway State DB: /home/node/.openclaw/state/openclaw.sqlite
  • Agent Workspace: /home/openclaw/.openclaw/workspace-<agent>
  • Standalone Triage Script: /home/node/.openclaw/scripts/triage.py (Repo: k8s/openclaw/scripts/openclaw-triage.py)

---

1. Diagnostic Playbook: "Bot is Not Answering / Silent"

When a user reports that a Telegram forum topic or direct chat is unresponsive:

⚠️ Golden Rule: Do NOT Assume Container Crash

Long turns (multi-step tools, deep research, or inline compaction) can cause multi-minute delays while the container is perfectly healthy (5/5 Running).

Step 1: Check Live Logs for In-Flight Turns

kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100

Look for:

• compact:start reason=auto-threshold: Inline safeguard compaction in progress (takes 30–60s).
• embedded run tool start ... tool=write|exec: Agent is actively executing tool steps.
• [model-fetch] start ... status=200: Waiting on LLM token streaming.
• heartbeat: webhooks=... active=X: Active turn count in gateway.

Step 2: Query Ground-Truth Session Status in SQLite

Each agent maintains its own SQLite database at /home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite.

kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import sqlite3, glob
for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
    agent = db.split('/')[5]
    con = sqlite3.connect(db)
    rows = con.execute('SELECT session_key, status, active_writer_run_id, updated_at FROM session_nodes WHERE session_key LIKE \"%topic:20%\"').fetchall()
    if rows:
        print(f'[{agent}]', rows)
"

• If status == "processing": Turn is actively executing.
• If status == "done": Turn completed. Check outbound Telegram dispatch.
• If status == "running" with old updated_at: Session is wedged (see Recovery below).

Step 3: Inspect Recent Transcript Events

kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import sqlite3, json
con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
sid = con.execute('SELECT current_session_id FROM session_nodes WHERE session_key = ?', ('<sessionKey>',)).fetchone()[0]
for seq, ca, ej in con.execute('SELECT seq, created_at, substr(event_json, 1, 250) FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT 5', (sid,)).fetchall():
    print(seq, ca, ej)
"

---

2. Session Recovery & Lock Clearing

Un-wedge a Stuck Session Node

If a turn was killed mid-execution (e.g. OOMKill or node reboot):

kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import sqlite3
con = sqlite3.connect('/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite')
con.execute('UPDATE session_nodes SET status = \"done\", active_writer_run_id = NULL, entry_valid = 1 WHERE session_key = ?', ('<sessionKey>',))
con.commit()
print('Session un-wedged successfully.')
"

Clean Stale State Locks

kubectl exec openclaw-0 -n openclaw -c openclaw -- rm -f /tmp/openclaw-1000/gateway.state.lock /tmp/.openclaw-1000/gateway.state.lock

Check Foreign Key Violations & Clear Quarantine

kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import sqlite3, glob
for db in glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite'):
    con = sqlite3.connect(db)
    fk = con.execute('PRAGMA foreign_key_check;').fetchall()
    if fk:
        print(f'FK violations in {db}:', fk)
"
# Clear quarantine table if present
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import sqlite3, os
p = '/home/node/.openclaw/state/openclaw-quarantine.sqlite'
if os.path.exists(p):
    c = sqlite3.connect(p)
    c.execute('DELETE FROM quarantined_databases')
    c.commit()
    print('Quarantine cleared.')
"

---

3. Multi-Agent Compaction & Transcript Bloat Control

Pitfall: Stale Sessions Report totalTokens: null

openclaw sessions --all-agents --json returns totalTokens: null (totalTokensFresh: false) for sessions that haven't received a message in several days. Always inspect SQLite directly:

# Check size of active sessions across all agents
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import glob, sqlite3, json
for db in sorted(glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')):
    agent = db.split('/')[5]
    con = sqlite3.connect(db)
    for skey, sid, ej in con.execute('SELECT session_key, current_session_id, entry_json FROM session_nodes WHERE session_key LIKE \"%:telegram:%\"').fetchall():
        cnt, bsz = con.execute('SELECT count(*), coalesce(sum(length(event_json)), 0) FROM transcript_events WHERE session_id=?', (sid,)).fetchone()
        if bsz > 300000 or cnt > 40:
            print(f'{agent:12} {bsz//1024:6} KB  {cnt:4} events  {skey}')
"

Manually Compact a Stale or Bloated Session

# LLM summarization with 3-minute timeout
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --timeout 180000

# Truncation fallback (if LLM summarization times out)
kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw sessions compact "<sessionKey>" --agent <agentId> --max-lines 80 --timeout 180000

Ensure User Compaction Notification is Enabled

To prevent silent user-perceived freezes during daytime turns:

kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw config set agents.defaults.compaction.notifyUser true --strict-json

---

4. Mechanical HDD / Page Cache Pre-Warming

Node haumdaucher uses spinning HDDs (rotational: 1). Cold database opens take up to 20–50 minutes without page cache warming.

Warm Page Cache Instantly (<1s)

kubectl exec openclaw-0 -n openclaw -c openclaw -- /bin/sh -c \
  "cat /home/node/.openclaw/state/*.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true"

WAL Checkpoints & SQLite Optimization

kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c "
import glob, sqlite3
dbs = ['/home/node/.openclaw/state/openclaw.sqlite'] + glob.glob('/home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite')
for db in dbs:
    con = sqlite3.connect(db)
    con.execute('PRAGMA wal_checkpoint(TRUNCATE);')
    con.execute('PRAGMA optimize;')
    con.close()
print('WAL checkpoints and optimizations complete.')
"

---

5. Emergency Telegram Direct Message Bypass

If OpenClaw's turn engine failed to send a message or an urgent human notification is required without spawning a conversation turn:

kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
  --channel telegram \
  --target "-1004326921510" \
  --thread-id <topicNumber> \
  --message "Dein Nachrichtentext hier..."

(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).

---

6. Post-Troubleshooting Evolution Loop & Simplicity Invariant

At the end of every single troubleshooting task, incident investigation, or bugfix:

🔄 Mandatory Dual-Reflection

The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding:

1. Troubleshooting Process Optimization:
   • Did gathering information or isolating the root cause take too long?
   • Did any query fail due to syntax, guessing, or slow filesystem sweeps?
   • What new diagnostic fastpath, schema shortcut, or check should be codified into openclaw-ops or triage.py so the next diagnosis takes seconds?
2. System Maturity & Long-Term Hardening:
   • What fragile assumption or missing guardrail allowed this error to surface?
   • How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs?

⚖️ The Simplicity Invariant (The 98% Rule)

• Simplicity and clean, transparent architecture are the central design principles.
• Always prefer a simple, pragmatic, easily debuggable solution that covers 98% of the goal over an intricate, over-engineered abstraction that attempts 100%.
• Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.