feat(ops): add openclaw-triage tool, instant diagnostic playbook, and simplicity invariant

This commit is contained in:
Moritz Graf
2026-10-10 14:25:16 +02:00
parent 5b3dfe2a50
commit bb5ca769db
3 changed files with 309 additions and 0 deletions
+56
View File
@@ -11,6 +11,40 @@ description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent s
---
## 0. Instant One-Shot Triage (< 2 Seconds)
Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**.
Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem.
### Primary Command:
```bash
# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research)
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicNumber>
# Or by agent name
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent <agentId>
```
The script automatically reports:
1. Ground-truth session status (`done`, `processing`, `running`) and token size.
2. Inbound user prompt (including speech-to-text audio transcripts).
3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`❌`), and final text output.
4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`.
### 🚫 Anti-Patterns & Operational Guardrails:
- ❌ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks.
- ❌ **NEVER guess SQLite table or column names.**
- **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)`
- **`transcript_events`**: `(session_id, seq, event_json, created_at)`
- 📍 **Known Path Anchors (Use Directly):**
- Knowledge Wiki: `/home/node/.openclaw/wiki/main`
- Agent SQLite DBs: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
- Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
- Agent Workspace: `/home/openclaw/.openclaw/workspace-<agent>`
- Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
---
## 1. Diagnostic Playbook: "Bot is Not Answering / Silent"
When a user reports that a Telegram forum topic or direct chat is unresponsive:
@@ -173,3 +207,25 @@ kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
--message "Dein Nachrichtentext hier..."
```
*(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).*
---
## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant
At the end of **every single troubleshooting task, incident investigation, or bugfix**:
### 🔄 Mandatory Dual-Reflection
The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding:
1. **Troubleshooting Process Optimization**:
- Did gathering information or isolating the root cause take too long?
- Did any query fail due to syntax, guessing, or slow filesystem sweeps?
- What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds?
2. **System Maturity & Long-Term Hardening**:
- What fragile assumption or missing guardrail allowed this error to surface?
- How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs?
### ⚖️ The Simplicity Invariant (The 98% Rule)
- **Simplicity and clean, transparent architecture are the central design principles.**
- Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%.
- Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.