feat(ops): add openclaw-triage tool, instant diagnostic playbook, and simplicity invariant

2026-10-10 14:25:16 +02:00
parent 5b3dfe2a50
commit bb5ca769db
3 changed files with 309 additions and 0 deletions
@@ -11,6 +11,40 @@ description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent s

 ---

+## 0. Instant One-Shot Triage (< 2 Seconds)
+
+Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**.
+Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem.
+
+### Primary Command:
+```bash
+# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research)
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicNumber>
+
+# Or by agent name
+kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent <agentId>
+```
+
+The script automatically reports:
+1. Ground-truth session status (`done`, `processing`, `running`) and token size.
+2. Inbound user prompt (including speech-to-text audio transcripts).
+3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`❌`), and final text output.
+4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`.
+
+### 🚫 Anti-Patterns & Operational Guardrails:
+- ❌ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks.
+- ❌ **NEVER guess SQLite table or column names.**
+  - **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)`
+  - **`transcript_events`**: `(session_id, seq, event_json, created_at)`
+- 📍 **Known Path Anchors (Use Directly):**
+  - Knowledge Wiki: `/home/node/.openclaw/wiki/main`
+  - Agent SQLite DBs: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
+  - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
+  - Agent Workspace: `/home/openclaw/.openclaw/workspace-<agent>`
+  - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
+
+---
+
 ## 1. Diagnostic Playbook: "Bot is Not Answering / Silent"

 When a user reports that a Telegram forum topic or direct chat is unresponsive:
@@ -173,3 +207,25 @@ kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \
  --message "Dein Nachrichtentext hier..."
 ```
 *(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).*
+
+---
+
+## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant
+
+At the end of **every single troubleshooting task, incident investigation, or bugfix**:
+
+### 🔄 Mandatory Dual-Reflection
+The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding:
+1. **Troubleshooting Process Optimization**:
+   - Did gathering information or isolating the root cause take too long?
+   - Did any query fail due to syntax, guessing, or slow filesystem sweeps?
+   - What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds?
+2. **System Maturity & Long-Term Hardening**:
+   - What fragile assumption or missing guardrail allowed this error to surface?
+   - How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs?
+
+### ⚖️ The Simplicity Invariant (The 98% Rule)
+- **Simplicity and clean, transparent architecture are the central design principles.**
+- Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%.
+- Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence.
+
@@ -92,6 +92,18 @@ OpenClaw connects to three foundational Git repositories:

 Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules:

+0. **Rapid Incident Triaging & Information Gathering (< 2s)**:
+   - When any issue, silence, or strange bot behavior is reported, **run the one-shot triage tool first**:
+     ```bash
+     kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicId>
+     ```
+   - Do NOT run piecemeal SQLite queries, guess table/column names, or scan the container filesystem.
+   - **Path Anchors (Never run `find /` or broad `grep -r` on rotational disk):**
+     - Knowledge Wiki: `/home/node/.openclaw/wiki/main`
+     - Agent Databases: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
+     - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
+     - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
+
 1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**:
   - When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**.
   - Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first.
@@ -112,6 +124,16 @@ Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these ru
   - Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`).
   - Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks.

+6. **Zero-Downtime Wiki Prompt Updates vs. Heavy Pod Rollouts**:
+   - Subagent prompt instructions, tone, and tool guidelines live in `knowledge-wiki/agents/<name>.md`. Edits made there are read live by OpenClaw on git sync without any pod restart.
+   - Avoid modifying `openclaw-instance.secret.yaml` unless modifying container limits, environment variables, or global operator configuration. A CR edit triggers a 7-init-container rolling update (~3 minutes).
+
+7. **Continuous Troubleshooting Evolution & The Simplicity Invariant (The 98% Rule)**:
+   - **Post-Fix Dual Reflection**: At the end of every troubleshooting cycle and fix, the AI MUST proactively evaluate:
+     1. *Troubleshooting Optimization*: How can the diagnostic process be made faster or codified into `triage.py` / `openclaw-ops`?
+     2. *System Maturity*: How can the root cause be hardened permanently to prevent recurrence?
+   - **Simplicity First**: Simplicity and straightforward architecture are the central design principles. A simple, pragmatic solution that cleanly reaches **98% of the goal** is strictly preferred over an over-engineered abstraction attempting 100%. Never introduce background workers, multi-stage state machines, or silent fallbacks when a transparent single-step tool or direct alert suffices.
+
 For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md).

 ### 1. Verify Deployment Health
@@ -0,0 +1,231 @@
+#!/usr/bin/env python3
+"""
+openclaw-triage.py: Rapid zero-dependency incident triage for OpenClaw multi-agent sessions.
+Directly inspects ground-truth SQLite session state and transcript events in < 1 second.
+"""
+import argparse
+import glob
+import json
+import os
+import sqlite3
+import sys
+import time
+
+TOPIC_TO_AGENT = {
+    "20": "fitness",
+    "21": "koch",
+    "22": "hebamme",
+    "51": "travel",
+    "115": "shopping",
+    "116": "regensburg",
+    "117": "recherche",
+    "118": "doktor",
+    "119": "itguy",
+    "164": "grabengasse_media",
+}
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="Rapid OpenClaw multi-agent session triage.")
+    parser.add_argument("--topic", "-t", help="Telegram forum topic ID (e.g. 21, 22, 117)")
+    parser.add_argument("--agent", "-a", help="Agent identifier (e.g. koch, hebamme, recherche)")
+    parser.add_argument("--limit", "-n", type=int, default=10, help="Number of recent events to display (default: 10)")
+    parser.add_argument("--db-root", default="/home/node/.openclaw/agents", help="Root path of agent SQLite databases")
+    return parser.parse_args()
+
+def triage_agent_session(agent, topic_id=None, limit=10, db_root="/home/node/.openclaw/agents"):
+    db_path = os.path.join(db_root, agent, "agent", "openclaw-agent.sqlite")
+    if not os.path.exists(db_path):
+        print(f"[-] Database not found for agent '{agent}': {db_path}")
+        return False
+
+    try:
+        con = sqlite3.connect(db_path, timeout=5.0)
+    except Exception as e:
+        print(f"[-] Failed to connect to SQLite DB for agent '{agent}': {e}")
+        return False
+
+    cur = con.cursor()
+    # Find matching session
+    if topic_id:
+        target_pattern = f"%topic:{topic_id}%"
+        cur.execute(
+            "SELECT session_key, current_session_id, status, updated_at, entry_json FROM session_nodes WHERE session_key LIKE ? ORDER BY updated_at DESC LIMIT 1",
+            (target_pattern,)
+        )
+    else:
+        cur.execute(
+            "SELECT session_key, current_session_id, status, updated_at, entry_json FROM session_nodes ORDER BY updated_at DESC LIMIT 1"
+        )
+
+    row = cur.fetchone()
+    if not row:
+        print(f"[-] No session found in {agent} DB (filter: topic={topic_id})")
+        con.close()
+        return False
+
+    skey, sid, status, updated_at_ms, entry_json = row
+    now_ms = time.time() * 1000
+    age_min = (now_ms - updated_at_ms) / 60000.0 if updated_at_ms else 0.0
+
+    entry = {}
+    if entry_json:
+        try:
+            entry = json.loads(entry_json)
+        except Exception:
+            pass
+
+    tokens = entry.get("totalTokens", "?")
+    model = entry.get("model", "default")
+    last_run_id = entry.get("activeWriterRunId") or entry.get("activeRunId", None)
+
+    # Detect anomalies
+    badges = []
+    if isinstance(tokens, int):
+        if tokens > 60000:
+            badges.append("🚨 [CRITICAL_CONTEXT_BLOAT: >60k tokens]")
+        elif tokens > 20000:
+            badges.append("⚠️ [HIGH_CONTEXT_TOKENS: >20k tokens]")
+    if status == "running" and age_min > 5.0:
+        badges.append("🔒 [POSSIBLY_WEDGED_RUNNING: updated >5m ago]")
+    elif status == "processing":
+        badges.append("⚡ [ACTIVE_PROCESSING]")
+
+    print("=" * 72)
+    print(f"🎯 AGENT: {agent.upper():<12} | TOPIC: {topic_id or 'N/A':<6} | STATUS: {status.upper()}")
+    print(f"   SESSION KEY: {skey}")
+    print(f"   SESSION ID:  {sid}")
+    print(f"   MODEL:       {model}")
+    print(f"   TOKENS:      {tokens} | LAST UPDATED: {age_min:.1f}m ago")
+    if badges:
+        print(f"   ANOMALIES:   {' '.join(badges)}")
+    print("=" * 72)
+
+    # Query transcript events
+    cur.execute(
+        "SELECT seq, created_at, event_json FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT ?",
+        (sid, limit)
+    )
+    events = cur.fetchall()
+    con.close()
+
+    if not events:
+        print("   (No transcript events found)")
+        return True
+
+    events.reverse()
+    print("\n📜 RECENT TRANSCRIPT EVENTS (Chronological):")
+    
+    last_role = None
+    for seq, ca_ms, ej_str in events:
+        try:
+            ev = json.loads(ej_str)
+        except Exception:
+            print(f"[{seq}] <unparseable event json>")
+            continue
+
+        etype = ev.get("type")
+        if etype != "message":
+            continue
+
+        msg = ev.get("message", {})
+        role = msg.get("role")
+        last_role = role
+        content = msg.get("content", [])
+
+        if role == "user":
+            txt = content if isinstance(content, str) else json.dumps(content)
+            # Check for audio transcript
+            if isinstance(content, str) and "[Audio]" in content:
+                print(f"\n👉 [SEQ {seq}] 🎤 USER AUDIO PROMPT:")
+                for line in content.splitlines()[:6]:
+                    print(f"   {line}")
+                if len(content.splitlines()) > 6:
+                    print("   ...")
+            else:
+                txt_preview = txt.replace("\n", " ")[:240]
+                print(f"\n👉 [SEQ {seq}] 👤 USER: {txt_preview}")
+
+        elif role == "assistant":
+            print(f"\n🤖 [SEQ {seq}] ASSISTANT TURN:")
+            if isinstance(content, list):
+                for item in content:
+                    if not isinstance(item, dict):
+                        continue
+                    itype = item.get("type")
+                    if itype == "thinking":
+                        th = item.get("thinking", "").replace("\n", " ")
+                        print(f"   🧠 THINKING: {th[:280]}...")
+                    elif itype == "toolCall":
+                        tname = item.get("name", "unknown")
+                        targs = json.dumps(item.get("arguments", {}))
+                        print(f"   ⚙️ TOOL CALL: {tname}({targs[:160]})")
+                    elif itype == "text":
+                        tx = item.get("text", "")
+                        lines = tx.splitlines()
+                        prev = lines[0][:150] if lines else ""
+                        print(f"   💬 TEXT OUTPUT: {prev}...")
+            elif isinstance(content, str):
+                print(f"   💬 TEXT OUTPUT: {content[:200]}...")
+
+        elif role == "toolResult":
+            is_err = ev.get("isError", False)
+            tname = ev.get("details", {}).get("tool") or msg.get("toolName", "tool")
+            res_txt = json.dumps(content) if not isinstance(content, str) else content
+            if is_err or "error" in res_txt.lower() or "must have required property" in res_txt:
+                print(f"   ❌ [SEQ {seq}] TOOL ERROR ({tname}): {res_txt[:260]}")
+            else:
+                print(f"   ✅ [SEQ {seq}] TOOL RESULT ({tname}): {res_txt[:180]}")
+
+    if last_role == "user":
+        print("\n⚠️ WARNING: The last event in the transcript is an UNANSWERED USER MESSAGE!")
+        print("   The agent turn either crashed, timed out, or was killed before producing a response.")
+
+    print("\n" + "=" * 72)
+    return True
+
+def main():
+    args = parse_args()
+    if not args.topic and not args.agent:
+        print("Usage: openclaw-triage.py (--topic <ID> | --agent <NAME>) [--limit <N>]")
+        sys.exit(1)
+
+    agent = args.agent
+    topic = args.topic
+
+    if topic and not agent:
+        agent = TOPIC_TO_AGENT.get(str(topic))
+        if not agent:
+            # Search all DBs
+            for a, t in TOPIC_TO_AGENT.items():
+                if t == topic:
+                    agent = a
+                    break
+
+    if not agent:
+        # Scan all DBs for the topic
+        for db in sorted(glob.glob(os.path.join(args.db_root, "*/agent/openclaw-agent.sqlite"))):
+            cand_agent = db.split(os.sep)[-3]
+            try:
+                con = sqlite3.connect(db, timeout=2.0)
+                cur = con.cursor()
+                cur.execute(
+                    "SELECT session_key FROM session_nodes WHERE session_key LIKE ? LIMIT 1",
+                    (f"%topic:{topic}%",)
+                )
+                if cur.fetchone():
+                    agent = cand_agent
+                    con.close()
+                    break
+                con.close()
+            except Exception:
+                pass
+
+    if not agent:
+        print(f"[-] Could not map topic '{topic}' to any known OpenClaw agent.")
+        sys.exit(1)
+
+    success = triage_agent_session(agent, topic_id=topic, limit=args.limit, db_root=args.db_root)
+    sys.exit(0 if success else 1)
+
+if __name__ == "__main__":
+    main()