diff --git a/.agents/skills/openclaw-ops/SKILL.md b/.agents/skills/openclaw-ops/SKILL.md index 055d3af..7865ef4 100644 --- a/.agents/skills/openclaw-ops/SKILL.md +++ b/.agents/skills/openclaw-ops/SKILL.md @@ -11,6 +11,40 @@ description: Diagnose, troubleshoot, compact, and recover OpenClaw multi-agent s --- +## 0. Instant One-Shot Triage (< 2 Seconds) + +Whenever investigating an issue in a Telegram forum topic or agent, **run the one-shot triage tool FIRST**. +Do NOT run piecemeal SQLite queries, guess column names, or scan the root filesystem. + +### Primary Command: +```bash +# By Telegram forum topic number (e.g. 21=Kochen, 22=Mila, 117=Research) +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic + +# Or by agent name +kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --agent +``` + +The script automatically reports: +1. Ground-truth session status (`done`, `processing`, `running`) and token size. +2. Inbound user prompt (including speech-to-text audio transcripts). +3. Assistant execution trace: thinking excerpts, tool calls with arguments, tool errors (`āŒ`), and final text output. +4. Instant anomaly flags: `[CRITICAL_CONTEXT_BLOAT >60k]`, `[HIGH_CONTEXT_TOKENS >20k]`, `[POSSIBLY_WEDGED_RUNNING]`, `[UNANSWERED USER MESSAGE]`. + +### šŸš« Anti-Patterns & Operational Guardrails: +- āŒ **NEVER run `find /` or broad `grep -r` on the root container filesystem.** Host `haumdaucher` uses spinning mechanical HDDs (`rotational: 1`). Root sweeps take 5+ minutes, freeze I/O, and trigger background tasks. +- āŒ **NEVER guess SQLite table or column names.** + - **`session_nodes`**: `(session_key, current_session_id, status, updated_at, entry_json)` + - **`transcript_events`**: `(session_id, seq, event_json, created_at)` +- šŸ“ **Known Path Anchors (Use Directly):** + - Knowledge Wiki: `/home/node/.openclaw/wiki/main` + - Agent SQLite DBs: `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite` + - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite` + - Agent Workspace: `/home/openclaw/.openclaw/workspace-` + - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`) + +--- + ## 1. Diagnostic Playbook: "Bot is Not Answering / Silent" When a user reports that a Telegram forum topic or direct chat is unresponsive: @@ -173,3 +207,25 @@ kubectl exec openclaw-0 -n openclaw -c openclaw -- openclaw message send \ --message "Dein Nachrichtentext hier..." ``` *(Common topics: Sport = 20, Kochen = 21, Mila = 22, Travel = 51, Shopping = 115, Regensburg = 116, Research = 117, Medizin = 118, IT Guy = 119, Media = 164).* + +--- + +## 6. Post-Troubleshooting Evolution Loop & Simplicity Invariant + +At the end of **every single troubleshooting task, incident investigation, or bugfix**: + +### šŸ”„ Mandatory Dual-Reflection +The AI agent MUST proactively evaluate and document two distinct vectors of improvement before concluding: +1. **Troubleshooting Process Optimization**: + - Did gathering information or isolating the root cause take too long? + - Did any query fail due to syntax, guessing, or slow filesystem sweeps? + - What new diagnostic fastpath, schema shortcut, or check should be codified into `openclaw-ops` or `triage.py` so the next diagnosis takes seconds? +2. **System Maturity & Long-Term Hardening**: + - What fragile assumption or missing guardrail allowed this error to surface? + - How can the prompt, schema, datastore script, or cronjob be hardened permanently so this class of error never recurs? + +### āš–ļø The Simplicity Invariant (The 98% Rule) +- **Simplicity and clean, transparent architecture are the central design principles.** +- Always prefer a simple, pragmatic, easily debuggable solution that covers **98% of the goal** over an intricate, over-engineered abstraction that attempts 100%. +- Avoid speculative background tasks, complex event chains, or hidden state manipulations. Single-responsibility scripts, transparent alerts, and direct deterministic data flows always take precedence. + diff --git a/k8s/openclaw/AGENTS.md b/k8s/openclaw/AGENTS.md index f552b59..0947ea4 100644 --- a/k8s/openclaw/AGENTS.md +++ b/k8s/openclaw/AGENTS.md @@ -92,6 +92,18 @@ OpenClaw connects to three foundational Git repositories: Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules: +0. **Rapid Incident Triaging & Information Gathering (< 2s)**: + - When any issue, silence, or strange bot behavior is reported, **run the one-shot triage tool first**: + ```bash + kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic + ``` + - Do NOT run piecemeal SQLite queries, guess table/column names, or scan the container filesystem. + - **Path Anchors (Never run `find /` or broad `grep -r` on rotational disk):** + - Knowledge Wiki: `/home/node/.openclaw/wiki/main` + - Agent Databases: `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite` + - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite` + - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`) + 1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**: - When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**. - Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first. @@ -112,6 +124,16 @@ Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these ru - Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`). - Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks. +6. **Zero-Downtime Wiki Prompt Updates vs. Heavy Pod Rollouts**: + - Subagent prompt instructions, tone, and tool guidelines live in `knowledge-wiki/agents/.md`. Edits made there are read live by OpenClaw on git sync without any pod restart. + - Avoid modifying `openclaw-instance.secret.yaml` unless modifying container limits, environment variables, or global operator configuration. A CR edit triggers a 7-init-container rolling update (~3 minutes). + +7. **Continuous Troubleshooting Evolution & The Simplicity Invariant (The 98% Rule)**: + - **Post-Fix Dual Reflection**: At the end of every troubleshooting cycle and fix, the AI MUST proactively evaluate: + 1. *Troubleshooting Optimization*: How can the diagnostic process be made faster or codified into `triage.py` / `openclaw-ops`? + 2. *System Maturity*: How can the root cause be hardened permanently to prevent recurrence? + - **Simplicity First**: Simplicity and straightforward architecture are the central design principles. A simple, pragmatic solution that cleanly reaches **98% of the goal** is strictly preferred over an over-engineered abstraction attempting 100%. Never introduce background workers, multi-stage state machines, or silent fallbacks when a transparent single-step tool or direct alert suffices. + For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md). ### 1. Verify Deployment Health diff --git a/k8s/openclaw/scripts/openclaw-triage.py b/k8s/openclaw/scripts/openclaw-triage.py new file mode 100644 index 0000000..1025387 --- /dev/null +++ b/k8s/openclaw/scripts/openclaw-triage.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +""" +openclaw-triage.py: Rapid zero-dependency incident triage for OpenClaw multi-agent sessions. +Directly inspects ground-truth SQLite session state and transcript events in < 1 second. +""" +import argparse +import glob +import json +import os +import sqlite3 +import sys +import time + +TOPIC_TO_AGENT = { + "20": "fitness", + "21": "koch", + "22": "hebamme", + "51": "travel", + "115": "shopping", + "116": "regensburg", + "117": "recherche", + "118": "doktor", + "119": "itguy", + "164": "grabengasse_media", +} + +def parse_args(): + parser = argparse.ArgumentParser(description="Rapid OpenClaw multi-agent session triage.") + parser.add_argument("--topic", "-t", help="Telegram forum topic ID (e.g. 21, 22, 117)") + parser.add_argument("--agent", "-a", help="Agent identifier (e.g. koch, hebamme, recherche)") + parser.add_argument("--limit", "-n", type=int, default=10, help="Number of recent events to display (default: 10)") + parser.add_argument("--db-root", default="/home/node/.openclaw/agents", help="Root path of agent SQLite databases") + return parser.parse_args() + +def triage_agent_session(agent, topic_id=None, limit=10, db_root="/home/node/.openclaw/agents"): + db_path = os.path.join(db_root, agent, "agent", "openclaw-agent.sqlite") + if not os.path.exists(db_path): + print(f"[-] Database not found for agent '{agent}': {db_path}") + return False + + try: + con = sqlite3.connect(db_path, timeout=5.0) + except Exception as e: + print(f"[-] Failed to connect to SQLite DB for agent '{agent}': {e}") + return False + + cur = con.cursor() + # Find matching session + if topic_id: + target_pattern = f"%topic:{topic_id}%" + cur.execute( + "SELECT session_key, current_session_id, status, updated_at, entry_json FROM session_nodes WHERE session_key LIKE ? ORDER BY updated_at DESC LIMIT 1", + (target_pattern,) + ) + else: + cur.execute( + "SELECT session_key, current_session_id, status, updated_at, entry_json FROM session_nodes ORDER BY updated_at DESC LIMIT 1" + ) + + row = cur.fetchone() + if not row: + print(f"[-] No session found in {agent} DB (filter: topic={topic_id})") + con.close() + return False + + skey, sid, status, updated_at_ms, entry_json = row + now_ms = time.time() * 1000 + age_min = (now_ms - updated_at_ms) / 60000.0 if updated_at_ms else 0.0 + + entry = {} + if entry_json: + try: + entry = json.loads(entry_json) + except Exception: + pass + + tokens = entry.get("totalTokens", "?") + model = entry.get("model", "default") + last_run_id = entry.get("activeWriterRunId") or entry.get("activeRunId", None) + + # Detect anomalies + badges = [] + if isinstance(tokens, int): + if tokens > 60000: + badges.append("šŸšØ [CRITICAL_CONTEXT_BLOAT: >60k tokens]") + elif tokens > 20000: + badges.append("āš ļø [HIGH_CONTEXT_TOKENS: >20k tokens]") + if status == "running" and age_min > 5.0: + badges.append("šŸ”’ [POSSIBLY_WEDGED_RUNNING: updated >5m ago]") + elif status == "processing": + badges.append("āš” [ACTIVE_PROCESSING]") + + print("=" * 72) + print(f"šŸŽÆ AGENT: {agent.upper():<12} | TOPIC: {topic_id or 'N/A':<6} | STATUS: {status.upper()}") + print(f" SESSION KEY: {skey}") + print(f" SESSION ID: {sid}") + print(f" MODEL: {model}") + print(f" TOKENS: {tokens} | LAST UPDATED: {age_min:.1f}m ago") + if badges: + print(f" ANOMALIES: {' '.join(badges)}") + print("=" * 72) + + # Query transcript events + cur.execute( + "SELECT seq, created_at, event_json FROM transcript_events WHERE session_id = ? ORDER BY seq DESC LIMIT ?", + (sid, limit) + ) + events = cur.fetchall() + con.close() + + if not events: + print(" (No transcript events found)") + return True + + events.reverse() + print("\nšŸ“œ RECENT TRANSCRIPT EVENTS (Chronological):") + + last_role = None + for seq, ca_ms, ej_str in events: + try: + ev = json.loads(ej_str) + except Exception: + print(f"[{seq}] ") + continue + + etype = ev.get("type") + if etype != "message": + continue + + msg = ev.get("message", {}) + role = msg.get("role") + last_role = role + content = msg.get("content", []) + + if role == "user": + txt = content if isinstance(content, str) else json.dumps(content) + # Check for audio transcript + if isinstance(content, str) and "[Audio]" in content: + print(f"\nšŸ‘‰ [SEQ {seq}] šŸŽ¤ USER AUDIO PROMPT:") + for line in content.splitlines()[:6]: + print(f" {line}") + if len(content.splitlines()) > 6: + print(" ...") + else: + txt_preview = txt.replace("\n", " ")[:240] + print(f"\nšŸ‘‰ [SEQ {seq}] šŸ‘¤ USER: {txt_preview}") + + elif role == "assistant": + print(f"\nšŸ¤– [SEQ {seq}] ASSISTANT TURN:") + if isinstance(content, list): + for item in content: + if not isinstance(item, dict): + continue + itype = item.get("type") + if itype == "thinking": + th = item.get("thinking", "").replace("\n", " ") + print(f" šŸ§  THINKING: {th[:280]}...") + elif itype == "toolCall": + tname = item.get("name", "unknown") + targs = json.dumps(item.get("arguments", {})) + print(f" āš™ļø TOOL CALL: {tname}({targs[:160]})") + elif itype == "text": + tx = item.get("text", "") + lines = tx.splitlines() + prev = lines[0][:150] if lines else "" + print(f" šŸ’¬ TEXT OUTPUT: {prev}...") + elif isinstance(content, str): + print(f" šŸ’¬ TEXT OUTPUT: {content[:200]}...") + + elif role == "toolResult": + is_err = ev.get("isError", False) + tname = ev.get("details", {}).get("tool") or msg.get("toolName", "tool") + res_txt = json.dumps(content) if not isinstance(content, str) else content + if is_err or "error" in res_txt.lower() or "must have required property" in res_txt: + print(f" āŒ [SEQ {seq}] TOOL ERROR ({tname}): {res_txt[:260]}") + else: + print(f" āœ… [SEQ {seq}] TOOL RESULT ({tname}): {res_txt[:180]}") + + if last_role == "user": + print("\nāš ļø WARNING: The last event in the transcript is an UNANSWERED USER MESSAGE!") + print(" The agent turn either crashed, timed out, or was killed before producing a response.") + + print("\n" + "=" * 72) + return True + +def main(): + args = parse_args() + if not args.topic and not args.agent: + print("Usage: openclaw-triage.py (--topic | --agent ) [--limit ]") + sys.exit(1) + + agent = args.agent + topic = args.topic + + if topic and not agent: + agent = TOPIC_TO_AGENT.get(str(topic)) + if not agent: + # Search all DBs + for a, t in TOPIC_TO_AGENT.items(): + if t == topic: + agent = a + break + + if not agent: + # Scan all DBs for the topic + for db in sorted(glob.glob(os.path.join(args.db_root, "*/agent/openclaw-agent.sqlite"))): + cand_agent = db.split(os.sep)[-3] + try: + con = sqlite3.connect(db, timeout=2.0) + cur = con.cursor() + cur.execute( + "SELECT session_key FROM session_nodes WHERE session_key LIKE ? LIMIT 1", + (f"%topic:{topic}%",) + ) + if cur.fetchone(): + agent = cand_agent + con.close() + break + con.close() + except Exception: + pass + + if not agent: + print(f"[-] Could not map topic '{topic}' to any known OpenClaw agent.") + sys.exit(1) + + success = triage_agent_session(agent, topic_id=topic, limit=args.limit, db_root=args.db_root) + sys.exit(0 if success else 1) + +if __name__ == "__main__": + main()