feat(skills): add openclaw-audit system diagnostic and commit correlation skill
This commit is contained in:
Moritz Graf
@@ -0,0 +1,69 @@
|
||||
---
|
||||
name: openclaw-audit
|
||||
description: Audit OpenClaw runtime logs, session transcripts, tool failures, and correlate errors with recent git commits across infrapuzzle, mop-skills, and knowledge-wiki. Use when performing system health audits, investigating tool failures, or running the weekly IT Guy self-audit.
|
||||
---
|
||||
|
||||
# 🩺 OpenClaw System & Log Self-Audit (`openclaw-audit`)
|
||||
|
||||
Comprehensive system diagnostic and correlation engine designed for **IT Guy** (`itguy`). It analyzes runtime container stdout logs, session transcripts, and tool execution failures across all agents, correlates errors with recent Git commits across the 3 core repositories, and files structured audit reports and actionable task queue entries into the Knowledge Wiki.
|
||||
|
||||
## 🎯 When to Use
|
||||
- **Weekly Scheduled Audit**: Triggered automatically via OpenClaw Cron (`itguy-weekly-health-audit`) every Sunday at 03:00 Europe/Berlin.
|
||||
- **On-Demand Diagnostics**: Whenever the user asks: *"Prüfe die Logs von OpenClaw"*, *"Gibt es Tool-Fehler?"*, *"Audit durchführen"*, *"System-Health prüfen"*.
|
||||
- **Post-Deployment Validation**: After making configuration, skill, or manifest changes to verify whether errors appeared.
|
||||
|
||||
---
|
||||
|
||||
## 🗂️ The Three Core Repositories Architecture
|
||||
The audit correlates runtime behavior with recent git commits across three foundational repositories:
|
||||
|
||||
1. **`infrapuzzle`** (`/home/node/infrapuzzle/infrapuzzle` or `/home/node/.openclaw/repos/infrapuzzle`):
|
||||
- Scope: Cluster infrastructure, Kubespray bootstrap, K8s manifests, Terraform DNS, ESPHome IoT, ADRs.
|
||||
2. **`mop-skills`** (`/home/node/mop-skills/mop-skills`):
|
||||
- Scope: Custom Python skills, CLI utilities, and MCP server bridges.
|
||||
3. **`knowledge-wiki`** (`/home/node/.openclaw/wiki/main`):
|
||||
- Scope: Agents-as-code (`agents/*.md`), active task queue (`tasks/antigravity.md`), family knowledge vault, and audit archives (`operations/audits/`).
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ CLI Operations Reference
|
||||
|
||||
### 1. Standard Audit (Inspection Only)
|
||||
```bash
|
||||
uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d
|
||||
```
|
||||
Fetches container logs from the last 7 days, parses session transcripts across all agents, summarizes recent commits from all 3 git repos, and outputs a formatted diagnostic markdown report to stdout.
|
||||
|
||||
### 2. Full Audit with Wiki Archival & Task Filing
|
||||
```bash
|
||||
uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py \
|
||||
--since 7d \
|
||||
--write-report \
|
||||
--add-task \
|
||||
--git-push
|
||||
```
|
||||
- **`--write-report`**: Writes the detailed diagnostic report to `/home/node/.openclaw/wiki/main/operations/audits/YYYY-Wxx.md`.
|
||||
- **`--add-task`**: Appends any newly detected, deduplicated high/critical issues to `/home/node/.openclaw/wiki/main/tasks/antigravity.md` under `## 📥 Pending Tasks`.
|
||||
- **`--git-push`**: Commits and pushes the modified wiki pages directly to `git.moritzgraf.de/moritz/knowledge-wiki.git`.
|
||||
|
||||
### 3. Machine-Readable JSON Output
|
||||
```bash
|
||||
uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 24h --json
|
||||
```
|
||||
Returns a structured JSON payload containing `status`, `recent_commits`, `error_clusters`, `session_tool_failures`, and `affected_agents`.
|
||||
|
||||
---
|
||||
|
||||
## 🤖 Protocol for IT Guy Agent (`itguy`)
|
||||
|
||||
When executing the weekly audit turn:
|
||||
1. Run the audit script:
|
||||
```bash
|
||||
uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d --write-report --add-task --git-push
|
||||
```
|
||||
2. Review the aggregated output:
|
||||
- Check if any new issues were added to `tasks/antigravity.md`.
|
||||
- Correlate any error spikes with commits in `infrapuzzle` or `mop-skills`.
|
||||
3. Delivery Decision:
|
||||
- **If issues were found and tasks added**: Send a concise summary of findings, affected agents/tools, and links to the wiki task queue directly into Telegram Topic `#IT Guy` (thread `119`).
|
||||
- **If zero issues / all healthy**: Respond with **`NO_REPLY`** to maintain intentional silence (no Telegram notification spam).
|
||||
+622
@@ -0,0 +1,622 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
OpenClaw System, Log & Repository Self-Audit Engine
|
||||
Analyzes container logs, session transcripts, and correlates error events
|
||||
with recent Git commits across infrapuzzle, mop-skills, and knowledge-wiki.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
# Default search directories (in-cluster vs local fallback)
|
||||
DEFAULT_CANDIDATE_PATHS = {
|
||||
"infrapuzzle": [
|
||||
Path("/home/node/infrapuzzle/infrapuzzle"),
|
||||
Path("/home/node/.openclaw/repos/infrapuzzle"),
|
||||
Path("/home/node/infrapuzzle"),
|
||||
Path("/Users/moritz/src/infrapuzzle"),
|
||||
],
|
||||
"mop-skills": [
|
||||
Path("/home/node/mop-skills/mop-skills"),
|
||||
Path("/Users/moritz/src/mop-skills"),
|
||||
],
|
||||
"knowledge-wiki": [
|
||||
Path("/home/node/.openclaw/wiki/main"),
|
||||
Path("/Users/moritz/src/infrapuzzle/knowledge-wiki"),
|
||||
],
|
||||
}
|
||||
|
||||
DEFAULT_SESSIONS_DIRS = [
|
||||
Path("/home/node/.openclaw/agents"),
|
||||
Path.home() / ".openclaw/agents",
|
||||
]
|
||||
|
||||
|
||||
def resolve_repo_path(repo_name: str, override: Optional[str] = None) -> Optional[Path]:
|
||||
if override:
|
||||
p = Path(override)
|
||||
if p.exists() and (p / ".git").exists():
|
||||
return p
|
||||
for candidate in DEFAULT_CANDIDATE_PATHS.get(repo_name, []):
|
||||
if candidate.exists() and (candidate / ".git").exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def resolve_sessions_dir(override: Optional[str] = None) -> Optional[Path]:
|
||||
if override:
|
||||
p = Path(override)
|
||||
if p.exists():
|
||||
return p
|
||||
for candidate in DEFAULT_SESSIONS_DIRS:
|
||||
if candidate.exists():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def resolve_wiki_dir(override: Optional[str] = None) -> Optional[Path]:
|
||||
return resolve_repo_path("knowledge-wiki", override)
|
||||
|
||||
|
||||
def run_cmd(cmd: List[str], cwd: Optional[Path] = None) -> Tuple[int, str, str]:
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd,
|
||||
cwd=str(cwd) if cwd else None,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
timeout=120,
|
||||
)
|
||||
return proc.returncode, proc.stdout, proc.stderr
|
||||
except Exception as e:
|
||||
return -1, "", str(e)
|
||||
|
||||
|
||||
def get_git_commits(repo_path: Path, since: str) -> List[Dict[str, str]]:
|
||||
if not repo_path or not repo_path.exists():
|
||||
return []
|
||||
# format: hash | iso_date | author | subject
|
||||
cmd = [
|
||||
"git",
|
||||
"-C",
|
||||
str(repo_path),
|
||||
"log",
|
||||
f"--since={since}",
|
||||
'--pretty=format:%h%x09%ad%x09%an%x09%s',
|
||||
"--date=iso-strict",
|
||||
"-n",
|
||||
"20",
|
||||
]
|
||||
rc, stdout, _ = run_cmd(cmd)
|
||||
if rc != 0 or not stdout.strip():
|
||||
return []
|
||||
|
||||
commits = []
|
||||
for line in stdout.strip().splitlines():
|
||||
parts = line.split("\t")
|
||||
if len(parts) >= 4:
|
||||
commits.append({
|
||||
"hash": parts[0],
|
||||
"date": parts[1],
|
||||
"author": parts[2],
|
||||
"subject": parts[3],
|
||||
})
|
||||
return commits
|
||||
|
||||
|
||||
def normalize_error(msg: str) -> str:
|
||||
"""Normalize error strings to group identical issues together."""
|
||||
# Strip ISO timestamps
|
||||
s = re.sub(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?([+-]\d{2}:\d{2}|Z)?", "<TS>", msg)
|
||||
# Strip UUIDs
|
||||
s = re.sub(r"[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "<UUID>", s)
|
||||
# Strip toolCallIds and runIds
|
||||
s = re.sub(r"(call_[0-9a-zA-Z]+|runId=[0-9a-zA-Z-]+)", "<ID>", s)
|
||||
# Strip memory addresses
|
||||
s = re.sub(r"0x[0-9a-fA-F]+", "<HEX>", s)
|
||||
# Strip excessive whitespace
|
||||
s = " ".join(s.split())
|
||||
return s[:250]
|
||||
|
||||
|
||||
def fetch_container_logs(since: str, tail: int = 25000) -> str:
|
||||
"""Fetch container stdout logs using kubectl or read from openclaw log files."""
|
||||
# 1. Try in-cluster kubectl directly
|
||||
kubectl_bin = "/home/node/.openclaw/bin/kubectl" if Path("/home/node/.openclaw/bin/kubectl").exists() else "kubectl"
|
||||
cmd = [
|
||||
kubectl_bin,
|
||||
"logs",
|
||||
f"--since={since}",
|
||||
f"--tail={tail}",
|
||||
"-n",
|
||||
"openclaw",
|
||||
"openclaw-0",
|
||||
"-c",
|
||||
"openclaw",
|
||||
]
|
||||
rc, stdout, _ = run_cmd(cmd)
|
||||
if rc == 0 and stdout.strip():
|
||||
return stdout
|
||||
|
||||
# 2. Fallback: Check local logs directory
|
||||
local_log_dir = Path("/home/node/.openclaw/logs")
|
||||
if local_log_dir.exists():
|
||||
combined = []
|
||||
for log_file in local_log_dir.glob("*.log"):
|
||||
try:
|
||||
combined.append(log_file.read_text(errors="replace"))
|
||||
except Exception:
|
||||
pass
|
||||
if combined:
|
||||
return "\n".join(combined)
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
def parse_container_logs(raw_logs: str) -> List[Dict[str, Any]]:
|
||||
"""Parse and cluster errors found in container stdout logs."""
|
||||
error_clusters: Dict[str, Dict[str, Any]] = defaultdict(lambda: {
|
||||
"signature": "",
|
||||
"category": "GATEWAY",
|
||||
"count": 0,
|
||||
"first_seen": None,
|
||||
"last_seen": None,
|
||||
"sample": "",
|
||||
})
|
||||
|
||||
if not raw_logs:
|
||||
return []
|
||||
|
||||
lines = raw_logs.splitlines()
|
||||
for line in lines:
|
||||
line_lower = line.lower()
|
||||
|
||||
# False positive exclusions
|
||||
if "iserror=false" in line_lower:
|
||||
continue
|
||||
if "[model-fetch] start" in line_lower:
|
||||
continue
|
||||
if "timeoutms=undefined" in line_lower and "status=200" in line_lower:
|
||||
continue
|
||||
if "response provider=" in line_lower and "status=200" in line_lower:
|
||||
continue
|
||||
|
||||
is_error = False
|
||||
category = "GATEWAY"
|
||||
|
||||
# Categorization patterns
|
||||
if "iserror=true" in line_lower or "tool failed" in line_lower or "[embedded run tool end: " in line and "error" in line_lower:
|
||||
is_error = True
|
||||
category = "TOOL"
|
||||
elif "mcp" in line_lower and ("exit" in line_lower or "crashed" in line_lower or "closed with code" in line_lower or "failed to start" in line_lower):
|
||||
is_error = True
|
||||
category = "MCP_SERVER"
|
||||
elif "unhandledrejection" in line_lower or "uncaughtexception" in line_lower or "traceback (most recent call last)" in line_lower or "fatal error" in line_lower:
|
||||
is_error = True
|
||||
category = "RUNTIME_EXCEPTION"
|
||||
elif any(f"status={code}" in line for code in ["500", "502", "503", "504", "429"]) or ("model-fetch" in line_lower and ("failed" in line_lower or "timed out" in line_lower)):
|
||||
is_error = True
|
||||
category = "LLM_PROVIDER"
|
||||
elif " [error] " in line_lower or "level=error" in line_lower or "econnrefused" in line_lower or "etimedout" in line_lower:
|
||||
is_error = True
|
||||
category = "SYSTEM_ERROR"
|
||||
|
||||
if is_error:
|
||||
# Extract timestamp if present at beginning of line
|
||||
ts_match = re.match(r"^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[^\s]*)", line)
|
||||
ts = ts_match.group(1) if ts_match else ""
|
||||
|
||||
sig = normalize_error(line)
|
||||
cluster = error_clusters[sig]
|
||||
cluster["signature"] = sig
|
||||
cluster["category"] = category
|
||||
cluster["count"] += 1
|
||||
cluster["sample"] = line[:400]
|
||||
if ts:
|
||||
if not cluster["first_seen"] or ts < cluster["first_seen"]:
|
||||
cluster["first_seen"] = ts
|
||||
if not cluster["last_seen"] or ts > cluster["last_seen"]:
|
||||
cluster["last_seen"] = ts
|
||||
|
||||
# Convert to sorted list by frequency
|
||||
sorted_clusters = sorted(error_clusters.values(), key=lambda x: x["count"], reverse=True)
|
||||
return sorted_clusters
|
||||
|
||||
|
||||
def parse_session_transcripts(sessions_dir: Path, since_hours: float = 168.0) -> List[Dict[str, Any]]:
|
||||
"""Scan and parse JSONL session transcripts for tool errors."""
|
||||
if not sessions_dir or not sessions_dir.exists():
|
||||
return []
|
||||
|
||||
cutoff = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(hours=since_hours)
|
||||
tool_failures: Dict[str, Dict[str, Any]] = defaultdict(lambda: {
|
||||
"agent": "",
|
||||
"tool": "",
|
||||
"signature": "",
|
||||
"count": 0,
|
||||
"sample": "",
|
||||
"first_seen": None,
|
||||
"last_seen": None,
|
||||
})
|
||||
|
||||
# Sessions are stored at <agent>/sessions/*.jsonl
|
||||
for jsonl_file in sessions_dir.glob("*/sessions/*.jsonl"):
|
||||
try:
|
||||
mtime = datetime.datetime.fromtimestamp(jsonl_file.stat().st_mtime, tz=datetime.timezone.utc)
|
||||
if mtime < cutoff:
|
||||
continue
|
||||
|
||||
agent_name = jsonl_file.parent.parent.name
|
||||
|
||||
with open(jsonl_file, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
# Check for tool errors in various OpenClaw transcript representations
|
||||
role = record.get("role")
|
||||
msg_type = record.get("type")
|
||||
is_err = record.get("isError", False)
|
||||
content = record.get("content", "")
|
||||
|
||||
if isinstance(content, list):
|
||||
content_str = json.dumps(content)
|
||||
else:
|
||||
content_str = str(content)
|
||||
|
||||
# Detect failure markers
|
||||
has_error_keyword = any(k in content_str.lower() for k in [
|
||||
"tool failed",
|
||||
"error:",
|
||||
"traceback",
|
||||
"validationerror",
|
||||
"status code 5",
|
||||
"connection refused",
|
||||
"timeout",
|
||||
])
|
||||
|
||||
if role == "tool" and (is_err or has_error_keyword):
|
||||
tool_name = record.get("name") or record.get("tool") or "unknown_tool"
|
||||
sig = f"[{agent_name}] {tool_name}: {normalize_error(content_str)}"
|
||||
entry = tool_failures[sig]
|
||||
entry["agent"] = agent_name
|
||||
entry["tool"] = tool_name
|
||||
entry["signature"] = sig
|
||||
entry["count"] += 1
|
||||
entry["sample"] = content_str[:300]
|
||||
ts = record.get("createdAt") or record.get("timestamp") or mtime.isoformat()
|
||||
if not entry["first_seen"] or ts < entry["first_seen"]:
|
||||
entry["first_seen"] = ts
|
||||
if not entry["last_seen"] or ts > entry["last_seen"]:
|
||||
entry["last_seen"] = ts
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return sorted(tool_failures.values(), key=lambda x: x["count"], reverse=True)
|
||||
|
||||
|
||||
def build_audit_report(
|
||||
since: str,
|
||||
repos_commits: Dict[str, List[Dict[str, str]]],
|
||||
container_errors: List[Dict[str, Any]],
|
||||
session_errors: List[Dict[str, Any]],
|
||||
) -> Dict[str, Any]:
|
||||
"""Compile structured audit report from all inputs."""
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
year, week, _ = now.isocalendar()
|
||||
|
||||
total_container_errors = sum(c["count"] for c in container_errors)
|
||||
total_session_tool_errors = sum(s["count"] for s in session_errors)
|
||||
|
||||
# Determine severity
|
||||
has_critical = any(c["category"] in ["MCP_SERVER", "RUNTIME_EXCEPTION"] and c["count"] >= 3 for c in container_errors)
|
||||
has_high = total_session_tool_errors > 5 or total_container_errors > 20
|
||||
|
||||
if has_critical:
|
||||
status = "CRITICAL"
|
||||
elif has_high:
|
||||
status = "DEGRADED"
|
||||
elif total_container_errors > 0 or total_session_tool_errors > 0:
|
||||
status = "WARNING"
|
||||
else:
|
||||
status = "HEALTHY"
|
||||
|
||||
return {
|
||||
"year": year,
|
||||
"week": week,
|
||||
"generated_at": now.isoformat(),
|
||||
"since": since,
|
||||
"status": status,
|
||||
"total_container_errors": total_container_errors,
|
||||
"total_session_tool_errors": total_session_tool_errors,
|
||||
"repos_commits": repos_commits,
|
||||
"container_errors": container_errors,
|
||||
"session_errors": session_errors,
|
||||
}
|
||||
|
||||
|
||||
def render_markdown_report(report: Dict[str, Any]) -> str:
|
||||
"""Format audit report into clean Markdown with YAML frontmatter."""
|
||||
year = report["year"]
|
||||
week = report["week"]
|
||||
status = report["status"]
|
||||
status_emoji = {
|
||||
"HEALTHY": "🟢",
|
||||
"WARNING": "🟡",
|
||||
"DEGRADED": "🟠",
|
||||
"CRITICAL": "🔴",
|
||||
}.get(status, "⚪")
|
||||
|
||||
lines = [
|
||||
"---",
|
||||
f'title: "Weekly OpenClaw System Audit {year}-W{week:02d}"',
|
||||
f'date: "{report["generated_at"]}"',
|
||||
f'status: "{status.lower()}"',
|
||||
f'evaluated_window: "{report["since"]}"',
|
||||
"type: audit-report",
|
||||
"---",
|
||||
"",
|
||||
f"# {status_emoji} OpenClaw System Health Audit — {year}-W{week:02d}",
|
||||
"",
|
||||
f"- **Generated At**: `{report['generated_at']}`",
|
||||
f"- **Audit Window**: Last `{report['since']}`",
|
||||
f"- **Overall System State**: **{status}**",
|
||||
f"- **Container Log Errors**: `{report['total_container_errors']}` occurrences",
|
||||
f"- **Session Tool Failures**: `{report['total_session_tool_errors']}` occurrences",
|
||||
"",
|
||||
"---",
|
||||
"",
|
||||
"## 🗂️ Recent Repository Commits Timeline",
|
||||
"",
|
||||
"Cross-repository correlation of recent code and configuration modifications:",
|
||||
"",
|
||||
]
|
||||
|
||||
repos = report.get("repos_commits", {})
|
||||
for repo_name, commits in repos.items():
|
||||
lines.append(f"### 📦 `{repo_name}` ({len(commits)} commits)")
|
||||
if not commits:
|
||||
lines.append("_No commits within this audit window._\n")
|
||||
continue
|
||||
lines.append("| Hash | Date | Author | Message |")
|
||||
lines.append("|---|---|---|---|")
|
||||
for c in commits[:8]:
|
||||
msg = c['subject'].replace("|", "\\|")
|
||||
lines.append(f"| `{c['hash']}` | {c['date'][:10]} | {c['author']} | {msg} |")
|
||||
lines.append("")
|
||||
|
||||
lines.extend([
|
||||
"---",
|
||||
"",
|
||||
"## 🛠️ Session Transcript Tool Failures",
|
||||
"",
|
||||
"Tool invocation errors across subagent conversations (`/home/node/.openclaw/agents/*/sessions/`):",
|
||||
"",
|
||||
])
|
||||
|
||||
session_errs = report.get("session_errors", [])
|
||||
if not session_errs:
|
||||
lines.append("✅ **No tool failures detected in agent session transcripts.**\n")
|
||||
else:
|
||||
lines.append("| Agent | Tool | Count | Sample Error |")
|
||||
lines.append("|---|---|---|---|")
|
||||
for s in session_errs[:15]:
|
||||
sample_clean = s['sample'].replace("|", "\\|").replace("\n", " ")[:120]
|
||||
lines.append(f"| `{s['agent']}` | `{s['tool']}` | **{s['count']}** | `{sample_clean}` |")
|
||||
lines.append("")
|
||||
|
||||
lines.extend([
|
||||
"---",
|
||||
"",
|
||||
"## 🖥️ Container Runtime & Gateway Log Anomalies",
|
||||
"",
|
||||
"Aggregated errors from OpenClaw pod stdout (`kubectl logs -n openclaw openclaw-0 -c openclaw`):",
|
||||
"",
|
||||
])
|
||||
|
||||
container_errs = report.get("container_errors", [])
|
||||
if not container_errs:
|
||||
lines.append("✅ **No runtime exceptions or container errors detected.**\n")
|
||||
else:
|
||||
lines.append("| Category | Count | Normalized Signature | Sample |")
|
||||
lines.append("|---|---|---|---|")
|
||||
for c in container_errs[:15]:
|
||||
sig_clean = c['signature'].replace("|", "\\|").replace("\n", " ")[:100]
|
||||
sample_clean = c['sample'].replace("|", "\\|").replace("\n", " ")[:100]
|
||||
lines.append(f"| `{c['category']}` | **{c['count']}** | `{sig_clean}` | `{sample_clean}` |")
|
||||
lines.append("")
|
||||
|
||||
lines.extend([
|
||||
"---",
|
||||
"",
|
||||
"## 🎯 Actionable Antigravity Recommendations",
|
||||
"",
|
||||
])
|
||||
|
||||
if status == "HEALTHY":
|
||||
lines.append("All services, MCP bridges, and subagent tools operate within healthy parameters. No remediation required.")
|
||||
else:
|
||||
lines.append("The following candidate fixes should be investigated and resolved:")
|
||||
# List top issues
|
||||
for idx, s in enumerate(session_errs[:3], start=1):
|
||||
lines.append(f"{idx}. **Fix Tool Failure in `{s['agent']}` / `{s['tool']}`**: `{s['signature'][:140]}`")
|
||||
for idx, c in enumerate(container_errs[:3], start=len(session_errs[:3]) + 1):
|
||||
lines.append(f"{idx}. **Resolve `{c['category']}` Anomaly**: `{c['signature'][:140]}`")
|
||||
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def update_tasks_antigravity(wiki_dir: Path, report: Dict[str, Any]) -> List[str]:
|
||||
"""Append actionable issues to tasks/antigravity.md under ## 📥 Pending Tasks."""
|
||||
tasks_file = wiki_dir / "tasks" / "antigravity.md"
|
||||
if not tasks_file.exists():
|
||||
return []
|
||||
|
||||
try:
|
||||
content = tasks_file.read_text(encoding="utf-8")
|
||||
except Exception as e:
|
||||
print(f"Error reading {tasks_file}: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
added_tasks = []
|
||||
today = datetime.datetime.now().strftime("%Y-%m-%d")
|
||||
|
||||
# Combine top session and container errors
|
||||
top_candidates = []
|
||||
for s in report.get("session_errors", []):
|
||||
if s["count"] >= 2:
|
||||
top_candidates.append({
|
||||
"title": f"Fix Tool Failure: {s['agent']} / {s['tool']}",
|
||||
"detail": f"Observed {s['count']} times in session transcripts. Sample: {s['sample'][:250]}",
|
||||
"key": f"{s['agent']}:{s['tool']}",
|
||||
})
|
||||
|
||||
for c in report.get("container_errors", []):
|
||||
if c["count"] >= 3 and c["category"] in ["MCP_SERVER", "RUNTIME_EXCEPTION", "TOOL"]:
|
||||
top_candidates.append({
|
||||
"title": f"Resolve {c['category']} Error in OpenClaw Gateway",
|
||||
"detail": f"Observed {c['count']} times in pod logs. Signature: {c['signature'][:200]}",
|
||||
"key": c["signature"][:40],
|
||||
})
|
||||
|
||||
# Only append if not already mentioned in tasks file
|
||||
pending_anchor = "## 📥 Pending Tasks\n"
|
||||
if pending_anchor not in content:
|
||||
return []
|
||||
|
||||
new_task_blocks = []
|
||||
for cand in top_candidates:
|
||||
# Check deduplication
|
||||
if cand["title"] in content or cand["key"] in content:
|
||||
continue
|
||||
|
||||
block = (
|
||||
f"- [ ] **{today}: {cand['title']}**\n"
|
||||
f" - **Details**: Detected by IT Guy weekly self-audit (`audit_openclaw.py`). {cand['detail']}\n"
|
||||
f" - **Target**: OpenClaw Gateway / MCP / Subagent configuration\n"
|
||||
f" - **Requested by**: IT Guy Agent (Autonomous Weekly Audit)\n"
|
||||
)
|
||||
new_task_blocks.append(block)
|
||||
added_tasks.append(cand["title"])
|
||||
|
||||
if new_task_blocks:
|
||||
parts = content.split(pending_anchor, 1)
|
||||
updated_content = parts[0] + pending_anchor + "\n" + "\n".join(new_task_blocks) + "\n" + parts[1]
|
||||
tasks_file.write_text(updated_content, encoding="utf-8")
|
||||
|
||||
return added_tasks
|
||||
|
||||
|
||||
def git_commit_and_push(wiki_dir: Path, commit_msg: str) -> bool:
|
||||
"""Commit and push changes in knowledge-wiki repository."""
|
||||
rc1, _, _ = run_cmd(["git", "-C", str(wiki_dir), "add", "-A"])
|
||||
if rc1 != 0:
|
||||
return False
|
||||
rc2, _, _ = run_cmd(["git", "-C", str(wiki_dir), "commit", "-m", commit_msg])
|
||||
# rc 1 can mean nothing to commit
|
||||
rc3, _, _ = run_cmd(["git", "-C", str(wiki_dir), "push", "origin", "main"])
|
||||
return rc3 == 0
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="OpenClaw System, Log & Repository Self-Audit Engine")
|
||||
parser.add_argument("--since", default="7d", help="Audit time window (e.g. 7d, 24h, 168h)")
|
||||
parser.add_argument("--write-report", action="store_true", help="Save markdown report to operations/audits/ in knowledge-wiki")
|
||||
parser.add_argument("--add-task", action="store_true", help="Append new actionable issues to tasks/antigravity.md")
|
||||
parser.add_argument("--git-push", action="store_true", help="Commit and push changes to knowledge-wiki git remote")
|
||||
parser.add_argument("--json", action="store_true", help="Output machine-readable JSON to stdout")
|
||||
parser.add_argument("--infrapuzzle-path", help="Path to infrapuzzle repo override")
|
||||
parser.add_argument("--mop-skills-path", help="Path to mop-skills repo override")
|
||||
parser.add_argument("--wiki-path", help="Path to knowledge-wiki repo override")
|
||||
parser.add_argument("--sessions-dir", help="Path to OpenClaw sessions directory override")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Convert since string to hours for file timestamps
|
||||
since_hours = 168.0
|
||||
if args.since.endswith("d"):
|
||||
since_hours = float(args.since[:-1]) * 24.0
|
||||
elif args.since.endswith("h"):
|
||||
since_hours = float(args.since[:-1])
|
||||
|
||||
# 1. Resolve Repositories
|
||||
repos = {
|
||||
"infrapuzzle": resolve_repo_path("infrapuzzle", args.infrapuzzle_path),
|
||||
"mop-skills": resolve_repo_path("mop-skills", getattr(args, "mop_skills_path", None)),
|
||||
"knowledge-wiki": resolve_wiki_dir(args.wiki_path),
|
||||
}
|
||||
|
||||
# 2. Correlate Git Commits
|
||||
repos_commits = {}
|
||||
for r_name, r_path in repos.items():
|
||||
if r_path:
|
||||
repos_commits[r_name] = get_git_commits(r_path, args.since)
|
||||
else:
|
||||
repos_commits[r_name] = []
|
||||
|
||||
# 3. Ingest Container Logs
|
||||
raw_logs = fetch_container_logs(args.since)
|
||||
container_errors = parse_container_logs(raw_logs)
|
||||
|
||||
# 4. Ingest Session Transcripts
|
||||
sessions_dir = resolve_sessions_dir(args.sessions_dir)
|
||||
session_errors = parse_session_transcripts(sessions_dir, since_hours) if sessions_dir else []
|
||||
|
||||
# 5. Build Report
|
||||
report = build_audit_report(args.since, repos_commits, container_errors, session_errors)
|
||||
|
||||
# 6. JSON output mode
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2))
|
||||
return
|
||||
|
||||
# 7. Render Markdown
|
||||
md_report = render_markdown_report(report)
|
||||
|
||||
# 8. Write Report if requested
|
||||
wiki_dir = repos.get("knowledge-wiki")
|
||||
if args.write_report and wiki_dir:
|
||||
audits_dir = wiki_dir / "operations" / "audits"
|
||||
audits_dir.mkdir(parents=True, exist_ok=True)
|
||||
report_path = audits_dir / f"{report['year']}-W{report['week']:02d}.md"
|
||||
report_path.write_text(md_report, encoding="utf-8")
|
||||
print(f"✅ Audit report saved: {report_path}")
|
||||
|
||||
# 9. Append to tasks/antigravity.md if requested
|
||||
added_tasks = []
|
||||
if args.add_task and wiki_dir:
|
||||
added_tasks = update_tasks_antigravity(wiki_dir, report)
|
||||
if added_tasks:
|
||||
print(f"📋 Added {len(added_tasks)} new task(s) to tasks/antigravity.md:")
|
||||
for t in added_tasks:
|
||||
print(f" - {t}")
|
||||
else:
|
||||
print("📋 No new tasks required (all issues healthy or already queued).")
|
||||
|
||||
# 10. Commit & Push if requested
|
||||
if args.git_push and wiki_dir and (args.write_report or added_tasks):
|
||||
success = git_commit_and_push(wiki_dir, f"chore(audit): weekly health audit {report['year']}-W{report['week']:02d}")
|
||||
if success:
|
||||
print("🚀 Pushed audit report and task updates to git.moritzgraf.de!")
|
||||
else:
|
||||
print("⚠️ Git push skipped or failed (check credentials/remote).")
|
||||
|
||||
# Print summary to stdout
|
||||
print(md_report)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user