# OpenClaw Agent Guide (Operator-Managed) This document provides a technical reference for AI agents managing the **OpenClaw** deployment in this repository. --- ## 🏗️ Architecture & Configuration Lifecycle The deployment has been migrated to the **OpenClaw Operator** framework. ### 1. Persistent Storage & State Protection * **Storage Claim**: Uses the existing PVC `openclaw-data` (`openebs-hostpath`). * **Zero-Drift Merge**: Configured with `mergeMode: merge`. Any declarative base config applied via K8s is safely deep-merged with runtime configuration mutations (e.g. linked channels, active sessions) written by the agent inside the PVC at `/home/openclaw/.openclaw/openclaw.json`. * **Stateful Memory**: The agent's wisdom (`MEMORY.md`, `SOUL.md`, SQLite databases) resides in the persistent claim and survives pod restarts naturally. ### 2. Sidecar Mapping The multi-container pod structure is managed natively by the operator controller: | Legay Pod Container | Operator Controller Replacement | |---|---| | `chromium-sidecar` | Managed natively under `spec.chromium` with autowired CDP endpoint | | `sidecar-proxy` | Managed natively by operator gateway proxy sidecar | | `git-sync` | Retained under `spec.sidecars` to fetch skills from private git repository | | `skill-stabilizer` | Retained under `spec.sidecars` to copy skills to the flat PVC structure | | `install-uv-python` (init) | Managed natively by the operator under `spec.runtimeDeps.python: true` | ### 3. Multi-Agent Architecture & Agents-as-Code (ADR 0001) * **Conductor**: The main session runs as the lean front-of-house coordinator (`SOUL.md`), handling fast actions (< 2s) directly with MCP tools and delegating complex domain tasks via `sessions_spawn`. * **Specialist Subagents & Telegram Forum Topics**: 10 specialized subagents defined as code in the knowledge wiki and pinned to Telegram forum topics: - 🏋️ `agents/fitness.md` (Topic 20 "Sport" — Garmin Connect 148 tools, Tacx Flux 2, home gym supersets) - 🍳 `agents/koch.md` (Topic 21 "Kochen" — Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization) - 🩺 `agents/hebamme.md` (Topic 22 "Mila" — Baby development & sleep, pediatrician safety boundaries) - ✈️ `agents/travel.md` (Topic 51 "Travel" — Vacation itineraries, packing lists, travel concierge) - 🛒 `agents/shopping.md` (Topic 115 "Shopping" — Supermarket deals, Geizhals best prices, Bring! lists) - 🏰 `agents/regensburg.md` (Topic 116 "Regensburg" — Stadtamhof insider, RVV, playgrounds, local dining) - 🔍 `agents/recherche.md` (Topic 117 "Research" — Deep research, ALLRIS Stadtrat, Friday bridge tracking) - 👨‍⚕️ `agents/doktor.md` (Topic 118 "Medizin" — Evidenzbasierte ärztliche Beratung, Laborwerte, Red-Flag Triage) - 🧑‍💻 `agents/itguy.md` (Topic 119 "IT Guy" — Nerdy CS Hacker, K8s Ops via `kubectl` & RBAC, Git PRs, Prometheus Alert Responder) - 🏡🎬 `agents/grabengasse_media.md` (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify) * **Inter-Agent Peer Escalation (A2A)**: Every subagent topic is configured with peer escalation authority via `sessions_spawn(agentId="", task="...")` to consult sibling agents when cross-domain expertise is needed. * **Modifying & Adding Agents**: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in `knowledge-wiki/agents/.md` and commit to git. No Kubernetes pod restarts required! * **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context. --- ## 🔧 Applying Configuration Changes To perform modifications or rotate api keys: 1. **Secrets**: Edit string fields in [openclaw.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw.secret.yaml). 2. **Declarative Base Config**: Edit fields under `spec.config.raw` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml). 3. **Apply & Rollout**: Apply the files. The operator tracks config hashes and performs an automated rolling update of the StatefulSet automatically (no manual restart needed!): ```bash kubectl apply -f k8s/openclaw/openclaw.secret.yaml kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml ``` 4. **Register / Sync Automations**: Run the automation setup scripts to register OpenClaw Cron tasks: ```bash ./k8s/openclaw/setup-itguy-cron.sh # Weekly health audit (Sun 03:00) ./k8s/openclaw/setup-compaction-cron.sh # Nightly multi-agent session compaction (Daily 03:30) ``` --- ### 4. 🗂️ The Three Core Repositories Architecture & Weekly Self-Audit OpenClaw connects to three foundational Git repositories: 1. **`infrapuzzle`** (`/home/node/.openclaw/repos/infrapuzzle` on PVC): - **Scope**: Source of truth for cluster infrastructure (`haumdaucher`), Kubespray bootstrap, K8s manifests, Terraform DNS, ESPHome IoT, and Architecture Decision Records (`docs/adr/`). 2. **`mop-skills`** (`/home/node/mop-skills/mop-skills` via `git-sync` sidecar): - **Scope**: Custom Python skills, CLI utilities, and MCP bridges (`openclaw-audit`, `garmin-auth`, `brother-printer`, `google-calendar`, etc.). 3. **`knowledge-wiki`** (`/home/node/.openclaw/wiki/main`): - **Scope**: Agents-as-code (`agents/*.md`), active task dispatch queue (`tasks/antigravity.md`), family knowledge vault, and audit archives (`operations/audits/`). #### 🩺 Autonomous Weekly Health Audit (`itguy-weekly-health-audit`) - Scheduled via OpenClaw Cron on **Sundays at 03:00 Europe/Berlin** (`0 3 * * 0`). - Executes `uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d --write-report --add-task --git-push`. - Correlates errors (pod stdout & session transcripts) with recent git commits across the 3 repositories. - Archives diagnostic reports into `operations/audits/YYYY-Wxx.md`. - Appends newly detected, actionable tasks into `tasks/antigravity.md` under `## 📥 Pending Tasks`. - **Telegram Notification Policy**: Alerts Topic `#IT Guy` (thread `119`) if new issues are filed; returns `NO_REPLY` for intentional silence when clean. --- ## 🚨 Troubleshooting & Incident Playbook > [!IMPORTANT] > **Mandatory Bug & Incident Documentation**: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, **always document the incident in [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)** with symptom, root cause, resolution, and how to avoid recurrence. ### 🛡️ Operational & Diagnostic Invariants (Learned Guidelines) Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules: 0. **Rapid Incident Triaging & Information Gathering (< 2s)**: - When any issue, silence, or strange bot behavior is reported, **run the one-shot triage tool first**: ```bash kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic ``` - Do NOT run piecemeal SQLite queries, guess table/column names, or scan the container filesystem. - **Path Anchors (Never run `find /` or broad `grep -r` on rotational disk):** - Knowledge Wiki: `/home/node/.openclaw/wiki/main` - Agent Databases: `/home/node/.openclaw/agents//agent/openclaw-agent.sqlite` - Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite` - Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`) 1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**: - When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**. - Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first. - Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (`compact:start`) can introduce multi-minute turn latency without any pod restart. 2. **Datastore Ground-Truth for Maintenance**: - High-level CLI summaries (`openclaw sessions --all-agents --json`) report `totalTokens: null` (`totalTokensFresh: false`) for stale sessions or after restarts. - Automated health checks and compaction scripts MUST query the SQLite database directly (`transcript_events` event count and byte size), triggering compaction when `bytes >= 300KB` or `events >= 40`, regardless of cached token values. 3. **Compaction User Visibility**: - Always ensure `agents.defaults.compaction.notifyUser: true` remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (`🧹 Compacting context...`) to avoid perceived bot stalls. 4. **In-Pod Container Restart Self-Healing**: - Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (`NODE_OPTIONS="--import=..."`) or application startup hooks. - Never place critical self-healing logic exclusively in `initContainers`, as Kubelet container restarts inside an existing Pod completely bypass `initContainers`. 5. **Rotational Storage (HDD) Page-Cache Pre-Warming**: - Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`). - Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks. 6. **Zero-Downtime Wiki Prompt Updates vs. Heavy Pod Rollouts**: - Subagent prompt instructions, tone, and tool guidelines live in `knowledge-wiki/agents/.md`. Edits made there are read live by OpenClaw on git sync without any pod restart. - Avoid modifying `openclaw-instance.secret.yaml` unless modifying container limits, environment variables, or global operator configuration. A CR edit triggers a 7-init-container rolling update (~3 minutes). 7. **Continuous Troubleshooting Evolution & The Simplicity Invariant (The 98% Rule)**: - **Post-Fix Dual Reflection**: At the end of every troubleshooting cycle and fix, the AI MUST proactively evaluate: 1. *Troubleshooting Optimization*: How can the diagnostic process be made faster or codified into `triage.py` / `openclaw-ops`? 2. *System Maturity*: How can the root cause be hardened permanently to prevent recurrence? - **Simplicity First**: Simplicity and straightforward architecture are the central design principles. A simple, pragmatic solution that cleanly reaches **98% of the goal** is strictly preferred over an over-engineered abstraction attempting 100%. Never introduce background workers, multi-stage state machines, or silent fallbacks when a transparent single-step tool or direct alert suffices. For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md). ### 1. Verify Deployment Health Ensure that the `OpenClawInstance` is successfully reconciled: ```bash kubectl get openclawinstance openclaw -n openclaw # Expected: PHASE=Running, READY=True ``` ### 2. Verify Pod & Container States Check that all 6 containers inside the StatefulSet are fully functional: ```bash kubectl get pods -n openclaw -l app.kubernetes.io/instance=openclaw ``` ### 3. Check Live Logs Inspect the core gateway engine: ```bash kubectl logs -n openclaw openclaw-0 -c openclaw --tail=100 ``` Inspect the Chromium browser engine: ```bash kubectl logs -n openclaw openclaw-0 -c chromium --tail=100 ```