163 lines
12 KiB
Markdown
163 lines
12 KiB
Markdown
# OpenClaw Agent Guide (Operator-Managed)
|
|
|
|
This document provides a technical reference for AI agents managing the **OpenClaw** deployment in this repository.
|
|
|
|
---
|
|
|
|
## 🏗️ Architecture & Configuration Lifecycle
|
|
|
|
The deployment has been migrated to the **OpenClaw Operator** framework.
|
|
|
|
### 1. Persistent Storage & State Protection
|
|
* **Storage Claim**: Uses the existing PVC `openclaw-data` (`openebs-hostpath`).
|
|
* **Zero-Drift Merge**: Configured with `mergeMode: merge`. Any declarative base config applied via K8s is safely deep-merged with runtime configuration mutations (e.g. linked channels, active sessions) written by the agent inside the PVC at `/home/openclaw/.openclaw/openclaw.json`.
|
|
* **Stateful Memory**: The agent's wisdom (`MEMORY.md`, `SOUL.md`, SQLite databases) resides in the persistent claim and survives pod restarts naturally.
|
|
|
|
### 2. Sidecar Mapping
|
|
|
|
The multi-container pod structure is managed natively by the operator controller:
|
|
|
|
| Legay Pod Container | Operator Controller Replacement |
|
|
|---|---|
|
|
| `chromium-sidecar` | Managed natively under `spec.chromium` with autowired CDP endpoint |
|
|
| `sidecar-proxy` | Managed natively by operator gateway proxy sidecar |
|
|
| `git-sync` | Retained under `spec.sidecars` to fetch skills from private git repository |
|
|
| `skill-stabilizer` | Retained under `spec.sidecars` to copy skills to the flat PVC structure |
|
|
| `install-uv-python` (init) | Managed natively by the operator under `spec.runtimeDeps.python: true` |
|
|
|
|
### 3. Multi-Agent Architecture & Agents-as-Code (ADR 0001)
|
|
* **Conductor**: The main session runs as the lean front-of-house coordinator (`SOUL.md`), handling fast actions (< 2s) directly with MCP tools and delegating complex domain tasks via `sessions_spawn`.
|
|
* **Specialist Subagents & Telegram Forum Topics**: 10 specialized subagents defined as code in the knowledge wiki and pinned to Telegram forum topics:
|
|
- 🏋️ `agents/fitness.md` (Topic 20 "Sport" — Garmin Connect 148 tools, Tacx Flux 2, home gym supersets)
|
|
- 🍳 `agents/koch.md` (Topic 21 "Kochen" — Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization)
|
|
- 🩺 `agents/hebamme.md` (Topic 22 "Mila" — Baby development & sleep, pediatrician safety boundaries)
|
|
- ✈️ `agents/travel.md` (Topic 51 "Travel" — Vacation itineraries, packing lists, travel concierge)
|
|
- 🛒 `agents/shopping.md` (Topic 115 "Shopping" — Supermarket deals, Geizhals best prices, Bring! lists)
|
|
- 🏰 `agents/regensburg.md` (Topic 116 "Regensburg" — Stadtamhof insider, RVV, playgrounds, local dining)
|
|
- 🔍 `agents/recherche.md` (Topic 117 "Research" — Deep research, ALLRIS Stadtrat, Friday bridge tracking)
|
|
- 👨⚕️ `agents/doktor.md` (Topic 118 "Medizin" — Evidenzbasierte ärztliche Beratung, Laborwerte, Red-Flag Triage)
|
|
- 🧑💻 `agents/itguy.md` (Topic 119 "IT Guy" — Nerdy CS Hacker, K8s Ops via `kubectl` & RBAC, Git PRs, Prometheus Alert Responder)
|
|
- 🏡🎬 `agents/grabengasse_media.md` (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify)
|
|
* **Inter-Agent Peer Escalation (A2A)**: Every subagent topic is configured with peer escalation authority via `sessions_spawn(agentId="<target>", task="...")` to consult sibling agents when cross-domain expertise is needed.
|
|
* **Modifying & Adding Agents**: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in `knowledge-wiki/agents/<name>.md` and commit to git. No Kubernetes pod restarts required!
|
|
* **Multi-Turn Domain Stickiness**: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context.
|
|
|
|
---
|
|
|
|
## 🔧 Applying Configuration Changes
|
|
|
|
To perform modifications or rotate api keys:
|
|
1. **Secrets**: Edit string fields in [openclaw.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw.secret.yaml).
|
|
2. **Declarative Base Config**: Edit fields under `spec.config.raw` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml).
|
|
3. **Apply & Rollout**: Apply the files. The operator tracks config hashes and performs an automated rolling update of the StatefulSet automatically (no manual restart needed!):
|
|
```bash
|
|
kubectl apply -f k8s/openclaw/openclaw.secret.yaml
|
|
kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml
|
|
kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml
|
|
```
|
|
4. **Register / Sync Automations**: Run the automation setup scripts to register OpenClaw Cron tasks:
|
|
```bash
|
|
./k8s/openclaw/setup-garmin-cron.sh # Nightly Garmin Connect sync (Daily 03:30)
|
|
./k8s/openclaw/setup-compaction-cron.sh # Nightly multi-agent session compaction (Daily 04:45)
|
|
./k8s/openclaw/setup-itguy-cron.sh # Weekly health audit (Sun 03:00)
|
|
```
|
|
|
|
---
|
|
|
|
### 4. 🗂️ The Three Core Repositories Architecture & Weekly Self-Audit
|
|
|
|
OpenClaw connects to three foundational Git repositories:
|
|
1. **`infrapuzzle`** (`/home/node/.openclaw/repos/infrapuzzle` on PVC):
|
|
- **Scope**: Source of truth for cluster infrastructure (`haumdaucher`), Kubespray bootstrap, K8s manifests, Terraform DNS, ESPHome IoT, and Architecture Decision Records (`docs/adr/`).
|
|
2. **`mop-skills`** (`/home/node/mop-skills/mop-skills` via `git-sync` sidecar):
|
|
- **Scope**: Custom Python skills, CLI utilities, and MCP bridges (`openclaw-audit`, `garmin-auth`, `brother-printer`, `google-calendar`, etc.).
|
|
3. **`knowledge-wiki`** (`/home/node/.openclaw/wiki/main`):
|
|
- **Scope**: Agents-as-code (`agents/*.md`), active task dispatch queue (`tasks/antigravity.md`), family knowledge vault, and audit archives (`operations/audits/`).
|
|
|
|
#### 🩺 Autonomous Weekly Health Audit (`itguy-weekly-health-audit`)
|
|
- Scheduled via OpenClaw Cron on **Sundays at 03:00 Europe/Berlin** (`0 3 * * 0`).
|
|
- Executes `uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d --write-report --add-task --git-push`.
|
|
- Correlates errors (pod stdout & session transcripts) with recent git commits across the 3 repositories.
|
|
- Archives diagnostic reports into `operations/audits/YYYY-Wxx.md`.
|
|
- Appends newly detected, actionable tasks into `tasks/antigravity.md` under `## 📥 Pending Tasks`.
|
|
- **Telegram Notification Policy**: Alerts Topic `#IT Guy` (thread `119`) if new issues are filed; returns `NO_REPLY` for intentional silence when clean.
|
|
|
|
---
|
|
|
|
## 🚨 Troubleshooting & Incident Playbook
|
|
|
|
> [!IMPORTANT]
|
|
> **Mandatory Bug & Incident Documentation**: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, **always document the incident in [TROUBLESHOOTING.md](./TROUBLESHOOTING.md)** with symptom, root cause, resolution, and how to avoid recurrence.
|
|
|
|
### 🛡️ Operational & Diagnostic Invariants (Learned Guidelines)
|
|
|
|
Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules:
|
|
|
|
0. **Rapid Incident Triaging & Information Gathering (< 2s)**:
|
|
- When any issue, silence, or strange bot behavior is reported, **run the one-shot triage tool first**:
|
|
```bash
|
|
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicId>
|
|
```
|
|
- Do NOT run piecemeal SQLite queries, guess table/column names, or scan the container filesystem.
|
|
- **Path Anchors (Never run `find /` or broad `grep -r` on rotational disk):**
|
|
- Knowledge Wiki: `/home/node/.openclaw/wiki/main`
|
|
- Agent Databases: `/home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite`
|
|
- Gateway State DB: `/home/node/.openclaw/state/openclaw.sqlite`
|
|
- Standalone Triage Script: `/home/node/.openclaw/scripts/triage.py` (Repo: `k8s/openclaw/scripts/openclaw-triage.py`)
|
|
|
|
1. **Verify Before Declaring Dead (Turn In-Flight Diagnostics)**:
|
|
- When a user reports that a Telegram topic is "not answering" or "silent", **NEVER assume the container or gateway has crashed**.
|
|
- Always inspect live pod logs (`kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100`) and the agent's SQLite database (`session_nodes` and `transcript_events`) first.
|
|
- Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (`compact:start`) can introduce multi-minute turn latency without any pod restart.
|
|
|
|
2. **Datastore Ground-Truth for Maintenance**:
|
|
- High-level CLI summaries (`openclaw sessions --all-agents --json`) report `totalTokens: null` (`totalTokensFresh: false`) for stale sessions or after restarts.
|
|
- Automated health checks and compaction scripts MUST query the SQLite database directly (`transcript_events` event count and byte size), triggering compaction when `bytes >= 300KB` or `events >= 40`, regardless of cached token values.
|
|
|
|
3. **Compaction User Visibility**:
|
|
- Always ensure `agents.defaults.compaction.notifyUser: true` remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (`🧹 Compacting context...`) to avoid perceived bot stalls.
|
|
|
|
4. **In-Pod Container Restart Self-Healing**:
|
|
- Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (`NODE_OPTIONS="--import=..."`) or application startup hooks.
|
|
- Never place critical self-healing logic exclusively in `initContainers`, as Kubelet container restarts inside an existing Pod completely bypass `initContainers`.
|
|
|
|
5. **Rotational Storage (HDD) Page-Cache Pre-Warming**:
|
|
- Host `haumdaucher` uses rotational mechanical disks (`/dev/sda2`, `rotational: 1`).
|
|
- Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to `/dev/null` (`cat ... > /dev/null`) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks.
|
|
|
|
6. **Zero-Downtime Wiki Prompt Updates vs. Heavy Pod Rollouts**:
|
|
- Subagent prompt instructions, tone, and tool guidelines live in `knowledge-wiki/agents/<name>.md`. Edits made there are read live by OpenClaw on git sync without any pod restart.
|
|
- Avoid modifying `openclaw-instance.secret.yaml` unless modifying container limits, environment variables, or global operator configuration. A CR edit triggers a 7-init-container rolling update (~3 minutes).
|
|
|
|
7. **Continuous Troubleshooting Evolution & The Simplicity Invariant (The 98% Rule)**:
|
|
- **Post-Fix Dual Reflection**: At the end of every troubleshooting cycle and fix, the AI MUST proactively evaluate:
|
|
1. *Troubleshooting Optimization*: How can the diagnostic process be made faster or codified into `triage.py` / `openclaw-ops`?
|
|
2. *System Maturity*: How can the root cause be hardened permanently to prevent recurrence?
|
|
- **Simplicity First**: Simplicity and straightforward architecture are the central design principles. A simple, pragmatic solution that cleanly reaches **98% of the goal** is strictly preferred over an over-engineered abstraction attempting 100%. Never introduce background workers, multi-stage state machines, or silent fallbacks when a transparent single-step tool or direct alert suffices.
|
|
|
|
For known issues, resolved bugs, and specific failure playbooks, consult [TROUBLESHOOTING.md](./TROUBLESHOOTING.md).
|
|
|
|
### 1. Verify Deployment Health
|
|
Ensure that the `OpenClawInstance` is successfully reconciled:
|
|
```bash
|
|
kubectl get openclawinstance openclaw -n openclaw
|
|
# Expected: PHASE=Running, READY=True
|
|
```
|
|
|
|
### 2. Verify Pod & Container States
|
|
Check that all 6 containers inside the StatefulSet are fully functional:
|
|
```bash
|
|
kubectl get pods -n openclaw -l app.kubernetes.io/instance=openclaw
|
|
```
|
|
|
|
### 3. Check Live Logs
|
|
Inspect the core gateway engine:
|
|
```bash
|
|
kubectl logs -n openclaw openclaw-0 -c openclaw --tail=100
|
|
```
|
|
|
|
Inspect the Chromium browser engine:
|
|
```bash
|
|
kubectl logs -n openclaw openclaw-0 -c chromium --tail=100
|
|
```
|