moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
0cbd39d0d8ef69a18d5098641dd50cfc5ace86a6
infrapuzzle/k8s/openclaw/AGENTS.md
T

9.6 KiB
Raw Blame History

OpenClaw Agent Guide (Operator-Managed)

This document provides a technical reference for AI agents managing the OpenClaw deployment in this repository.

🏗️ Architecture & Configuration Lifecycle

The deployment has been migrated to the OpenClaw Operator framework.

1. Persistent Storage & State Protection

  • Storage Claim: Uses the existing PVC openclaw-data (openebs-hostpath).
  • Zero-Drift Merge: Configured with mergeMode: merge. Any declarative base config applied via K8s is safely deep-merged with runtime configuration mutations (e.g. linked channels, active sessions) written by the agent inside the PVC at /home/openclaw/.openclaw/openclaw.json.
  • Stateful Memory: The agent's wisdom (MEMORY.md, SOUL.md, SQLite databases) resides in the persistent claim and survives pod restarts naturally.

2. Sidecar Mapping

The multi-container pod structure is managed natively by the operator controller:

Legay Pod Container Operator Controller Replacement
chromium-sidecar Managed natively under spec.chromium with autowired CDP endpoint
sidecar-proxy Managed natively by operator gateway proxy sidecar
git-sync Retained under spec.sidecars to fetch skills from private git repository
skill-stabilizer Retained under spec.sidecars to copy skills to the flat PVC structure
install-uv-python (init) Managed natively by the operator under spec.runtimeDeps.python: true

3. Multi-Agent Architecture & Agents-as-Code (ADR 0001)

  • Conductor: The main session runs as the lean front-of-house coordinator (SOUL.md), handling fast actions (< 2s) directly with MCP tools and delegating complex domain tasks via sessions_spawn.
  • Specialist Subagents & Telegram Forum Topics: 10 specialized subagents defined as code in the knowledge wiki and pinned to Telegram forum topics:
    • 🏋️ agents/fitness.md (Topic 20 "Sport" — Garmin Connect 148 tools, Tacx Flux 2, home gym supersets)
    • 🍳 agents/koch.md (Topic 21 "Kochen" — Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization)
    • 🩺 agents/hebamme.md (Topic 22 "Mila" — Baby development & sleep, pediatrician safety boundaries)
    • ✈️ agents/travel.md (Topic 51 "Travel" — Vacation itineraries, packing lists, travel concierge)
    • 🛒 agents/shopping.md (Topic 115 "Shopping" — Supermarket deals, Geizhals best prices, Bring! lists)
    • 🏰 agents/regensburg.md (Topic 116 "Regensburg" — Stadtamhof insider, RVV, playgrounds, local dining)
    • 🔍 agents/recherche.md (Topic 117 "Research" — Deep research, ALLRIS Stadtrat, Friday bridge tracking)
    • 👨‍⚕️ agents/doktor.md (Topic 118 "Medizin" — Evidenzbasierte ärztliche Beratung, Laborwerte, Red-Flag Triage)
    • 🧑‍💻 agents/itguy.md (Topic 119 "IT Guy" — Nerdy CS Hacker, K8s Ops via kubectl & RBAC, Git PRs, Prometheus Alert Responder)
    • 🏡🎬 agents/grabengasse_media.md (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify)
  • Inter-Agent Peer Escalation (A2A): Every subagent topic is configured with peer escalation authority via sessions_spawn(agentId="<target>", task="...") to consult sibling agents when cross-domain expertise is needed.
  • Modifying & Adding Agents: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in knowledge-wiki/agents/<name>.md and commit to git. No Kubernetes pod restarts required!
  • Multi-Turn Domain Stickiness: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context.

🔧 Applying Configuration Changes

To perform modifications or rotate api keys:

  1. Secrets: Edit string fields in openclaw.secret.yaml.
  2. Declarative Base Config: Edit fields under spec.config.raw in openclaw-instance.secret.yaml.
  3. Apply & Rollout: Apply the files. The operator tracks config hashes and performs an automated rolling update of the StatefulSet automatically (no manual restart needed!): 
    kubectl apply -f k8s/openclaw/openclaw.secret.yaml
kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml
kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml
  4. Register / Sync Automations: Run the automation setup scripts to register OpenClaw Cron tasks: 
    ./k8s/openclaw/setup-itguy-cron.sh        # Weekly health audit (Sun 03:00)
./k8s/openclaw/setup-compaction-cron.sh   # Nightly multi-agent session compaction (Daily 03:30)

4. 🗂️ The Three Core Repositories Architecture & Weekly Self-Audit

OpenClaw connects to three foundational Git repositories:

  1. infrapuzzle (/home/node/.openclaw/repos/infrapuzzle on PVC):
    • Scope: Source of truth for cluster infrastructure (haumdaucher), Kubespray bootstrap, K8s manifests, Terraform DNS, ESPHome IoT, and Architecture Decision Records (docs/adr/).
  2. mop-skills (/home/node/mop-skills/mop-skills via git-sync sidecar):
    • Scope: Custom Python skills, CLI utilities, and MCP bridges (openclaw-audit, garmin-auth, brother-printer, google-calendar, etc.).
  3. knowledge-wiki (/home/node/.openclaw/wiki/main):
    • Scope: Agents-as-code (agents/*.md), active task dispatch queue (tasks/antigravity.md), family knowledge vault, and audit archives (operations/audits/).

🩺 Autonomous Weekly Health Audit (itguy-weekly-health-audit)

  • Scheduled via OpenClaw Cron on Sundays at 03:00 Europe/Berlin (0 3 * * 0).
  • Executes uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d --write-report --add-task --git-push.
  • Correlates errors (pod stdout & session transcripts) with recent git commits across the 3 repositories.
  • Archives diagnostic reports into operations/audits/YYYY-Wxx.md.
  • Appends newly detected, actionable tasks into tasks/antigravity.md under ## 📥 Pending Tasks.
  • Telegram Notification Policy: Alerts Topic #IT Guy (thread 119) if new issues are filed; returns NO_REPLY for intentional silence when clean.

🚨 Troubleshooting & Incident Playbook

Important

Mandatory Bug & Incident Documentation: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, always document the incident in TROUBLESHOOTING.md with symptom, root cause, resolution, and how to avoid recurrence.

🛡️ Operational & Diagnostic Invariants (Learned Guidelines)

Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules:

  1. Verify Before Declaring Dead (Turn In-Flight Diagnostics):

    • When a user reports that a Telegram topic is "not answering" or "silent", NEVER assume the container or gateway has crashed.
    • Always inspect live pod logs (kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100) and the agent's SQLite database (session_nodes and transcript_events) first.
    • Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (compact:start) can introduce multi-minute turn latency without any pod restart.

  2. Datastore Ground-Truth for Maintenance:

    • High-level CLI summaries (openclaw sessions --all-agents --json) report totalTokens: null (totalTokensFresh: false) for stale sessions or after restarts.
    • Automated health checks and compaction scripts MUST query the SQLite database directly (transcript_events event count and byte size), triggering compaction when bytes >= 300KB or events >= 40, regardless of cached token values.

  3. Compaction User Visibility:

    • Always ensure agents.defaults.compaction.notifyUser: true remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (🧹 Compacting context...) to avoid perceived bot stalls.

  4. In-Pod Container Restart Self-Healing:

    • Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (NODE_OPTIONS="--import=...") or application startup hooks.
    • Never place critical self-healing logic exclusively in initContainers, as Kubelet container restarts inside an existing Pod completely bypass initContainers.

  5. Rotational Storage (HDD) Page-Cache Pre-Warming:

    • Host haumdaucher uses rotational mechanical disks (/dev/sda2, rotational: 1).
    • Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to /dev/null (cat ... > /dev/null) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks.

For known issues, resolved bugs, and specific failure playbooks, consult TROUBLESHOOTING.md.

1. Verify Deployment Health

Ensure that the OpenClawInstance is successfully reconciled:

kubectl get openclawinstance openclaw -n openclaw
# Expected: PHASE=Running, READY=True

2. Verify Pod & Container States

Check that all 6 containers inside the StatefulSet are fully functional:

kubectl get pods -n openclaw -l app.kubernetes.io/instance=openclaw

3. Check Live Logs

Inspect the core gateway engine:

kubectl logs -n openclaw openclaw-0 -c openclaw --tail=100

Inspect the Chromium browser engine:

kubectl logs -n openclaw openclaw-0 -c chromium --tail=100
Reference in New Issue View Git Blame Copy Permalink