moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
ed3529a9d7768740efba086c5d4e682ce2fc9e11
infrapuzzle/k8s/openclaw/AGENTS.md
T

14 KiB
Raw Blame History

OpenClaw Agent Guide (Operator-Managed)

This document provides a technical reference for AI agents managing the OpenClaw deployment in this repository.

🏗️ Architecture & Configuration Lifecycle

The deployment has been migrated to the OpenClaw Operator framework.

1. Persistent Storage & State Protection

  • Storage Claim: Uses the existing PVC openclaw-data (openebs-hostpath).
  • Zero-Drift Merge: Configured with mergeMode: merge. Any declarative base config applied via K8s is safely deep-merged with runtime configuration mutations (e.g. linked channels, active sessions) written by the agent inside the PVC at /home/openclaw/.openclaw/openclaw.json.
  • Stateful Memory: The agent's wisdom (MEMORY.md, SOUL.md, SQLite databases) resides in the persistent claim and survives pod restarts naturally.

2. Sidecar Mapping

The multi-container pod structure is managed natively by the operator controller:

Legay Pod Container Operator Controller Replacement
chromium-sidecar Managed natively under spec.chromium with autowired CDP endpoint
sidecar-proxy Managed natively by operator gateway proxy sidecar
git-sync Retained under spec.sidecars to fetch skills from private git repository
skill-stabilizer Retained under spec.sidecars to copy skills to the flat PVC structure
install-uv-python (init) Managed natively by the operator under spec.runtimeDeps.python: true

3. Multi-Agent Architecture & Agents-as-Code (ADR 0001)

  • Conductor: The main session runs as the lean front-of-house coordinator (SOUL.md), handling fast actions (< 2s) directly with MCP tools and delegating complex domain tasks via sessions_spawn.
  • Specialist Subagents & Telegram Forum Topics: 10 specialized subagents defined as code in the knowledge wiki and pinned to Telegram forum topics:
    • 🏋️ agents/fitness.md (Topic 20 "Sport" — Garmin Connect 148 tools, Tacx Flux 2, home gym supersets)
    • 🍳 agents/koch.md (Topic 21 "Kochen" — Ninja CREAMi Deluxe, Airfryer, KitchenAid, leftover utilization)
    • 🩺 agents/hebamme.md (Topic 22 "Mila" — Baby development & sleep, pediatrician safety boundaries)
    • ✈️ agents/travel.md (Topic 51 "Travel" — Vacation itineraries, packing lists, travel concierge)
    • 🛒 agents/shopping.md (Topic 115 "Shopping" — Supermarket deals, Geizhals best prices, Bring! lists)
    • 🏰 agents/regensburg.md (Topic 116 "Regensburg" — Stadtamhof insider, RVV, playgrounds, local dining)
    • 🔍 agents/recherche.md (Topic 117 "Research" — Deep research, ALLRIS Stadtrat, Friday bridge tracking)
    • 👨‍⚕️ agents/doktor.md (Topic 118 "Medizin" — Evidenzbasierte ärztliche Beratung, Laborwerte, Red-Flag Triage)
    • 🧑‍💻 agents/itguy.md (Topic 119 "IT Guy" — Nerdy CS Hacker, K8s Ops via kubectl & RBAC, Git PRs, Prometheus Alert Responder)
    • 🏡🎬 agents/grabengasse_media.md (Topic 164 "Grabengasse & Media" — Home Assistant, Hue, Sonos, Brother Drucker, Jellyfin & Spotify)
  • Inter-Agent Peer Escalation (A2A): Every subagent topic is configured with peer escalation authority via sessions_spawn(agentId="<target>", task="...") to consult sibling agents when cross-domain expertise is needed.
  • Modifying & Adding Agents: To update prompt logic, tone, or tool instructions for any agent, edit its markdown file directly in knowledge-wiki/agents/<name>.md and commit to git. No Kubernetes pod restarts required!
  • Multi-Turn Domain Stickiness: When a conversation spans multiple turns in a subagent domain (e.g. refining a recipe or planning a trip), the Conductor detects domain continuity and routes to the same agent with clean conversational context.

🔧 Applying Configuration Changes

To perform modifications or rotate api keys:

  1. Secrets: Edit string fields in openclaw.secret.yaml.
  2. Declarative Base Config: Edit fields under spec.config.raw in openclaw-instance.secret.yaml.
  3. Apply & Rollout: Apply the files. The operator tracks config hashes and performs an automated rolling update of the StatefulSet automatically (no manual restart needed!): 
    kubectl apply -f k8s/openclaw/openclaw.secret.yaml
kubectl apply -f k8s/openclaw/openclaw-instance.secret.yaml
kubectl apply -f k8s/openclaw/cronjob-wiki-backup.secret.yaml
  4. Register / Sync Automations: Run the automation setup scripts to register OpenClaw Cron tasks: 
    ./k8s/openclaw/setup-garmin-cron.sh       # Nightly Garmin Connect sync (Daily 03:30)
./k8s/openclaw/setup-compaction-cron.sh   # Nightly multi-agent session compaction (Daily 04:45)
./k8s/openclaw/setup-itguy-cron.sh        # Weekly health audit (Sun 03:00)
# Or run declarative reconciler:
python3 k8s/openclaw/scripts/sync_crons.py

🛡️ Everything-as-Code (EaC) Governance for New Crons, Skills & Binaries (ADR 0003)

To prevent PVC snowflake drift, every new automation, skill, CLI binary, or Python dependency added to OpenClaw MUST be persisted as code:

  1. New Cron Jobs (Crons-as-Code):
    • System / Recurring Automations: Never leave a recurring cron job exclusively inside the pod's SQLite database (openclaw cron add). Every permanent automation MUST be codified in k8s/openclaw/crons.yaml (or the declarative cron sync scripts) and committed to infrapuzzle.
    • Dynamic / User-Created Reminders: Conversational or temporary crons created by agents at runtime must be exported to knowledge-wiki/operations/crons-backup.json during nightly backup so they survive PVC loss.
  2. New Skills (Skills-as-Code):
    • Never leave newly created or agent-authored skills only in /home/node/.openclaw/skills/, workspace*/skills/, or skill-workshop/ on the PVC.
    • All custom skills MUST be committed and pushed to the mop-skills repository (git.moritzgraf.de/moritz/mop-skills) so they are version-controlled and synced cleanly via the git-sync sidecar.
  3. New CLI Binaries, Packages & Patches (Dependencies-as-Code):
    • Never manually drop binaries into /home/node/.openclaw/bin/ or run ad-hoc uv pip install / npm install -g on the PVC without codifying them.
    • Every new binary (e.g. ffmpeg, kubectl, jq, openhue, sonos), Python package in mcp-venv, or runtime patch MUST be added to the declarative bootstrap script (k8s/openclaw/scripts/bootstrap-deps.sh / configmap-scripts.yaml) so a fresh PVC rebuild has 100% tool parity.

4. 🗂️ The Three Core Repositories Architecture & Weekly Self-Audit

OpenClaw connects to three foundational Git repositories:

  1. infrapuzzle (/home/node/.openclaw/repos/infrapuzzle on PVC):
    • Scope: Source of truth for cluster infrastructure (haumdaucher), Kubespray bootstrap, K8s manifests, Terraform DNS, ESPHome IoT, and Architecture Decision Records (docs/adr/).
  2. mop-skills (/home/node/mop-skills/mop-skills via git-sync sidecar):
    • Scope: Custom Python skills, CLI utilities, and MCP bridges (openclaw-audit, garmin-auth, brother-printer, google-calendar, etc.).
  3. knowledge-wiki (/home/node/.openclaw/wiki/main):
    • Scope: Agents-as-code (agents/*.md), active task dispatch queue (tasks/antigravity.md), family knowledge vault, and audit archives (operations/audits/).

🩺 Autonomous Weekly Health Audit (itguy-weekly-health-audit)

  • Scheduled via OpenClaw Cron on Sundays at 03:00 Europe/Berlin (0 3 * * 0).
  • Executes uv run /home/node/mop-skills/mop-skills/skills/openclaw-audit/scripts/audit_openclaw.py --since 7d --write-report --add-task --git-push.
  • Correlates errors (pod stdout & session transcripts) with recent git commits across the 3 repositories.
  • Archives diagnostic reports into operations/audits/YYYY-Wxx.md.
  • Appends newly detected, actionable tasks into tasks/antigravity.md under ## 📥 Pending Tasks.
  • Telegram Notification Policy: Alerts Topic #IT Guy (thread 119) if new issues are filed; returns NO_REPLY for intentional silence when clean.

🚨 Troubleshooting & Incident Playbook

Important

Mandatory Bug & Incident Documentation: Whenever a bug, outage, schema error, or operational regression is fixed in OpenClaw, always document the incident in TROUBLESHOOTING.md with symptom, root cause, resolution, and how to avoid recurrence.

🛡️ Operational & Diagnostic Invariants (Learned Guidelines)

Whenever troubleshooting or maintaining OpenClaw, agents MUST adhere to these rules:

  1. Rapid Incident Triaging & Information Gathering (< 2s):

    • When any issue, silence, or strange bot behavior is reported, run the one-shot triage tool first: 
      kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 /home/node/.openclaw/scripts/triage.py --topic <topicId>
    • Do NOT run piecemeal SQLite queries, guess table/column names, or scan the container filesystem.
    • Path Anchors (Never run find / or broad grep -r on rotational disk):
      • Knowledge Wiki: /home/node/.openclaw/wiki/main
      • Agent Databases: /home/node/.openclaw/agents/<agent>/agent/openclaw-agent.sqlite
      • Gateway State DB: /home/node/.openclaw/state/openclaw.sqlite
      • Standalone Triage Script: /home/node/.openclaw/scripts/triage.py (Repo: k8s/openclaw/scripts/openclaw-triage.py)

  2. Verify Before Declaring Dead (Turn In-Flight Diagnostics):

    • When a user reports that a Telegram topic is "not answering" or "silent", NEVER assume the container or gateway has crashed.
    • Always inspect live pod logs (kubectl logs openclaw-0 -n openclaw -c openclaw --tail=100) and the agent's SQLite database (session_nodes and transcript_events) first.
    • Check whether a turn is currently executing: multi-turn tool loops or inline safeguard compaction (compact:start) can introduce multi-minute turn latency without any pod restart.

  3. Datastore Ground-Truth for Maintenance:

    • High-level CLI summaries (openclaw sessions --all-agents --json) report totalTokens: null (totalTokensFresh: false) for stale sessions or after restarts.
    • Automated health checks and compaction scripts MUST query the SQLite database directly (transcript_events event count and byte size), triggering compaction when bytes >= 300KB or events >= 40, regardless of cached token values.

  4. Compaction User Visibility:

    • Always ensure agents.defaults.compaction.notifyUser: true remains set in OpenClaw configuration. Live inline compactions must send visible status feedback (🧹 Compacting context...) to avoid perceived bot stalls.

  5. In-Pod Container Restart Self-Healing:

    • Critical database unlocking, lockfile cleanup, and state recovery MUST run via runtime preflight (NODE_OPTIONS="--import=...") or application startup hooks.
    • Never place critical self-healing logic exclusively in initContainers, as Kubelet container restarts inside an existing Pod completely bypass initContainers.

  6. Rotational Storage (HDD) Page-Cache Pre-Warming:

    • Host haumdaucher uses rotational mechanical disks (/dev/sda2, rotational: 1).
    • Whenever the pod or container restarts, SQLite databases MUST be sequentially streamed to /dev/null (cat ... > /dev/null) to pre-warm the Linux page cache, avoiding single-threaded event loop starvation on cold seeks.

  7. Zero-Downtime Wiki Prompt Updates vs. Heavy Pod Rollouts:

    • Subagent prompt instructions, tone, and tool guidelines live in knowledge-wiki/agents/<name>.md. Edits made there are read live by OpenClaw on git sync without any pod restart.
    • Avoid modifying openclaw-instance.secret.yaml unless modifying container limits, environment variables, or global operator configuration. A CR edit triggers a 7-init-container rolling update (~3 minutes).

  8. Continuous Troubleshooting Evolution & The Simplicity Invariant (The 98% Rule):

    • Post-Fix Dual Reflection: At the end of every troubleshooting cycle and fix, the AI MUST proactively evaluate:
      1. Troubleshooting Optimization: How can the diagnostic process be made faster or codified into triage.py / openclaw-ops?
      2. System Maturity: How can the root cause be hardened permanently to prevent recurrence?
    • Simplicity First: Simplicity and straightforward architecture are the central design principles. A simple, pragmatic solution that cleanly reaches 98% of the goal is strictly preferred over an over-engineered abstraction attempting 100%. Never introduce background workers, multi-stage state machines, or silent fallbacks when a transparent single-step tool or direct alert suffices.

For known issues, resolved bugs, and specific failure playbooks, consult TROUBLESHOOTING.md.

1. Verify Deployment Health

Ensure that the OpenClawInstance is successfully reconciled:

kubectl get openclawinstance openclaw -n openclaw
# Expected: PHASE=Running, READY=True

2. Verify Pod & Container States

Check that all 6 containers inside the StatefulSet are fully functional:

kubectl get pods -n openclaw -l app.kubernetes.io/instance=openclaw

3. Check Live Logs

Inspect the core gateway engine:

kubectl logs -n openclaw openclaw-0 -c openclaw --tail=100

Inspect the Chromium browser engine:

kubectl logs -n openclaw openclaw-0 -c chromium --tail=100
Reference in New Issue View Git Blame Copy Permalink