diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index 5dac100..a453dbb 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -13,7 +13,9 @@ This document captures fixed issues, operational quirks, and incident root-cause | :--- | :--- | :--- | :---: | :--- | | **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). | | **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. | -| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Probe budget vs WAL checkpointing on large SQLite files (~1.5GB across subagents). | +| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. | +| **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. | +| **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. | --- @@ -84,7 +86,57 @@ This document captures fixed issues, operational quirks, and incident root-cause - `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files. On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases. - **Resolution**: - 1. Ensure `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` allows adequate startup headroom (e.g. at least 120 attempts with 5s period = 600s). - 2. Periodically ensure SQLite VACUUM / WAL checkpoints run during maintenance windows or let the database open without interrupting it. + 1. Increase `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` to 240 (with 5s period = 1200s / 20 min). + 2. Periodically run `PRAGMA wal_checkpoint(TRUNCATE)` across all SQLite databases to compact WAL journals into main files. - **How to Avoid Recurrence**: Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization. + +--- + +### Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary + +- **Date**: 2026-09-28 +- **Severity**: High (Subagents lose access to user profile and memory context) +- **Symptom**: + On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports: + ```text + [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-/USER.md reason=path must not be hardlinked + [workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-/MEMORY.md reason=path must not be hardlinked + ``` +- **Root Cause**: + OpenClaw’s embedded execution harness enforces a strict filesystem security check: `stat.nlink > 1` triggers `path must not be hardlinked`. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files. +- **Resolution**: + Decouple all hardlinked files so that each workspace has its own standalone copy with `nlink == 1`: + ```bash + for dir in /home/node/.openclaw/workspace-*; do + for file in USER.md MEMORY.md; do + cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file" + done + done + ``` + Ensure the init container in `openclaw-instance.secret.yaml` maintains this pattern with `cp --remove-destination`. +- **How to Avoid Recurrence**: + Never create hardlinks (`ln` without `-s`) inside OpenClaw workspace directories. Always use standalone file copies (`cp --remove-destination`). + +--- + +### Issue 5: Cron Trigger Sandbox Scope ReferenceError + +- **Date**: 2026-09-28 +- **Severity**: Medium (Automated recurring cron job fails to evaluate) +- **Symptom**: + In `cron_run_receipts`, the job `rennerhof-website-watch` failed with: + ```text + cron trigger evaluation failed (internal_error): ReferenceError: state is not defined + at (openclaw-code-mode:user.js:12:172) + ``` +- **Root Cause**: + The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access `state && state.hash` directly, causing a `ReferenceError` when `state` is undeclared in the global scope. +- **Resolution**: + Use `typeof state !== "undefined"` to safely guard against undeclared variables in the trigger script: + ```javascript + const s = (typeof state !== 'undefined' && state) ? state : {}; + const prev = s.hash; + ``` +- **How to Avoid Recurrence**: + In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard. diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index a4828b6..4530447 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/knowledge-wiki b/knowledge-wiki index 1297690..2772e1b 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit 1297690940c3503410ae1c52787092f2aa80d243 +Subproject commit 2772e1b4b864149f29952f11ce119804ae574029