diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index 1c0e20f..edb8050 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -17,6 +17,7 @@ This document captures fixed issues, operational quirks, and incident root-cause | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. | | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. | | **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. | +| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. | --- @@ -167,3 +168,34 @@ This document captures fixed issues, operational quirks, and incident root-cause - **How to Avoid Recurrence**: Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance. +--- + +### Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck + +- **Date**: 2026-09-29 +- **Severity**: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in `Dl` state) +- **Symptom**: + 1. Telegram bot stopped answering messages entirely on the morning of Sep 29. + 2. `openclaw-0` pod was killed by kubelet with exit code 137 (`Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded`). + 3. Upon restarting, the container remained in `4/5 Running` for over 15 minutes, with the gateway process (`openclaw-gateway`, PID 15930) stuck in Linux uninterruptible sleep state `Dl` (`generic_file_buffered_read`) and logging repeated: + ```text + [state/agent-db] slow OpenClaw agent database open + ``` +- **Root Cause**: + 1. **Recurring Hardlink Regression**: The nightly `openclaw-wiki-backup` CronJob runs daily at 03:30 UTC. Lines 41-42 in `cronjob-wiki-backup.secret.yaml` contained: + ```bash + [ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true + [ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true + ``` + This routinely re-created hardlinks (`nlink == 11`) across all subagent workspaces every night, directly undoing the `nlink == 1` fix from Issue 4 and causing OpenClaw to reject `USER.md` and `MEMORY.md`. When a user message arrived in `#Mila` at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137). + 2. **Spinning Disk Random I/O Bottleneck on Startup**: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs `verifyAndRepairCanonicalSqliteIndexes` on each database. On rotational SATA hard drives (`/dev/sda2`, `rotational=1`), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold. +- **Resolution**: + 1. **Permanent Manifest Fix**: Patched `k8s/openclaw/cronjob-wiki-backup.secret.yaml` to replace `ln -f` with `cp --remove-destination`, ensuring subagent workspace bootstrap files remain strict standalone files (`nlink == 1`). Applied the updated manifest to the cluster and pushed commit `74e32ed` to git. + 2. **Volume Decoupling**: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies. + 3. **Page Cache Pre-warming**: Sequentially streamed each agent SQLite database into the Linux RAM page cache (`cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null`). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent. + 4. The gateway successfully bound `:18789`, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy `5/5 Running` state. +- **How to Avoid Recurrence**: + - Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`. + - When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage. + +