docs(openclaw): document issue 7 root cause and resolution in troubleshooting guide

2026-09-29 08:08:42 +02:00
parent 74e32ed6bf
commit 5bf1a29f4a
1 changed files with 32 additions and 0 deletions
@@ -17,6 +17,7 @@ This document captures fixed issues, operational quirks, and incident root-cause
 | **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
 | **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
 | **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |
+| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |

 ---

@@ -167,3 +168,34 @@ This document captures fixed issues, operational quirks, and incident root-cause
 - **How to Avoid Recurrence**:
  Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.

+---
+
+### Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck
+
+- **Date**: 2026-09-29
+- **Severity**: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in `Dl` state)
+- **Symptom**:
+  1. Telegram bot stopped answering messages entirely on the morning of Sep 29.
+  2. `openclaw-0` pod was killed by kubelet with exit code 137 (`Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded`).
+  3. Upon restarting, the container remained in `4/5 Running` for over 15 minutes, with the gateway process (`openclaw-gateway`, PID 15930) stuck in Linux uninterruptible sleep state `Dl` (`generic_file_buffered_read`) and logging repeated:
+     ```text
+     [state/agent-db] slow OpenClaw agent database open
+     ```
+- **Root Cause**:
+  1. **Recurring Hardlink Regression**: The nightly `openclaw-wiki-backup` CronJob runs daily at 03:30 UTC. Lines 41-42 in `cronjob-wiki-backup.secret.yaml` contained:
+     ```bash
+     [ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true
+     [ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true
+     ```
+     This routinely re-created hardlinks (`nlink == 11`) across all subagent workspaces every night, directly undoing the `nlink == 1` fix from Issue 4 and causing OpenClaw to reject `USER.md` and `MEMORY.md`. When a user message arrived in `#Mila` at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137).
+  2. **Spinning Disk Random I/O Bottleneck on Startup**: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs `verifyAndRepairCanonicalSqliteIndexes` on each database. On rotational SATA hard drives (`/dev/sda2`, `rotational=1`), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold.
+- **Resolution**:
+  1. **Permanent Manifest Fix**: Patched `k8s/openclaw/cronjob-wiki-backup.secret.yaml` to replace `ln -f` with `cp --remove-destination`, ensuring subagent workspace bootstrap files remain strict standalone files (`nlink == 1`). Applied the updated manifest to the cluster and pushed commit `74e32ed` to git.
+  2. **Volume Decoupling**: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies.
+  3. **Page Cache Pre-warming**: Sequentially streamed each agent SQLite database into the Linux RAM page cache (`cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null`). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent.
+  4. The gateway successfully bound `:18789`, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy `5/5 Running` state.
+- **How to Avoid Recurrence**:
+  - Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`.
+  - When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.
+
+