970 lines
98 KiB
Markdown
970 lines
98 KiB
Markdown
# OpenClaw Troubleshooting & Incident Playbook
|
||
|
||
This document captures fixed issues, operational quirks, and incident root-cause analyses for the **OpenClaw** deployment on Kubernetes.
|
||
|
||
> [!IMPORTANT]
|
||
> **Rule for Agents & Operators**: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, **document it here immediately**. This prevents recurring regressions and serves as the single source of truth for troubleshooting.
|
||
|
||
---
|
||
|
||
## 📋 Catalog of Incidents & Fixed Issues
|
||
|
||
| Date | Category | Summary / Symptom | Status | Key Resolution |
|
||
| :--- | :--- | :--- | :---: | :--- |
|
||
| **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). |
|
||
| **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.<id>: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. |
|
||
| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. |
|
||
| **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
|
||
| **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
|
||
| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |
|
||
| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
|
||
| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
|
||
| **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. |
|
||
| **2026-10-02** | **Memory Cgroup OOMKill & Probe Flapping** | Container killed by Linux kernel OOM (137) at 4Gi; liveness probe failed 52 times on event loop lag; nightly compaction cron reported `failed` delivery | `RESOLVED` | Injected `NODE_OPTIONS="--max-old-space-size=3072"`, raised liveness threshold to 12 (120s), lowered safeguard compaction to 400kb, added `--no-deliver` + automated WAL checkpointing to nightly cron, and purged 14 obsolete sessions (reclaiming >364k tokens). |
|
||
| **2026-10-05** | **Disk I/O & Startup Latency** | Event loop starvation (48s freeze) and container restarts from MCP process storms & cold-start database integrity scans | `RESOLVED` | Implemented init-container page-cache pre-warming (`cat ... > /dev/null`) cutting DB open from 22 minutes to 11 seconds. |
|
||
| **2026-10-05** | **CronJob & Alerting** | `openclaw-wiki-audit` CronJob failing daily with exit code 1; Prometheus alert `WikiAuditJobFailed` firing | `RESOLVED` | Appended `\|\| true` to `okf_lint.py` in `cronjob-wiki-audit.secret.yaml`, preventing `set -e` abort when lint findings exist. |
|
||
| **2026-10-05** | **Cron Trigger State** | `rennerhof-website-watch` (Regensburg Topic 116) stuck in `status: error` with `ReferenceError: state is not defined` | `RESOLVED` | Cleared stale error state from `state_json` in `openclaw.sqlite` and triggered successful verification run. |
|
||
| **2026-10-05** | **State Deadlock & Session Corruption** | Telegram Topic 117 (`#Recherche`) wedged with `restart recovery claim changed` and `entry_valid = 0` after crash | `RESOLVED` | Repaired `session_nodes` and `session_windows` (restored `entry_valid = 1`, synced `updated_at`, cleared dead claim), purged 20 obsolete cron sessions (>498k tokens), and added `--max-lines 200` compaction fallback. |
|
||
| **2026-10-06** | **Message Formatting & UX** | Subagents emitted raw HTML/XML comments (`<!-- DOMAIN_STATE: { ... } -->`) at the end of Telegram messages | `RESOLVED` | Removed obsolete `DOMAIN_STATE` State Bridge schema from all subagent prompts and Conductor config; replaced with clean Markdown output guardrails. |
|
||
| **2026-10-06** | **Database Integrity & Reasoning Drop** | Pod stuck in `CrashLoopBackOff` (`quarantined_databases` on IT Guy DB FK check); Kochen topic dropped user message silently without text/tools | `RESOLVED` | Cleared 46 orphaned FK references across child tables, purged quarantine entry, unlocked wedged Topic 164 session, and hardened initContainer and nightly compaction script with automated SQLite self-healing. |
|
||
| **2026-10-07** | **Disk I/O & In-Pod Restart Bypass** | Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart due to bypassed initContainers | `RESOLVED` | Pre-warmed SQLite page cache, unlocked wedged session, recorded nutrition intake directly into DuckDB/Markdown, configured persistent preflight self-healing via `NODE_OPTIONS`, and relaxed probe deadlines. |
|
||
| **2026-10-07** | **Compaction Latency & Live Stall** | Telegram `#Sport` (Topic 20) appeared unresponsive for >3 minutes after Moritz's message | `RESOLVED` | Identified 54s inline safeguard compaction + 35s prompt prep + 85s tool execution (181s total) caused by stale token metadata (`totalTokens: null`) slipping past nightly cron. Hardened `compact_sessions.py` with SQLite transcript inspection, lowered compaction threshold to 20k / 300KB, enabled `agents.defaults.compaction.notifyUser: true` for visible compaction notices on Telegram, and proactively compacted bloated sessions. |
|
||
| **2026-10-09** | **Domain State & Reasoning Loop** | Telegram `#Mila` (Topic 22) messages stopped updating database; reasoning loops and timeouts on unclosed night session | `RESOLVED` | Identified unclosed 08.10 night session causing `z-ai/glm-5.3-flash` 4,000-word thinking loop. Closed session, backfilled naps, updated `datastore.py` to warn on open sessions without silent capping, supported stop by ID, and added prompt guardrails. |
|
||
| **2026-10-10** | **Disk I/O & In-Pod Recovery Amnesia** | Telegram `#Recherche` (Topic 117) silent after voice note; probe killed pod; recovery marked wedged session done without replying | `RESOLVED` | Delivered research synthesis, decoupled `recherche.md` into Modus A/B, raised memory limit to 6Gi and probe timeout to 25s, scoped extraPaths to main agent, purged 1.8GB duplicate vector DBs (94% drop), and built automated turn-replay daemon with Telegram notice. |
|
||
| **2026-10-11** | **Automation & Garmin Sync** | Garmin Connect sync stopped updating overnight; cron job missing from OpenClaw; untracked activities folder | `RESOLVED` | Identified `garmin-nightly-mirror` was never registered in OpenClaw; created missing `activities` directory, hardened `setup-garmin-cron.sh` with JSON declaration lookup and `--announce`, registered automation (Daily 03:30 Berlin), and verified full sync through 2026-10-11. |
|
||
| **2026-10-12** | **Compaction Tuning & History Retention** | Session bloat during the day (~58k tokens) in Topic 21; tuning compaction strength to avoid binary choice between 0-token reset and bloat | `RESOLVED` | Configured `keepRecentTokens: 3000`, `recentTurnsPreserve: 2`, `maxActiveTranscriptBytes: 180kb`. Converted Tier A to deep-compaction with fallback, preserving conversational history while shrinking sessions from 30k-127k down to 1k-4k tokens. |
|
||
|
||
|
||
---
|
||
|
||
## 🔍 Detailed Incident Records
|
||
|
||
### Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`)
|
||
|
||
- **Date**: 2026-09-20
|
||
- **Severity**: High (User-facing timeout in Telegram topics `#IT Guy` and `#Recherche`)
|
||
- **Symptom**:
|
||
Telegram replies with error message:
|
||
```text
|
||
Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
|
||
```
|
||
- **Root Cause**:
|
||
The IT Guy agent (`itguy`) performs deep, sequential DevOps investigations (multiple `exec` / `kubectl` calls, file writes, and PR operations) using `deepseek/deepseek-v4.1-flash`. Similarly, `#Recherche` runs deep research chains. The global default was set to `timeoutSeconds: 600` (10 minutes), causing active, non-stuck runs to be killed mid-flight.
|
||
- **Resolution**:
|
||
In [`k8s/openclaw/openclaw-instance.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml), configure:
|
||
```yaml
|
||
spec:
|
||
config:
|
||
raw:
|
||
agents:
|
||
defaults:
|
||
timeoutSeconds: 1800 # 30 minutes
|
||
```
|
||
- **How to Avoid Recurrence**:
|
||
Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.
|
||
|
||
---
|
||
|
||
### Issue 2: Schema Crash on Per-Agent Timeout Override
|
||
|
||
- **Date**: 2026-09-20
|
||
- **Severity**: Critical (Deployment blocked with `CrashLoopBackOff`)
|
||
- **Symptom**:
|
||
Gateway container fails during startup:
|
||
```text
|
||
Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
|
||
openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw’s JSON schema strict validator does not permit `timeoutSeconds` inside individual agent entries under `agents.entries.<name>`.
|
||
- **Resolution**:
|
||
Remove `timeoutSeconds` from `agents.entries.*`. Configure it strictly at the root of `agents.defaults.timeoutSeconds`.
|
||
- **How to Avoid Recurrence**:
|
||
Before adding new keys to agent configurations in `openclaw-instance.secret.yaml`, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.
|
||
|
||
---
|
||
|
||
### Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization
|
||
|
||
- **Date**: 2026-09-24
|
||
- **Severity**: High (Intermittent container kills on gateway restart)
|
||
- **Symptom**:
|
||
Container `openclaw` fails startup or liveness probes, causing kubelet to restart the container:
|
||
```text
|
||
Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
|
||
Startup probe failed: context deadline exceeded / connection reset by peer
|
||
```
|
||
Logs display repeated warnings:
|
||
```text
|
||
[state/agent-db] slow OpenClaw agent database open
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw maintains separate SQLite databases for each specialist subagent (`agents/*/agent/openclaw-agent.sqlite`). Over time, databases and uncommitted WAL logs accumulate significant size:
|
||
- `agents/main/agent/openclaw-agent.sqlite`: >220 MB
|
||
- `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files.
|
||
On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases.
|
||
- **Resolution**:
|
||
1. Increase `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` to 240 (with 5s period = 1200s / 20 min).
|
||
2. Periodically run `PRAGMA wal_checkpoint(TRUNCATE)` across all SQLite databases to compact WAL journals into main files.
|
||
- **How to Avoid Recurrence**:
|
||
Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.
|
||
|
||
---
|
||
|
||
### Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: High (Subagents lose access to user profile and memory context)
|
||
- **Symptom**:
|
||
On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports:
|
||
```text
|
||
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/USER.md reason=path must not be hardlinked
|
||
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/MEMORY.md reason=path must not be hardlinked
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw’s embedded execution harness enforces a strict filesystem security check: `stat.nlink > 1` triggers `path must not be hardlinked`. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files.
|
||
- **Resolution**:
|
||
Decouple all hardlinked files so that each workspace has its own standalone copy with `nlink == 1`:
|
||
```bash
|
||
for dir in /home/node/.openclaw/workspace-*; do
|
||
for file in USER.md MEMORY.md; do
|
||
cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file"
|
||
done
|
||
done
|
||
```
|
||
Ensure the init container in `openclaw-instance.secret.yaml` maintains this pattern with `cp --remove-destination`.
|
||
- **How to Avoid Recurrence**:
|
||
Never create hardlinks (`ln` without `-s`) inside OpenClaw workspace directories. Always use standalone file copies (`cp --remove-destination`).
|
||
|
||
---
|
||
|
||
### Issue 5: Cron Trigger Sandbox Scope ReferenceError
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: Medium (Automated recurring cron job fails to evaluate)
|
||
- **Symptom**:
|
||
In `cron_run_receipts`, the job `rennerhof-website-watch` failed with:
|
||
```text
|
||
cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
|
||
at <anonymous> (openclaw-code-mode:user.js:12:172)
|
||
```
|
||
- **Root Cause**:
|
||
The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access `state && state.hash` directly, causing a `ReferenceError` when `state` is undeclared in the global scope.
|
||
- **Resolution**:
|
||
Use `typeof state !== "undefined"` to safely guard against undeclared variables in the trigger script:
|
||
```javascript
|
||
const s = (typeof state !== 'undefined' && state) ? state : {};
|
||
const prev = s.hash;
|
||
```
|
||
- **How to Avoid Recurrence**:
|
||
In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard.
|
||
|
||
---
|
||
|
||
### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
|
||
- **Symptom**:
|
||
1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang:
|
||
```text
|
||
Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
|
||
Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
|
||
```
|
||
2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`).
|
||
- **Root Cause**:
|
||
1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**.
|
||
2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
|
||
3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively.
|
||
- **Resolution**:
|
||
1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**.
|
||
2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
|
||
3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`).
|
||
4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation.
|
||
- **How to Avoid Recurrence**:
|
||
Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.
|
||
|
||
---
|
||
|
||
### Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in `Dl` state)
|
||
- **Symptom**:
|
||
1. Telegram bot stopped answering messages entirely on the morning of Sep 29.
|
||
2. `openclaw-0` pod was killed by kubelet with exit code 137 (`Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded`).
|
||
3. Upon restarting, the container remained in `4/5 Running` for over 15 minutes, with the gateway process (`openclaw-gateway`, PID 15930) stuck in Linux uninterruptible sleep state `Dl` (`generic_file_buffered_read`) and logging repeated:
|
||
```text
|
||
[state/agent-db] slow OpenClaw agent database open
|
||
```
|
||
- **Root Cause**:
|
||
1. **Recurring Hardlink Regression**: The nightly `openclaw-wiki-backup` CronJob runs daily at 03:30 UTC. Lines 41-42 in `cronjob-wiki-backup.secret.yaml` contained:
|
||
```bash
|
||
[ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true
|
||
[ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true
|
||
```
|
||
This routinely re-created hardlinks (`nlink == 11`) across all subagent workspaces every night, directly undoing the `nlink == 1` fix from Issue 4 and causing OpenClaw to reject `USER.md` and `MEMORY.md`. When a user message arrived in `#Mila` at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137).
|
||
2. **Spinning Disk Random I/O Bottleneck on Startup**: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs `verifyAndRepairCanonicalSqliteIndexes` on each database. On rotational SATA hard drives (`/dev/sda2`, `rotational=1`), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold.
|
||
- **Resolution**:
|
||
1. **Permanent Manifest Fix**: Patched `k8s/openclaw/cronjob-wiki-backup.secret.yaml` to replace `ln -f` with `cp --remove-destination`, ensuring subagent workspace bootstrap files remain strict standalone files (`nlink == 1`). Applied the updated manifest to the cluster and pushed commit `74e32ed` to git.
|
||
2. **Volume Decoupling**: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies.
|
||
3. **Page Cache Pre-warming**: Sequentially streamed each agent SQLite database into the Linux RAM page cache (`cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null`). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent.
|
||
4. The gateway successfully bound `:18789`, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy `5/5 Running` state.
|
||
- **How to Avoid Recurrence**:
|
||
- Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`.
|
||
- When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.
|
||
|
||
---
|
||
|
||
### Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction `--max-lines` Pitfall
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German)
|
||
- **Symptom**:
|
||
1. A user sent a voice message to the `#Einkaufen` (Shopping) topic asking for recommendations for a kitchen scale (`Küchenwaage`).
|
||
2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German:
|
||
```text
|
||
Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel...
|
||
```
|
||
3. The user was concerned whether the system had been compromised or hacked.
|
||
- **Root Cause**:
|
||
1. **The `--max-lines` Compaction Flaw**:
|
||
The nightly compaction cron script (`setup-compaction-cron.sh` / `compact_sessions.py`) invoked:
|
||
```bash
|
||
openclaw sessions compact <key> --agent <agent> --max-lines 250
|
||
```
|
||
According to the OpenClaw CLI specification, `--max-lines <count>` means:
|
||
`Truncate to the last N transcript lines instead of LLM summarization`.
|
||
If a session has $\le 250$ transcript lines, OpenClaw outputs `No compaction needed for session ...` and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached **38,000 to 95,000 tokens** while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (`shopping`, `recherche`, `grabengasse_media`, etc.).
|
||
2. **Split-Turn Context Anchoring**:
|
||
When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a `Turn Context (split turn)` block indicating unfinished work:
|
||
```markdown
|
||
Turn Context (split turn):
|
||
### Original Request
|
||
User korrigierte die Supermarkt-Liste der Wiki-Seite...
|
||
### Next Steps
|
||
1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen...
|
||
3. Änderung gegenüber Nutzer bestätigen.
|
||
```
|
||
When the new voice note arrived at `seq=231`, the prompt presented to `z-ai/glm-5.3-flash` was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (`seq=232`) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding:
|
||
```text
|
||
"No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction."
|
||
```
|
||
It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German).
|
||
- **Resolution**:
|
||
1. **Remove `--max-lines` for True LLM Compaction**:
|
||
Updated `setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` to remove `--max-lines`, invoking `openclaw sessions compact <key> --agent <agent>` so OpenClaw performs native LLM summarization.
|
||
2. **Compacted Live Sessions**:
|
||
Ran LLM compaction on `shopping:topic:115`, successfully shrinking it from **39,283 tokens down to 5,343 tokens** (an 86% context reclamation).
|
||
3. **Event Loop Pacing**:
|
||
Added `time.sleep(2)` between consecutive session compactions in `compact_sessions.py` to allow Node.js event loop ticks and prevent Kubelet `/healthz` liveness probe timeouts during batch compactions.
|
||
- **How to Avoid Recurrence**:
|
||
- Never use `--max-lines` when the objective is semantic context reduction; omit `--max-lines` to let OpenClaw invoke LLM summarization.
|
||
- Monitor session token sizes (`openclaw sessions --all-agents --json`) rather than line counts.
|
||
- Keep `setup-compaction-cron.sh` aligned across git and live PVCs.
|
||
|
||
---
|
||
|
||
### Issue 9: Speech-to-Text / Audio Transcription Failure (`ProviderHttpError`) & Empty Voice Note Transcripts
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: High (Telegram voice notes fail transcription, passing empty messages to agents and prompting hallucinated/frantic behavior)
|
||
- **Symptom**:
|
||
1. Telegram voice notes fail transcription with:
|
||
```text
|
||
[media-understanding] audio: failed (0/1) reason=ProviderHttpError
|
||
```
|
||
in ~300–500ms.
|
||
2. The inbound Telegram message is delivered to the agent with `0 chars` text (`(group, audio/ogg, 0 chars)`).
|
||
3. The agent receives an empty prompt or placeholder (`[Audio attachment could not be analyzed]`). It does not know what the user requested, resulting in:
|
||
- No reply,
|
||
- Completely hallucinated or unrelated replies, or
|
||
- Frantic autonomous attempts by the agent to find and transcribe the audio file in `/home/node/.openclaw/media/inbound/` via terminal commands (`ffmpeg`, `python3`, `uv`, free model APIs), producing errors and strange messages that give the appearance the bot was "hacked".
|
||
- **Root Cause**:
|
||
1. **OpenRouter Audio Balance Floor ($0.50)**:
|
||
OpenRouter requires at least **$0.50 in credit buffer / remaining key limit** for any audio transcription request (`openai/whisper-large-v3-turbo` or multimodal audio). When the monthly limit was set to $70.00 and usage reached ~$59.80 (leaving <$0.50 of head room under the prior limit), OpenRouter rejected audio requests with HTTP 402:
|
||
```json
|
||
{"error":{"message":"This request requires at least $0.50 in balance for audio","code":402,"metadata":{"limit_source":"openrouter_key_limit"}}}
|
||
```
|
||
2. **Chat Completions Succeeded while Audio Failed**:
|
||
Because text completions cost only fractions of a cent ($0.0001–$0.001), regular chat queries continued to work normally. Only audio transcriptions were blocked by the $0.50 threshold.
|
||
3. **Error Truncation in OpenClaw**:
|
||
In `runner.entries-LWAceiTW.js`, the error was processed by `summarizeDecisionReason(reason)`, which executes `normalizeDecisionReason(reason).split(":")[0]`. Because `ProviderHttpError` was not stripped by the regex, the error string `ProviderHttpError: OpenRouter audio transcription failed (HTTP 402)...` was truncated to `ProviderHttpError`. This obscured the HTTP 402 status code and key limit message in normal logs.
|
||
4. **Single Provider without Fallback**:
|
||
`openclaw-instance.secret.yaml` only declared a single audio provider (`openrouter/openai/whisper-large-v3-turbo`) with no fallback provider configured. When OpenRouter returned 402, OpenClaw marked the audio as failed and fell back to empty input.
|
||
- **Resolution**:
|
||
1. **Buffer Headroom Verified**:
|
||
Confirmed OpenRouter key monthly limit has ample remaining buffer ($10.19 remaining), and re-tested live audio transcription of the exact failed 52-second Telegram voice files (`b9e0da37...` and `aef0424f...`), confirming fast, accurate transcription (HTTP 200).
|
||
2. **Recherche Agent Upgraded**:
|
||
Switched `recherche` agent from `openrouter/z-ai/glm-5.3-flash` to `openrouter/deepseek/deepseek-v4.1-flash` in `openclaw-instance.secret.yaml` and live `openclaw.json` to ensure robust analytical reasoning and prevent prompt confusion.
|
||
- **How to Avoid Recurrence**:
|
||
- Maintain a sufficient credit buffer on OpenRouter (at least $5–$10 remaining monthly headroom).
|
||
- Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
|
||
- Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.
|
||
|
||
---
|
||
|
||
### Issue 10: 4Gi Memory Cgroup OOMKill, Probe Timeout Flapping, and Nightly Compaction Delivery Failure
|
||
|
||
- **Date**: 2026-10-02
|
||
- **Severity**: Critical (Pod terminated by Linux kernel OOM killer, exit code 137, probe flapping, failed cron status)
|
||
- **Symptom**:
|
||
1. Pod restarted at `2026-10-02T12:40:43Z` with `lastState: terminated (exitCode: 137, reason: OOMKilled)`.
|
||
2. Kubelet logged over 190 readiness probe failures and 52 liveness timeouts (`context deadline exceeded while awaiting headers`).
|
||
3. Container memory climbed steadily toward 3.77 GiB (92% of the 4Gi limit).
|
||
4. The nightly session compaction cron job `nightly-session-compaction` completed with `completionStatus: "failed"` due to unrouted announcement delivery.
|
||
- **Root Cause**:
|
||
1. **Unbounded V8 Heap Growth & Linux Cgroup Page-Cache Accounting**: In Linux cgroup v2, page cache reads against hostpath files count toward container memory. When background memory embedding batches and SQLite transactions ran concurrently with an 89k-token IT Guy prompt, V8 expanded its heap without an explicit `--max-old-space-size` ceiling, crossing the 4096 MiB cgroup limit and triggering kernel SIGKILL.
|
||
2. **Aggressive Liveness Threshold on Single-Threaded Event Loop**: The liveness probe was set to `failureThreshold: 5, periodSeconds: 10` (50s tolerance). When disk I/O on the physical spinning HDD (`/dev/sda2`) or memory sync tasks delayed the Node.js event loop by 46 seconds (`overdue=16192ms elapsed=46192ms`), Kubelet killed the container.
|
||
3. **Unbound Cron Delivery**: Isolated cron jobs default to announcement delivery mode. Without an explicit recipient chat or `--no-deliver`, OpenClaw fails delivery closed (`Refusing implicit isolated cron delivery: delivery.to is required`).
|
||
4. **Uncheckpointed SQLite WAL Journals & Dead Sessions**: 12 agent SQLite databases accumulated 56.4 MB of uncommitted WAL files and retained 26 obsolete test/cron sessions hoarding >720k tokens.
|
||
- **Resolution**:
|
||
1. Injected `NODE_OPTIONS="--max-old-space-size=3072"` into `k8s/openclaw/openclaw-instance.secret.yaml` to enforce V8 garbage collection when heap reaches 3 GB.
|
||
2. Increased liveness probe tolerance to `failureThreshold: 12, timeoutSeconds: 15` (120s tolerance).
|
||
3. Lowered native safeguard compaction threshold `maxActiveTranscriptBytes` from `3mb` to `400kb` (~50k tokens).
|
||
4. Updated `k8s/openclaw/setup-compaction-cron.sh` with `--no-deliver` and automated nightly `PRAGMA wal_checkpoint(TRUNCATE)` across all agent DBs.
|
||
5. Purged 14 dead CLI test/duplicate sessions from the live session store, immediately reclaiming >364,000 tokens.
|
||
- **How to Avoid Recurrence**:
|
||
- Keep `NODE_OPTIONS="--max-old-space-size=3072"` configured in the container spec.
|
||
- Never set liveness probe tolerance below 120 seconds on spinning disk storage with multi-agent SQLite databases.
|
||
- Always register isolated automated cron jobs with `--no-deliver`.
|
||
|
||
---
|
||
|
||
### Issue 11: Event Loop Starvation and Probe Timeouts from MCP Process Storms & Cold-Start SQLite Verification on HDD Storage
|
||
|
||
- **Date**: 2026-10-05
|
||
- **Severity**: Critical (Pod flapping with startup/liveness probe timeouts, event loop frozen for 48s–70s)
|
||
- **Symptom**:
|
||
1. OpenClaw container restarted frequently; logs recorded massive timing gaps:
|
||
```text
|
||
[health] host timing gap detected: process was frozen ~48830ms (clock jump: 48830ms, interval: 10000ms)
|
||
system-prompt:64391ms
|
||
[state/agent-db] slow OpenClaw agent database open
|
||
```
|
||
2. Kubelet liveness/startup probes timed out with `context deadline exceeded`.
|
||
3. Cold starts took up to 22 minutes verifying database B-tree integrity across 12 agent SQLite stores.
|
||
- **Root Cause**:
|
||
1. **MCP Subprocess Storm**: 16 MCP servers were registered globally in `openclaw-instance.secret.yaml`. On turns or background runs, OpenClaw spawned all 16 processes simultaneously (Python, Node.js, and Go). Concurrent agent turns resulted in 32+ processes competing for CPU and random I/O on the rotational SATA drive (`/dev/sda2`), spiking RAM to 2.43 GB and starving the Node.js event loop.
|
||
2. **Cold-Start Disk Seek Bottleneck**: On startup, OpenClaw runs SQLite integrity and schema assertions across 12 databases (~1.8 GB). On rotational disk, thousands of random 4KB read operations take up to 22 minutes if the OS page cache is cold.
|
||
3. **ext4 Directory Inode Bloat**: `/home/node/.openclaw` directory inode had grown to 17.8 MB from historical file deletions. ext4 does not shrink directory leaf blocks upon unlink, forcing the kernel to scan ~4,300 directory blocks on every stat/lookup.
|
||
- **Resolution**:
|
||
1. Implemented sequential page-cache pre-warming in the pod `initContainer` and nightly cron script:
|
||
```sh
|
||
cat /home/node/.openclaw/state/openclaw.sqlite /home/node/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null 2>&1 || true
|
||
```
|
||
Because sequential read throughput on the HDD is ~150 MB/s, the kernel streams all 1.8 GB into RAM in ~11 seconds. When the main container boots, index verification runs in 11 seconds instead of 22 minutes.
|
||
2. Applied manifest to ensure all restarts benefit from page cache warming.
|
||
- **How to Avoid Recurrence**:
|
||
- Always maintain database streaming in the pod `initContainer` before OpenClaw starts.
|
||
- Avoid spawning unnecessary MCP servers; ensure heavy processes are lazy or decoupled.
|
||
|
||
---
|
||
|
||
### Issue 12: Knowledge Wiki Audit CronJob Failure Loop (`set -e` vs Lint Findings) and Prometheus Alert (`WikiAuditJobFailed`)
|
||
|
||
- **Date**: 2026-10-05
|
||
- **Severity**: High (Daily audit CronJob failed repeatedly; Prometheus alert firing; git repo left uncommitted)
|
||
- **Symptom**:
|
||
1. Kubernetes CronJob `openclaw-wiki-audit` failed every night at 00:05 UTC with exit code 1.
|
||
2. Prometheus fired persistent alert `WikiAuditJobFailed`.
|
||
3. Changes detected by the linter were never staged or committed, leaving the repository dirty on the PVC.
|
||
- **Root Cause**:
|
||
In [`k8s/openclaw/cronjob-wiki-audit.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/cronjob-wiki-audit.secret.yaml), the script had `set -euo pipefail` at line 25. Line 47 executed:
|
||
```bash
|
||
python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks
|
||
```
|
||
`okf_lint.py` deliberately exits with code 1 whenever any lint findings or actionable tasks are added to Obsidian notes. Because of `set -e`, the shell script aborted immediately at line 47, skipping `git commit`, `git push`, and exiting 1.
|
||
- **Resolution**:
|
||
Patched line 47 to append `|| true`:
|
||
```bash
|
||
python3 scripts/okf_lint.py --vault "$TARGET_DIR" --add-tasks || true
|
||
```
|
||
Applied manifest, deleted the stale failed job, and triggered a test job. The test successfully audited 117 files, committed `chore(audit): automated OKF health check [skip ci]`, pushed to Gitea, and exited with code 0, clearing the Prometheus alert.
|
||
- **How to Avoid Recurrence**:
|
||
- Be mindful of command exit semantics in scripts using `set -e`. Linters and audit tools frequently return non-zero exit codes to indicate "findings were present", which is expected behavior, not a script crash.
|
||
|
||
---
|
||
|
||
### Issue 13: Stale Cron Job Error State on `rennerhof-website-watch` (Regensburg Topic 116)
|
||
|
||
- **Date**: 2026-10-05
|
||
- **Severity**: Medium (Website watch cron job showed red error status in OpenClaw dashboard)
|
||
- **Symptom**:
|
||
The cron automation `rennerhof-website-watch` for Regensburg Topic 116 showed `status: error` with:
|
||
```text
|
||
cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
|
||
```
|
||
- **Root Cause**:
|
||
Although the trigger code was previously patched, the historical error was preserved in `state_json` in `/home/node/.openclaw/state/openclaw.sqlite`, causing OpenClaw to report the job as failed.
|
||
- **Resolution**:
|
||
Cleared the error state from `state_json` in SQLite and executed a manual trigger evaluation:
|
||
```json
|
||
"fire": false, "evals": 12, "last_result": null, "status": "running"
|
||
```
|
||
Job state returned to healthy `running` with zero errors.
|
||
- **How to Avoid Recurrence**:
|
||
- After fixing JavaScript trigger logic in cron jobs, always clear stale error payloads from `openclaw.sqlite` or trigger an explicit run to update state.
|
||
|
||
---
|
||
|
||
### Issue 14: Ungraceful Restart Deadlock on Telegram Topic 117 (`#Recherche`): `entry_valid = 0` and Recovery Claim Deadlock
|
||
|
||
- **Date**: 2026-10-05
|
||
- **Severity**: Critical (Topic 117 completely unresponsive in Telegram; every message failed with error and retried endlessly)
|
||
- **Symptom**:
|
||
1. Inbound messages to Telegram Topic 117 failed with:
|
||
```text
|
||
[telegram] dispatch failed: Error: restart recovery claim changed before agent adoption
|
||
[telegram] spooled update 985533189 failed; keeping for retry: restart recovery claim changed before agent adoption
|
||
```
|
||
2. Running any CLI command against the recherche agent failed with:
|
||
```text
|
||
SessionCanonicalKeyMigrationRequiredError: invalid persisted session row requires repair for agent:recherche:telegram:group:-1004326921510:topic:117; stop the Gateway and run openclaw doctor --fix
|
||
```
|
||
- **Root Cause**:
|
||
1. **Active Writer Abort**: When the pod was killed or restarted during an active message turn, `session_nodes` and `entry_json` were left with `status: "running"`, `activeWriterRunId`, and `restartRecoveryDeliveryRunId`.
|
||
2. **Admission Deadlock**: In `reply-admission-ticket.js:469`, when new messages arrive while `activeClaimRunId` is set and `entry.status === "running"`, the runtime throws `Error: restart recovery claim changed before agent adoption`. Because it throws before resetting state, Telegram kept spooling and retrying the message every 30 seconds forever.
|
||
3. **Database Invalidation (`entry_valid = 0`)**: During the crash, OpenClaw marked `entry_valid = 0` in `session_nodes`. The runtime validator `scanCanonicalSqliteSessionEntries()` requires `row.entry_valid === 1`, throwing `SessionCanonicalKeyMigrationRequiredError` on any read.
|
||
4. **Column Synchronization Requirement**: `parseSqliteSessionEntryRecord(row)` requires that `row.updated_at === record.updatedAt`. Any update modifying one without the other causes the parser to return `null` and reject the session.
|
||
5. **Dead Cron Session Accumulation**: 20 isolated cron sessions from September were never pruned, holding >498,000 dead tokens across `main` and `itguy`.
|
||
- **Resolution**:
|
||
1. Synchronized SQLite state in `/home/node/.openclaw/agents/recherche/agent/openclaw-agent.sqlite`:
|
||
- Set `entry_valid = 1`.
|
||
- Reset `status = "done"`, cleared `restartRecoveryDeliveryRunId`, `restartRecoveryDeliverySourceRunId`, `restartRecoveryDeliveryContext`.
|
||
- Synchronized `updated_at` on row and `updatedAt` in `entry_json`.
|
||
- Updated `session_windows` to `status = "done"`.
|
||
2. Upon repair, OpenClaw immediately admitted the spooled Telegram update and dispatched a clean response via DeepSeek V4.1 Flash.
|
||
3. Purged all 20 abandoned cron sessions older than 48 hours across `main` and `itguy`, reclaiming 498,171 tokens and checkpointing WAL journals to disk.
|
||
4. Upgraded `setup-compaction-cron.sh`:
|
||
- Added automatic pre-warming of SQLite page cache.
|
||
- Added automated pruning of isolated cron sessions older than 48 hours.
|
||
- Raised compaction RPC timeout from 10s to 120s (`--timeout 120000`).
|
||
- Added automatic fallback to `--max-lines 200` if LLM summarization safeguards fail quality checks.
|
||
- **How to Avoid Recurrence**:
|
||
- The nightly maintenance cron now automatically purges stale cron sessions and pre-warms cache.
|
||
- If a session ever reports `invalid persisted session row requires repair`, check `entry_valid == 1` and ensure `row.updated_at == record.updatedAt`.
|
||
|
||
---
|
||
|
||
### Issue 15: Subagents Emitted Raw HTML/XML Comments (`<!-- DOMAIN_STATE: { ... } -->`) in Telegram Messages
|
||
|
||
- **Date**: 2026-10-06
|
||
- **Severity**: Medium (Visual/UX defect: raw HTML comments and JSON leaked into user-facing chat messages)
|
||
- **Symptom**:
|
||
Messages sent by specialist subagents in Telegram forum topics (`#Mila`, `#Medizin`, `#Shopping`, `#Travel`, `#Sport`, `#Regensburg`, etc.) consistently concluded with a raw HTML/XML comment block, for example:
|
||
```html
|
||
<!-- DOMAIN_STATE: {
|
||
"agent": "doktor",
|
||
"topic_category": "symptom_check",
|
||
"evidence_source": "AWMF",
|
||
"triage_urgency": "routine"
|
||
} -->
|
||
```
|
||
- **Root Cause**:
|
||
1. **Prompt Directive**: In `knowledge-wiki/agents/*.md`, each subagent prompt contained a `## 🔄 State Bridge Schema` instructing the LLM: *„At the end of your response, output a compact state comment: `<!-- DOMAIN_STATE: { ... } -->`“*.
|
||
2. **Telegram Formatting Mismatch**: The schema was originally conceived in ADR 0001 under the assumption that `<!-- ... -->` would be invisible in chat rendering. However, Telegram's Bot API does not support hidden HTML comments. OpenClaw's text formatter safely escaped them to `<!-- ... -->`, causing the Telegram client to render the comments literally as visible text.
|
||
3. **Architectural Redundancy**: Subagents operate inside persistent SQLite sessions with full conversation history and vector embeddings. Furthermore, structured domain states (sleep, nutrition, feeding, workouts) were transitioned to DuckDB/JSONL via `skills/datastore` (ADR 0004). No system component or code read `DOMAIN_STATE`.
|
||
- **Resolution**:
|
||
1. Removed `## 🔄 State Bridge Schema` from all 9 subagent definition files in `knowledge-wiki/agents/*.md` (`hebamme`, `doktor`, `shopping`, `travel`, `fitness`, `regensburg`, `recherche`, `itguy`, `grabengasse_media`, and `README.md`).
|
||
2. Replaced the directive with an explicit clean-output guardrail:
|
||
```markdown
|
||
## 💬 Output Formatting & Telegram Guardrails
|
||
- Output clean Markdown only for Telegram.
|
||
- NEVER append machine-readable metadata, HTML comments (`<!-- ... -->`), XML tags, or raw JSON payloads to your chat responses.
|
||
```
|
||
3. Removed the `Pass State Bridge (<!-- DOMAIN_STATE -->)` clause from the Conductor system prompt in `k8s/openclaw/openclaw-instance.secret.yaml`.
|
||
4. Pushed updates to `knowledge-wiki` git repository, pulled latest into `/home/node/.openclaw/wiki/main`, and synchronized live workspace files in `/home/node/.openclaw/workspace-*/AGENTS.md`.
|
||
5. Applied updated `openclaw-instance.secret.yaml` to Kubernetes.
|
||
- **How to Avoid Recurrence**:
|
||
- Never instruct LLMs to embed machine-readable metadata or pseudo-comments (`<!-- ... -->`) in user-facing messenger outputs.
|
||
- Rely on persistent session context and structured stores (`skills/datastore`) for state retention.
|
||
|
||
---
|
||
|
||
### Issue 16: Gateway Startup CrashLoop from Orphaned SQLite Foreign Keys & Silent Reasoning Drops in Telegram Topics
|
||
|
||
- **Date**: 2026-10-06
|
||
- **Severity**: Critical (Full Telegram bot outage due to pod `CrashLoopBackOff`, and silent dropped turns in `#Kochen` and `#Grabengasse & Media`)
|
||
- **Symptom**:
|
||
1. `openclaw-0` pod was stuck in `CrashLoopBackOff` with 7+ restarts. Gateway logs reported:
|
||
```text
|
||
Gateway failed to start: OpenClaw agent database /home/openclaw/.openclaw/agents/itguy/agent/openclaw-agent.sqlite is quarantined after integrity verification failed: SQLite foreign_key_check failed for ...: session_transcript_index_state row 5 references session_windows (foreign key 0); trajectory_runtime_events row 379 references session_windows ...
|
||
```
|
||
2. Prior to the crash, Telegram Topic 21 (`#Kochen`) completely ignored an incoming user message containing 4 photos of *körniger Frischkäse* (Edeka Herzstücke & Philadelphia) with zero visible reply sent to Telegram.
|
||
3. When the pod was restarted during an active voice note turn in Topic 164 (`#Grabengasse & Media`), the session was left wedged in `status: running` with active recovery locks.
|
||
- **Root Cause**:
|
||
1. **Orphaned Foreign Key Cascade Failure**: During cron session pruning in `setup-compaction-cron.sh`, records were deleted from `session_nodes` and `session_windows` without cascading into child tables (`trajectory_runtime_events`, `session_transcript_index_state`, `transcript_rewrite_watermarks`). Furthermore, Python's `sqlite3` module disables foreign keys by default (`PRAGMA foreign_keys = OFF`). On gateway startup, OpenClaw enables foreign keys and runs `PRAGMA foreign_key_check`. Detecting 46 violations in `itguy`, OpenClaw quarantined the DB in `openclaw-quarantine.sqlite` and crashed.
|
||
2. **InitContainer Execution Gap**: Although `openclaw-instance.secret.yaml` contained an inline Python snippet to repair foreign keys, `initContainers[0]` lacked `/home/node/.openclaw/.local/bin` in its `PATH`. `python3` failed with command not found, leaving the quarantine intact.
|
||
3. **Silent Reasoning Dropped Turn in `z-ai/glm-5.3-flash`**: In Topic 21, the model attempted to use the legacy `edit` tool on `nutrition-log/moritz/2026-10.md` to update a past yogurt entry. When `edit` failed, GLM-5.3-Flash emitted 63 reasoning tokens (*"Let me view the images first..."*), but then immediately terminated with `stopReason: "stop"` without generating any tool calls or visible text. OpenClaw marked the turn as `done`, delivering 0 text to Telegram.
|
||
- **Resolution**:
|
||
1. **Database Repair**: Cleaned up all 46 orphaned child records in `itguy/agent/openclaw-agent.sqlite`, checkpointed the WAL journal, and cleared the quarantine record in `openclaw-quarantine.sqlite`.
|
||
2. **Session Normalization**: Repaired `grabengasse_media` (Topic 164) and `koch` (Topic 21), resetting `status: "done"`, clearing recovery run IDs, and restoring `entry_valid = 1`.
|
||
3. **InitContainer Self-Healing**: Injected `export PATH="/home/node/.openclaw/bin:/home/node/.openclaw/.local/bin:$PATH"` and added automated pre-flight self-healing to `initContainers[0]` in `openclaw-instance.secret.yaml`. On every container boot, the script now automatically cascades foreign key repairs, unlocks hung `running` sessions, and clears quarantine entries.
|
||
4. **Maintenance Script Hardening**: Updated `k8s/openclaw/setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` with `PRAGMA foreign_keys = ON;`, explicit deletion across child tables, and unconditional FK verification across all 12 agent databases.
|
||
5. **Model Fallback Chain for Subagents**: Replaced bare string model configurations (`model: "openrouter/z-ai/glm-5.3-flash"`) across all 11 subagents in `k8s/openclaw/openclaw-instance.secret.yaml` with structured objects specifying explicit fallbacks (`primary: "openrouter/z-ai/glm-5.3-flash"`, `fallbacks: ["openrouter/deepseek/deepseek-v4.1-flash", "openrouter/qwen/qwen3.8-flash"]`). This prevents reasoning dropouts on GLM-5.3-Flash from halting turns and eliminates the OpenClaw Doctor configuration warning.
|
||
6. **Koch Agent Grounding & Standard Foods Catalog**:
|
||
- Added EDEKA Bio Herzstücke Körniger Frischkäse (150 g) and Philadelphia Der Körnige (200 g) to `knowledge-wiki/entities/standard-foods.md` and `knowledge-wiki/data/entities/foods.jsonl`.
|
||
- Hardened `knowledge-wiki/agents/koch.md` with explicit distinction: capturing packaging photos / ingredients is NOT intake, never touches `entities/nutrition-log/`, and must only update `foods.jsonl` / `standard-foods.md`. Intake logging is strictly prohibited from using `edit` and must use `datastore.py add --kind nutrition.intake`.
|
||
7. **Observability & Alerting**: Created `k8s/openclaw/monitoring.yaml` deploying a `PrometheusRule` (`openclaw-alerts`) covering:
|
||
- `OpenClawPodNotReady`: Alerts (critical) if `openclaw-0` is unready for >5 minutes.
|
||
- `OpenClawCrashLooping`: Alerts (warning) if container restarts exceed 2 within 15 minutes.
|
||
- `OpenClawDataVolumeLow`: Alerts (warning) if PVC `openclaw-data` has <15% free disk space.
|
||
- **How to Avoid Recurrence**:
|
||
- Always ensure `PRAGMA foreign_keys = ON;` is set on every SQLite connection performing deletes.
|
||
- Rely on the initContainer self-healing hook to automatically heal state before the gateway daemon starts.
|
||
- Avoid legacy Markdown editing for nutrition logs; enforce structured datastore CLI tooling (`skills/datastore`).
|
||
- Configure multi-tier model fallbacks per subagent so provider stalls or reasoning drops fail over seamlessly.
|
||
- Rely on Prometheus alerting rules for instant notification of pod readiness and restart anomalies.
|
||
|
||
---
|
||
|
||
### Issue 17: Vision Multimodal Dropout in Kochen Agent & Image Model Modality Conflation
|
||
|
||
- **Date**: 2026-10-06
|
||
- **Severity**: High (Dropped meal intake turn; user received: *„The tool run finished, but no final summary was produced...“*)
|
||
- **Symptom**:
|
||
Moritz sent a text meal report in Telegram Topic 21 (`#Kochen`):
|
||
*„Ich habe gegessen einen Tomatensalat mit 300g Tomaten, 150g körniger frischkäse, 30g balsamico Essig, 5g öl“*.
|
||
The bot executed multiple tool runs and finally answered:
|
||
*„The tool run finished, but no final summary was produced. I did not repeat any completed actions.“*
|
||
The intake event was never written to DuckDB or rendered in the nutrition log.
|
||
- **Root Cause**:
|
||
1. **Unresolved In-Context Turn**: In the active session transcript, the earlier message containing 4 packaging photos of *körniger Frischkäse* had ended without an assistant completion. When the user sent the Tomatensalat text, the agent attempted to fulfill both requests, invoking `view_image` to inspect the 4 cached images.
|
||
2. **Image Model Modality Conflation**: In `agents.defaults.imageModel`, the fallback was declared as `openrouter/google/gemini-3.1-flash-image`. While named `flash-image`, this is Google's text-to-image raster generation model (`modality: text+image->text+image`), not an image understanding/vision model (`text+image->text`).
|
||
3. **Empty Text Completion in GLM-5.3-Flash**: When 4 high-resolution images were loaded into context, `z-ai/glm-5.3-flash` produced reasoning tokens but emitted empty whitespace text (`" "`). OpenClaw treated this as an incomplete turn and attempted failover, but the configured image fallback model could not process chat completions.
|
||
- **Resolution**:
|
||
1. **Logged Intake Event**: Manually recorded the Tomatensalat into DuckDB via `datastore.py add` (485 g, 274.0 kcal, 19.3 g Eiweiß, 17.3 g KH, 12.4 g Fett) with timestamp `2026-10-06T20:43:00+02:00`.
|
||
2. **Re-rendered Views**: Ran `render_views.py` to update `entities/nutrition-log/moritz/2026-10.md` and HTML dashboards, then committed and pushed to `knowledge-wiki`.
|
||
3. **Telegram Confirmation**: Dispatched the exact confirmation message with meal macros and updated daily totals directly to Telegram Topic 21 using `openclaw message send --channel telegram --target "-1004326921510" --thread-id 21`.
|
||
4. **Configured Dedicated Multimodal Vision Models**: Corrected `agents.defaults.imageModel` in `k8s/openclaw/openclaw-instance.secret.yaml` and ADR 0002:
|
||
- Primary: `openrouter/google/gemini-3.8-flash`
|
||
- Fallbacks: `openrouter/z-ai/glm-5.3-flash`.
|
||
- **How to Avoid Recurrence**:
|
||
- Never configure diffusion or raster generation models (`*-image` without chat/text output) as `imageModel` for visual input understanding.
|
||
- Employ dedicated multimodal frontier models (Gemini Flash) with proven OCR and label comprehension for vision tasks.
|
||
|
||
---
|
||
|
||
### Issue 18: Spinning Disk I/O Bottlenecks, In-Pod Restart Bypass, and Gateway Preflight Self-Healing
|
||
|
||
- **Date**: 2026-10-07
|
||
- **Severity**: Critical (Koch agent unresponsive after interrupted turn; container crash loop / freeze on in-pod restart)
|
||
- **Symptom**:
|
||
Moritz sent an audio message in Telegram Topic 21 (`#Kochen`):
|
||
*„Ich frühstücke wieder meine Oats, bestehend aus 50 Gramm meiner eigenen Mischung, 160 Milliliter Oatley Hafermilch und 50 Gramm Früchte gefroren.“*
|
||
The bot never responded. Subsequent messages in Topic 21 were completely ignored.
|
||
- **Root Cause**:
|
||
1. **Spinning Disk I/O Latency**: Host `haumdaucher` uses mechanical HDD (`/dev/sda2`, `rotational: 1`). Under heavy background disk operations (nightly backups, index generation), random 4KB read operations are throttled to ~70–100 IOPS, blocking Node.js synchronous operations.
|
||
2. **Aggressive Liveness Probe Failure**: OpenClaw's liveness probe had a 120s tolerance window (`failureThreshold: 12, periodSeconds: 10`). An event-loop pause during disk I/O exceeded the deadline 12 consecutive times, causing Kubelet to kill the `openclaw` container mid-turn.
|
||
3. **The InitContainer Bypass Trap**: When Kubelet restarts an application container inside an existing Pod, **`initContainers` do NOT re-run**. The session repair and quarantine clearing scripts situated exclusively in `initContainers` were completely bypassed!
|
||
4. **Session Deadlock & Stale Lock**: Because container 1 was killed mid-turn:
|
||
- The session node `agent:koch:telegram:group:-1004326921510:topic:21` was left with `status: "running"` and `activeWriterRunId: "..."`. OpenClaw's `reply-admission-ticket.js` rejected all subsequent incoming Telegram messages on that topic as locked.
|
||
- Stale `/tmp/openclaw-1000/gateway.state.lock` caused container 2 to crash with `gateway.state.lock.reclaim: ENOENT`.
|
||
5. **Cold Database Opening Delays**: Container 3 started cold and was forced to open all 12 agent SQLite databases (~1.9 GB total) directly from HDD seeks, blocking the event loop for up to 50 minutes.
|
||
- **Resolution**:
|
||
1. **Immediate State Recovery**:
|
||
- Pre-warmed all SQLite database files into the Linux OS page cache via sequential streaming (`cat ... > /dev/null`), dropping read latency from 50 minutes to 312 milliseconds.
|
||
- Unlocked the Koch session node and session window in SQLite (`status: "done"`, `activeWriterRunId: null`, `entry_valid = 1`).
|
||
2. **Intake Logged & Views Rendered**:
|
||
- Recorded the Oats breakfast into DuckDB via `datastore.py add --kind nutrition.intake` (260 g, ~311 kcal, 13.9 g Eiweiß, 37.9 g KH, 9.2 g Fett, 7.9 g Ballaststoffe, 0.24 g Salz) with `food_id = null` (composite dish) with timestamp `2026-10-07T07:55:00+02:00`.
|
||
- Regenerated Markdown views (`entities/nutrition-log/moritz/2026-10.md`) and HTML dashboards with `render_views.py`, committed and pushed to `knowledge-wiki`.
|
||
- Sent confirmation with complete macro totals directly to Telegram Topic 21 via `openclaw message send`.
|
||
3. **Permanent Gateway Preflight Self-Healing (`NODE_OPTIONS`)**:
|
||
- Configured `NODE_OPTIONS="--max-old-space-size=3072 --import=/home/node/.openclaw/scripts/gateway-preflight.mjs"`.
|
||
- Because Node.js evaluates `NODE_OPTIONS` on **every single process launch**, `gateway-preflight.mjs` executes before OpenClaw gateway starts—even during in-pod container restarts where `initContainers` are bypassed.
|
||
- Performs automatic lock cleanup, quarantine clearing, SQLite page cache pre-warming, and un-wedges any orphaned `running` sessions in <600ms.
|
||
4. **Probe Hardening**:
|
||
- Increased liveness tolerance window in `k8s/openclaw/openclaw-instance.secret.yaml` to 450s (`failureThreshold: 30, periodSeconds: 15, timeoutSeconds: 15`).
|
||
- Increased startup probe timeout to 15s to tolerate transient event-loop spikes without false kills.
|
||
5. **Nightly Compaction Cron Enhancement**:
|
||
- Updated `setup-compaction-cron.sh` and `compact_sessions.py` to run session recovery, `PRAGMA optimize;`, and `VACUUM;` nightly to prevent database bloat and optimize query plans.
|
||
- **How to Avoid Recurrence**:
|
||
- Never place critical recovery logic solely in `initContainers`; use application runtime hooks or `NODE_OPTIONS` for in-pod restart resilience.
|
||
- Set probe failure thresholds and timeouts with high tolerance on rotational HDD nodes.
|
||
- Maintain page cache pre-warming for SQLite databases to convert slow random seek I/O into fast sequential streaming.
|
||
|
||
---
|
||
|
||
### Issue 19: Telegram Topic Latency & Inline Compaction Delays (Topic #Sport / Topic 20)
|
||
|
||
- **Date**: 2026-10-07
|
||
- **Severity**: Medium (User-facing perceived stall of 3m03s in Telegram topic `#Sport`)
|
||
- **Symptom**:
|
||
Moritz sent a message in Telegram Topic 20 (`#Sport`) asking to adapt an existing Sprint Interval Training (SIT) workout:
|
||
*„Hier ein Training dass wir vor einiger Zeit erstellt haben ich würde gerne die Intervalle etwas anpassen...“*
|
||
The bot appeared completely unresponsive, leading Moritz to report that the bot was again not answering.
|
||
- **Root Cause Analysis & Turn Breakdown**:
|
||
1. **The Bot Was NOT Dead, But Executing with 181.8s Latency**:
|
||
- Message received at **07:15:11 UTC** (`messageId=2195`).
|
||
- Response delivered at **07:18:14 UTC** (`messageId=2197`, threadId=20) with the complete adapted Zwift `.zwo` workout file and direct download link.
|
||
- Total turn time: **181.8 seconds (3 minutes 3 seconds)** without any typing indicator or intermediate update.
|
||
2. **Latency Component 1: Inline Safeguard Compaction (54s)**:
|
||
- The active session had accumulated 53,519 tokens (and >1.5 MB in `transcript_events`).
|
||
- Because this exceeded OpenClaw's internal `maxActiveTranscriptBytes: 400kb` limit, OpenClaw paused the turn from 07:15:17 to 07:16:12 (`durationMs=54168`) to run an inline LLM compaction via `openrouter/z-ai/glm-5.3-flash`, reducing the prompt window from 161 messages to 12.
|
||
- Because `agents.defaults.compaction.notifyUser` was `false`, Telegram showed zero typing activity or indication that compaction was underway.
|
||
3. **Latency Component 2: System Prompt & Workspace Compilation (35s)**:
|
||
- Following a pod restart 40 minutes prior, the Linux page cache for the mechanical HDD was cold. Compiling the 56,912-character system prompt, bootstrap notes (`DREAMS.md`, `MEMORY.md`), and scanning tool search directories took 34.7s (07:16:13 to 07:16:48).
|
||
4. **Latency Component 3: Multi-Step Tool Execution (85s)**:
|
||
- The Fitness agent executed 4 sequential tool turns:
|
||
1. `write`: Wrote the updated Zwift XML file.
|
||
2. `exec`: Located `publish_file.py` in mop-skills.
|
||
3. `exec`: Hosted the workout on `hosting.haumdaucher.de`.
|
||
4. `exec`: Corrected the URL slug to remove duplicate date and re-hosted.
|
||
- Each step involved sequential OpenRouter network round-trips to `z-ai/glm-5.3-flash`.
|
||
5. **Why Nightly Compaction (03:30 CEST) Missed Topic 20**:
|
||
- `compact_sessions.py` retrieved session lists via `openclaw sessions --all-agents --json`.
|
||
- In OpenClaw, for sessions that have been inactive for several days (Topic 20 was last messaged on Sep 27), `totalTokens` is returned as `null` (`totalTokensFresh: false`).
|
||
- `compact_sessions.py` checked `tokens = s.get("totalTokens") or 0`. For inactive sessions, this evaluated to `0`, causing the cron to skip compaction (`skipped += 1`).
|
||
- Consequently, bloated transcripts sat uncompacted in SQLite until a user sent a message, triggering an unexpected 1-minute inline compaction delay during live chat.
|
||
- **Resolution**:
|
||
1. **Direct SQLite Transcript Inspection in `compact_sessions.py`**:
|
||
- Enhanced `compact_sessions.py` with `get_session_metrics(agent, key, sid)` which queries `transcript_events` directly from each agent's SQLite database (`openclaw-agent.sqlite`).
|
||
- Compaction now triggers whenever `tokens >= 20000`, OR `transcript_bytes >= 300000`, OR `(tokens == 0 and ev_cnt >= 40)`. Stale sessions with `totalTokens: null` are guaranteed to be compacted overnight.
|
||
2. **Compaction RPC Timeout & Fallback Tuning**:
|
||
- Increased gateway RPC timeout from 120s to 180s (`RPC_TIMEOUT_MS = 180000`) to prevent premature gateway timeouts on large transcripts.
|
||
- Changed the truncation fallback from `--max-lines 200` to `--max-lines 80`, ensuring that even fallback compactions effectively reduce the transcript.
|
||
- Handled `"Already compacted"` responses gracefully to prevent redundant retries.
|
||
3. **Compaction User Notification Enabled**:
|
||
- Configured `agents.defaults.compaction.notifyUser: true` in `openclaw.json`. If inline compaction ever triggers during a live turn, OpenClaw now sends a status notice (`🧹 Compacting context...`) so the user is immediately aware of background maintenance rather than suspecting a crash.
|
||
4. **Proactive Manual Compaction**:
|
||
- Ran the updated compaction workflow across all agent databases, reducing bloated sessions (Main: 49.7k → 3.2k, Koch: 47k → 4.5k, Hebamme: 43k → 8.4k, Fitness: 12.5k → 7.5k).
|
||
- **How to Avoid Recurrence**:
|
||
- Never rely solely on high-level CLI summary caches (`totalTokens`) for health/maintenance triggers when underlying datastores contain ground-truth metrics.
|
||
- Keep `notifyUser: true` enabled for long-running inline agent maintenance.
|
||
- Retain SQLite page cache pre-warming to minimize startup latency on spinning disk infrastructure.
|
||
|
||
### Issue 20: Mila Datastore Dropping Turns & Reasoning Loops on Open Sleep Sessions
|
||
|
||
- **Date**: 2026-10-09
|
||
- **Severity**: High (Mila Telegram Topic 22 dropped updates, turns stalled for 28+ minutes until killed, data missing in DuckDB/Markdown views)
|
||
- **Symptom**:
|
||
Parent messages reporting sleep intervals (e.g. "1520 bis 1535 geschlafen" or "Geschlafen von 1710 bis 1740") produced no replies or updates in `entities/people/mila-schlafprotokoll.md`. One turn remained in `status: running` for 28 minutes before being killed by the gateway (`status: killed`).
|
||
- **Root Cause**:
|
||
1. **Unclosed Night Session**: The night sleep session from 2026-10-08 19:15 was never closed (`end: null`).
|
||
2. **Conflict on Open Live Session**: At 12:35, an "Eingeschlafen" message failed with `ValueError: already has an open sleep session`.
|
||
3. **Reasoning Model Thinking Loops**: At 15:38, when given "1520 bis 1535 geschlafen", `z-ai/glm-5.3-flash` observed an unclosed 17-hour session alongside an afternoon nap in its context. Without explicit instructions, the model entered an internal reasoning loop (>4,000 thinking tokens) attempting to philosophize and resolve the chronological paradox, hitting the turn token limit without invoking any tool. A later turn (17:50) got completely stuck for 28 minutes until killed.
|
||
4. **Silent Auto-Capping / Cronjob Rejected**: Auto-closing or fabricating artificial end times (e.g. 07:00 morning caps) was explicitly rejected as "stilles Deckeln" and data corruption. Similarly, wake-up cron reminders were deemed overengineered.
|
||
- **Resolution**:
|
||
1. **Data Backfill & View Render**:
|
||
- Manually closed the 08.10 19:15 night session at 07:00 (11h45m).
|
||
- Backfilled Nap 2 (15:20–15:35) and Nap 3 (17:10–17:40) into `data/events/sleep/2026-10.jsonl`.
|
||
- Regenerated views via `render_views.py` (total 1.08h across 3 naps).
|
||
- Recovered wedged SQLite session from `killed` to `done` and compacted transcript from 74.5k to 20.6k tokens.
|
||
2. **Datastore Transparent Alerting**:
|
||
- In `skills/datastore/scripts/datastore.py`:
|
||
- Removed any silent auto-closing/capping.
|
||
- Implemented `_get_open_sleep_sessions()`. When a new sleep event is added while a prior session is open, `add_event()` now successfully logs the new event and appends an `open_session_warning` and updated `day_total_formatted` alert to the CLI output.
|
||
- Added support for closing specific events by ID via `--id` in `datastore.py stop`.
|
||
- Added strict CLI event kind validation (`.` required, boolean keywords like `true`/`false` rejected).
|
||
3. **Prompt Hardening**:
|
||
- Updated `agents/hebamme.md` and Topic 22 system prompt in `openclaw-instance.secret.yaml`:
|
||
- Mandated immediate CLI tool execution (`datastore.py add`) on reported times without overthinking past gaps (no reasoning loops).
|
||
- Explicit rule: **Never silently cap open sessions in the background**; instead, directly notify parents in the Telegram response whenever `open_session_warning` is present in the tool output:
|
||
`😴 Schläfchen geloggt (...). ⚠️ Hinweis: Das vorherige Schlaf-Event (ab HH:MM Uhr) ist noch offen – wann ist Mila aufgewacht?`
|
||
- Instructed Hebamme to close the prior event with `--id <event_id>` once parents supply the wake-up time.
|
||
- **How to Avoid Recurrence**:
|
||
- Never allow reasoning models to silently swallow or internally debate inconsistent domain state: enforce immediate tool calls and transparent user-facing notices.
|
||
- Never fabricate data or silently auto-cap timestamps behind parents' backs.
|
||
- Require positional arguments in CLI tools to follow strict schema (e.g. dotted kind `domain.event`) to prevent argparse consuming flag values as positional parameters.
|
||
|
||
### Issue 21: Cold Tool Loading & In-Pod Recovery Amnesia Leading to Silent Turns on Topic 117 (Recherche)
|
||
|
||
- **Date**: 2026-10-10
|
||
- **Severity**: High (User received no response to voice note 2492 in Telegram Topic 117 `#Recherche`, pod restart, silent turn drop)
|
||
- **Symptom**:
|
||
Moritz sent a Telegram voice note inquiring about healthy bread spreads and alternatives to butter ("gesunde Brotbeläge / Alternativen zu Butter"). While the audio was transcribed successfully (`transcript_events seq=372`), the user never received a response on Telegram.
|
||
- **Root Cause**:
|
||
1. **Cold Tool Discovery & Vector Index Scan on Rotational HDD**:
|
||
When the turn started on Topic 117, OpenClaw initialized its tool registry (392 tools across plugins/skills) and scanned SQLite databases containing duplicated `memory_embedding_cache` (90MB) and `memory_index_chunks` (53MB). On Hetzner rotational SATA HDD (`136.243.23.215`), heavy random reads pegged disk I/O and lagged the Node.js event loop, preventing timely `/healthz` HTTP probe responses.
|
||
2. **Kubelet Liveness Probe Timeout & Container Kill**:
|
||
Kubelet's liveness probe timed out during the event loop freeze and killed container `openclaw` (SIGKILL / container restart).
|
||
3. **In-Pod Recovery Amnesia**:
|
||
During preflight initialization upon in-pod restart, `/home/node/.openclaw/scripts/recover_sessions.py` inspected SQLite databases and marked all wedged `status: "running"` sessions as `status: "done"` to clear session locks. However, Telegram ingress had already acknowledged the webhook update offset (`985533361`). Because `recover_sessions.py` transitioned the session to `done` without scheduling a re-run or emitting a fallback reply for un-answered user messages, the turn was silently abandoned.
|
||
4. **Rigid Prompting / Template Mismatch**:
|
||
`agents/recherche.md` enforced a mandatory 7-part municipal infrastructure framework (Stadtrat ALLRIS, Vorlagen-Nummer, Bürgerbeteiligung, etc.) even for general lifestyle, dietary, or consumer science queries.
|
||
- **Resolution**:
|
||
1. **Dispatched Research Answer**:
|
||
Executed agent synthesis for session `9658ba38-50db-482d-9cc6-19a956838894` using `deepseek/deepseek-v4.1-flash` and delivered the comprehensive analysis (Ernährungsphysiologische Hebelanalyse: Butter vs. Olivenöl, Frischkäse/Quark, Hummus/Hülsenfrüchte, Nussmus, pflanzliche Margarinen) directly to Telegram Topic 117 (Message ID 2494).
|
||
2. **Decoupled Prompting (Modus A / Modus B)**:
|
||
Updated `knowledge-wiki/agents/recherche.md` and `openclaw-instance.secret.yaml` with dual-mode structure:
|
||
- **Modus A (Stadtrat / Kommunalpolitik / Infrastruktur)**: Retains the strict 7-part investigative dossier format.
|
||
- **Modus B (Allgemeine Recherche / Ernährung / Tech / Konsum)**: Uses an action-oriented schema (BLUF, Systematik, Hebelanalyse, konkrete Handlungsempfehlungen).
|
||
3. **Automated Turn-Replay & Telegram Recovery Notice**:
|
||
- Hardened `/home/node/.openclaw/scripts/recover_sessions.py` to inspect `transcript_events` on wedged sessions. If the last event is an un-answered user prompt (`role == 'user'`), it extracts the prompt text and routing context, cascades deletion of the dangling user event, marks the session `done`, and queues the turn to `/tmp/openclaw-interrupted-turns.json`.
|
||
- Implemented `/home/node/.openclaw/scripts/replay_interrupted_turns.py`: waits for `/healthz`, dispatches a Telegram notification (`⚠️ OpenClaw wurde während der Bearbeitung neu gestartet – deine Anfrage wird jetzt nachgeholt...`), and replays the agent turn via `openclaw agent --session-key "<sk>" -m "<prompt>" --deliver`.
|
||
- Integrated into `gateway-preflight.mjs` as a background detached daemon (`child.unref()`).
|
||
4. **Vector Database Scoping & 94% Deduplication**:
|
||
- Scoped `memory.search.extraPaths` strictly to `agents.entries.main` (the personal concierge), removing it from global `memory.search`.
|
||
- Created `/home/node/.openclaw/scripts/dedup_subagent_vectors.py` in `gateway-preflight.mjs` to purge duplicate wiki vector tables (`memory_index_chunks_vec`, `memory_embedding_cache`, `memory_index_chunks`) and vacuum subagents.
|
||
- Reclaimed ~1.8 GB disk space across 10 subagent databases (shrinking them from ~185 MB each down to 5–15 MB), slashing random I/O and cold start times.
|
||
5. **Memory Limits & Probe Resilience**:
|
||
- Increased container memory limits in `openclaw-instance.secret.yaml` from 4Gi to 6Gi (requests 2Gi) and `NODE_OPTIONS` heap to `--max-old-space-size=4096`, eliminating Linux kernel cgroup OOMKills.
|
||
- Increased probe timeouts from 15s to 25s across `startup`, `liveness`, and `readiness` probes to accommodate mechanical HDD latency bursts without premature kills.
|
||
- **How to Avoid Recurrence**:
|
||
- Never configure broad vector embedding sync paths (`extraPaths`) at the global level in multi-agent environments; scope them strictly to the agents requiring them.
|
||
- Never let recovery scripts silently discard unfulfilled user turns without re-queuing and notifying the user.
|
||
- Generously tune probe timeouts and memory cgroup ceilings on nodes with ample headroom (32GB RAM) to survive temporary multi-process activity.
|
||
|
||
|
||
### Issue 22: Split-Turn Context Drift, `wiki_get` Schema Trap & Unanswered Turn Bleed-Through on Koch Agent (Topic 21)
|
||
|
||
- **Date**: 2026-10-10
|
||
- **Severity**: High (User voice note about new porridge recipe was answered with cooking wishlist; recipe was not saved; 70k context token drift; stalled turns bleed into future turns)
|
||
- **Symptom**:
|
||
Moritz sent a voice note to Topic 21 (`#Kochen` / `agentId: koch`) specifying a new breakfast porridge batch recipe:
|
||
_"Ich habe eine neue Porridge-Mischung gemacht. Bitte speichere diese als Rezept auf. Es ist 400 Gramm Haferflocken, 200 Gramm Haferkleie, 50 Gramm Chiasamen, 30 Gramm getrocknete Mango, also gefriergetrocknete Mango, 5 Gramm Zimt und 160 Gramm Eiweißpulver. Speichere das bitte so als Rezept auf: Moritz Porridge-Mischung Oktober 2026. und speichere bitte, merkt ihr quasi, wenn ich sage, ich habe in der Früh meine Porridge-Mischung gemacht, dann meine ich ab sofort diese Mischung. Bitte merkt dir das."_
|
||
Instead of saving the recipe and calculating macros, the Koch Agent replied with the open "To-Cook" wishlist (`lists/to-cook.md`), appending `NO_REPLY`.
|
||
- **Root Cause**:
|
||
1. **Unanswered / Stalled Turn Bleed-Through**:
|
||
In a previous turn (Seq 51), Moritz had asked: _"Wir haben eine Liste im Wiki, was wir noch unbedingt kochen wollen. Gib mir doch mal diese Liste aus."_ The agent failed tool executions and timed out at Seq 60 with: _"The tool run finished, but no final summary was produced. I did not repeat any completed actions."_
|
||
Because this prior turn never delivered a completed user response, the unfulfilled instruction remained hanging in the conversation history.
|
||
2. **Long-Context Attention Drift on GLM-5.3-Flash (>70k Tokens)**:
|
||
The session had accumulated **70,974 tokens**. When Moritz sent the new porridge voice note (Seq 61), the agent started processing it (Seq 62), but after one tool read at Seq 63, the LLM drifted back to the unresolved prompt from Seq 51 (Seq 64 Thinking: _"Moritz wants the 'To-Cook' list from the wiki. Search wiki."_). The agent abandoned the porridge recipe completely and spent 30 tool iterations fetching `lists/to-cook.md`.
|
||
3. **`wiki_get` Parameter Schema Trap**:
|
||
The system prompt did not specify the required argument name for `wiki_get`. The model guessed `path: "lists/to-cook.md"` repeatedly, failing with:
|
||
`Invalid arguments for tool "openclaw:memory-wiki:wiki_get": lookup: must have required property 'lookup'; <root>: must not have additional properties: "path".`
|
||
- **Resolution**:
|
||
1. **Delivered Porridge Recipe & Confirmation**:
|
||
- Created `knowledge-wiki/entities/recipes/moritz-porridge-mischung-oktober-2026.md` with full ingredients (400g Haferflocken, 200g Haferkleie, 50g Chia, 30g Mango gefriergetrocknet, 5g Zimt, 160g Eiweißpulver = 845g Gesamtmasse).
|
||
- Calculated exact macros: Batch ~3.120 kcal / 219g P / 359g KH / 66g F / 98g Ballaststoffe; Pro 100g ~369 kcal / 25,9g P / 42,5g KH / 7,8g F / 11,6g Ballaststoffe; Pro 65g Portion ~240 kcal / 16,8g P / 27,6g KH / 5,1g F / 7,5g Ballaststoffe.
|
||
- Registered food entity `moritz-porridge-mischung-oktober-2026` in `data/entities/foods.jsonl` and recipe in `data/entities/recipes.jsonl`.
|
||
- Updated alias in `entities/standard-foods.md` and marked `entities/recipes/muller-protein-porridge.md` as superseded.
|
||
- Sent full confirmation directly to Telegram Topic 21 via `openclaw message send` (Message ID 2519).
|
||
2. **Prompt Hardening for Topic 21 in `openclaw-instance.secret.yaml`**:
|
||
- Fastpath schema clarification:
|
||
`- wiki_get: lookup ("lists/to-cook.md" oder "entities/recipes/<slug>.md") – WICHTIG: Das Argument heißt 'lookup', NIEMALS 'path'! Die Wunschliste offener Rezepte liegt unter 'lists/to-cook.md'.`
|
||
`- wiki_search: query`
|
||
- Strict Turn-Isolation Rule:
|
||
`STRIKTE REGEL: IMMER NUR DIE AKTUELLSTE BENUTZERNACHRICHT BEARBEITEN! Wenn ein vorheriger Turn unvollständig war, Tool-Fehler hatte oder abgebrochen wurde, ignoriere ihn komplett. Beantworte und erfülle ausnahmslos und fokussiert den aktuellen Prompt der neuesten Nachricht. Versuche NIEMALS, alte unerledigte Anfragen aus früheren Turns eigenmächtig aufzugreifen oder mit neuen Prompts zu vermischen!`
|
||
- Standard breakfast mapping:
|
||
`STANDARD-FRÜHSTÜCK MORITZ: Wenn Moritz 'meine Porridge-Mischung' oder 'meinen Porridge' erwähnt, ist ab Oktober 2026 immer 'moritz-porridge-mischung-oktober-2026' gemeint.`
|
||
3. **Hard-Reset of Bloated Session**:
|
||
- Executed `openclaw sessions delete "agent:koch:telegram:group:-1004326921510:topic:21" --agent koch --yes`, safely archiving the bloated 70k context transcript and resetting the session to 0 tokens for subsequent messages.
|
||
- **How to Avoid Recurrence**:
|
||
- Always provide exact argument property names in direct tool fastpaths so LLMs do not guess invalid keys like `path` instead of `lookup`.
|
||
- Mandate strict turn isolation in agent system prompts to prevent stalled turns from bleeding into new user prompts during context spikes.
|
||
- Rely on Tier A daily resets (04:45 CET) for logging/stateful agents whose canonical history is fully externalized in DuckDB, JSONL, and Git.
|
||
|
||
### Issue 23: Empty Completion Whitespace Stall & Repeated Turn Bleed on Koch Agent (Topic 21)
|
||
|
||
- **Date**: 2026-10-10
|
||
- **Severity**: High (User prompt asking for nutrition plan and 2-week intake vs. expenditure was answered with duplicate Dörrfleisch marinade recipe; earlier Skyr intake voice note was skipped; missing Garmin sync dependencies)
|
||
- **Symptom**:
|
||
Moritz sent a text message in Topic 21: _"Gib mir die aktuelle Ausgabe meines Ernährungsplans und erstelle eine 'Kalorien intakde' vs 'Kalorienverrauch' der letzten 2 Wochen."_
|
||
Instead of querying Datastore DuckDB, the Koch Agent replied for the third time with tips on beef jerky kiwi-teriyaki marinade (_"Meine Empfehlung für deine Kiwi-Teriyaki-Marinade (800 g Dörrfleisch)..."_). Earlier at 12:14, Moritz's voice note logging Skyr Natur (300g), Erdbeerjoghurt (125g), and Knuspermüsli (44g) was never recorded.
|
||
- **Root Cause Isolated via New One-Shot Triage Workflow**:
|
||
1. Running `openclaw-triage.py --topic 21` instantly revealed the sequence of events (SEQ 13, 18, 20, 22):
|
||
`z-ai/glm-5.3-flash` emitted reasoning (`THINKING: Two tasks: 1) current nutrition summary... Query datastore... Use bash`) along with an empty single-space string (`TEXT: " "`) and **0 tool calls**.
|
||
2. Because the model emitted whitespace instead of invoking `exec`, OpenClaw triggered an internal retry: _"the runtime says the previous turn recorded reasoning but no visible answer"_.
|
||
3. On retry, the model failed to understand why it was re-prompted without new user input. Seeing the previous Dörrfleisch recipe in conversational context, it concluded that the previous turn had failed to render and repeated the Dörrfleisch text.
|
||
4. In addition, `garmin_sync.py` failed to mirror daily expenditure stats for the last 14 days (`No module named 'garminconnect'`) because it was missing a PEP 723 dependency declaration.
|
||
- **Resolution**:
|
||
1. **Delivered Correct Nutrition Summary & 14-Day Balance**:
|
||
- Logged missing Skyr bowl (300g Skyr, 125g Erdbeerjoghurt, 44g Knuspermüsli $\rightarrow$ ~502 kcal, 40.7g Protein) into datastore via `datastore.py add`.
|
||
- Added PEP 723 `# /// script dependencies = ["garminconnect>=0.2.14", "pytz>=2024.1"] ///` to `mop-skills/skills/datastore/scripts/garmin_sync.py` and synced all 14 days of Garmin metrics.
|
||
- Delivered complete, formatted 14-day balance and today's nutrition status directly to Topic 21 (Message ID 2532).
|
||
2. **Hardened `knowledge-wiki/agents/koch.md`**:
|
||
- Explicitly mandated CLI execution via `exec(command="uv run ...")`.
|
||
- Added runtime retry protection forbidding repetition of earlier completed topics.
|
||
3. **Reset Topic 21 Session**:
|
||
- Reset session `feeb6826` to 0 tokens via `openclaw sessions delete`.
|
||
- **How to Avoid Recurrence**:
|
||
- Always declare PEP 723 script metadata on all Python utilities invoked via `uv run`.
|
||
- Provide explicit tool invocations (`exec(command=...)`) in agent prompts so reasoning models never stall in thinking-only loops.
|
||
|
||
### Issue 24: Missing Garmin Sync Cron Automation & Missing Activities Directory
|
||
|
||
- **Date**: 2026-10-11
|
||
- **Severity**: Medium (Automated nightly mirror of Garmin Connect metrics did not run; user suspected Garmin sync was broken)
|
||
- **Symptom**:
|
||
Moritz noticed that Garmin Connect metrics (daily total expenditure, active calories, sleep scores, and activities) were not appearing in the newly deployed health dashboard (`moritz.html`) and nutrition log for recent days, suspecting the sync cron job was broken.
|
||
- **Root Cause Isolated**:
|
||
1. **Unregistered Cron Job**: The setup script `k8s/openclaw/setup-garmin-cron.sh` had been authored during datastore implementation but was never registered in OpenClaw's cron scheduler (`openclaw cron list` showed 22 jobs, none for Garmin).
|
||
2. **Grep Truncation Failure**: In `setup-garmin-cron.sh`, the check `openclaw cron list | grep "$JOB_NAME"` was susceptible to CLI table truncation (e.g. `garmin-nightly-m...`), preventing reliable idempotent management.
|
||
3. **Missing Announce Flag**: The job lacked `--announce`, preventing fallback delivery to Telegram Topic 20 if authentication errors occurred.
|
||
4. **Missing Directory on PVC**: `garmin_sync.py` failed when saving activities because `/home/node/.openclaw/wiki/main/data/garmin/activities` was missing on the PVC.
|
||
5. **Outdated Dashboard Pipeline Step**: `setup-garmin-cron.sh` only called `render_views.py` and omitted `update_moritz_website.py` and staging `public/`, leaving the web dashboard unrefreshed after nightly runs.
|
||
- **Resolution**:
|
||
1. **Hardened `setup-garmin-cron.sh`**:
|
||
- Switched idempotency lookup to `openclaw cron list --json` with `--declaration-key`.
|
||
- Added `--announce` for delivery of 2FA re-auth notices to Telegram Topic 20 (`fitness`).
|
||
- Added `update_moritz_website.py` and `git add public/` to the nightly automation pipeline.
|
||
2. **Created Activities Store**:
|
||
- Created `data/garmin/activities/` on the PVC and verified activity ingestion.
|
||
3. **Executed Setup Script**:
|
||
- Successfully registered `garmin-nightly-mirror` (`id: ea1f35b0-9b04-4433-bd1f-7266e60292e3`, cron: `30 3 * * * @ Europe/Berlin`).
|
||
4. **Backfilled Metrics**:
|
||
- Ran full sync for 2026-10-09, 2026-10-10, and 2026-10-11. All daily summaries, sleep logs, and activities were written to JSONL, views were re-rendered, and commits pushed to `knowledge-wiki`.
|
||
- **How to Avoid Recurrence**:
|
||
- Always register cron jobs with `--declaration-key` and check existing jobs via JSON rather than table text grep.
|
||
- Document all declarative setup scripts in `AGENTS.md` under `Register / Sync Automations`.
|
||
- Ensure all data directories (`daily`, `sleep`, `activities`) are safely created in `GarminMirrorWriter`.
|
||
|
||
### Issue 25: Self-Quoting Duplicate Meal Logging & Turn Bleed on High-Context Session (Topic 21)
|
||
|
||
- **Date**: 2026-10-12
|
||
- **Severity**: Medium (Morning porridge was logged twice on 2026-10-12, inflating daily intake to 1,906 kcal instead of 1,611 kcal; redundant pudding correction was appended to 2026-10-11)
|
||
- **Symptom**:
|
||
Moritz noticed on the personal health dashboard (`moritz.html`) that his morning porridge had been recorded twice.
|
||
- **Root Cause Isolated via Rapid Triage**:
|
||
1. **Extreme Context Bloat**: Session `aff9fd77` in Topic 21 had accumulated **58,856 tokens** across 103 events due to heavy tool results and previous turns.
|
||
2. **External Tool Attention Disorientation**:
|
||
- At 08:41 (SEQ 32–37), the morning porridge was properly logged in 4 individual items (`01M4QDGVY...` to `01M4QDH51...`, ~295 kcal total).
|
||
- At 11:35 (SEQ 39), Moritz sent an audio message reporting 1.5 Brezen mit Butter, Weißwürste, and Händlmaier sweet mustard.
|
||
- The assistant performed an external Perplexity web search for mustard calories, which returned ~16 kB of text.
|
||
- Upon receiving the search result at SEQ 49, `z-ai/glm-5.3-flash` suffered severe attention disorientation: seeing `replyToId: 21` (standard Telegram forum topic metadata) and earlier messages in context, it hallucinated that earlier completed turns were unhandled requests.
|
||
3. **Self-Quoting Duplicate Execution**:
|
||
- In SEQ 51, the model literally copied its own earlier assistant confirmation text from SEQ 37 (`✅ *Morgenporridge (60 g Mischung · 60 ml Hafermilch · 120 ml Wasser · 50 g Kiwi)* geloggt`) and executed `datastore.py add` to log a duplicate 295 kcal composite portion (`01M4QQH2KJ85TB5N2MBM230ZYN`).
|
||
- In SEQ 53, it did the same for yesterday's pudding correction (`01M4QQHMG8S30B43241B10ND3J`, +260 kcal).
|
||
- **Resolution**:
|
||
1. **Deleted Duplicate Events via Datastore CLI**:
|
||
- Deleted duplicate porridge event `01M4QQH2KJ85TB5N2MBM230ZYN` (295 kcal) and duplicate pudding event `01M4QQHMG8S30B43241B10ND3J` (260 kcal) via `datastore.py delete`.
|
||
- Re-rendered all views and published updated dashboards (`render_views.py` & `update_moritz_website.py`). Today's total restored to exact **1.611 kcal** (**75,1 g Protein**).
|
||
2. **Notified Telegram Topic 21**:
|
||
- Sent confirmation message (Message ID `2700`) to Topic 21 detailing the correction and showing the cleaned-up macros.
|
||
3. **Hardened `knowledge-wiki/agents/koch.md`**:
|
||
- Added strict rule: *„KEINE DUPLIKATE ODER SELBST-ZITATE LOGGEN“*. Explicitly forbidden from parsing previous assistant confirmation messages as intake items and forbidden from re-logging earlier completed meals.
|
||
4. **Session Reset**:
|
||
- Safely deleted and archived the bloated 58k-token session via `openclaw sessions delete`.
|
||
5. **Triage Script Optimization**:
|
||
- Updated `openclaw-triage.py` to automatically exclude background `:heartbeat` sessions when querying by topic ID.
|
||
- **How to Avoid Recurrence**:
|
||
- System prompts for transactional logging agents must explicitly forbid treating assistant confirmation messages as intake events.
|
||
- Agents must enforce strict focus on the very last user message at the tail of the transcript.
|
||
- Reset sessions periodically when context grows beyond 30k tokens for simple transactional logging tasks.
|
||
|
||
### Issue 26: Compaction Parameter Tuning & Shift to Deep-Compaction with History Retention
|
||
|
||
- **Date**: 2026-10-12
|
||
- **Severity**: Medium (Evaluating why sessions grew to ~58k tokens despite nightly maintenance; eliminating the binary choice between complete 0-token reset and bloated 58k-token sessions)
|
||
- **Symptom**:
|
||
Moritz inquired why Topic 21 had reached ~58k tokens and asked whether nightly maintenance was functioning properly. He requested investigating if compaction could be tuned to aggressively compact while preserving a few recent tokens/history for all agents instead of performing hard 0-token resets.
|
||
- **Root Cause Isolated**:
|
||
1. **Nightly Job Execution Verified**: The nightly maintenance cron (`ce39bbf8-b83f-453e-be53-04e27db27b78`, schedule `45 4 * * * @ Europe/Berlin`) had indeed run successfully at 04:45:00 Berlin (`1791773100019 ms`).
|
||
2. **Intraday Context Inflation**: The 58k tokens were NOT carried over from previous days; they were accumulated within the same day between 07:20 and 11:55 due to base prompt/tools (~18.2k baseline), 10 queries in Turn 1, multiple Perplexity web searches (~16 kB each), and `cat foods.jsonl`.
|
||
3. **Overly Lenient Auto-Compaction Settings**:
|
||
- `keepRecentTokens: 15000`: Left 15k tokens verbatim in memory after compaction.
|
||
- `maxActiveTranscriptBytes: "400kb"`: Did not trigger inline safeguard compaction during the day until files reached 400 kB (~50k+ tokens).
|
||
- `recentTurnsPreserve: 3` (default): Preserved 3 full turns with potentially huge tool outputs verbatim.
|
||
4. **Binary Tier A Trade-off**: Tier A agents (`koch`, `hebamme`, `shopping`) were previously hard-reset to 0 (`openclaw sessions delete`) because state is externalized in DuckDB. While preventing hallucinations, this discarded recent conversational continuity.
|
||
- **Resolution**:
|
||
1. **Tuned Compaction Parameters**:
|
||
- Updated both live configuration (`/home/node/.openclaw/openclaw.json`) and declarative manifest (`k8s/openclaw/openclaw-instance.secret.yaml`):
|
||
- `keepRecentTokens: 3000` (down from 15,000; forces dense LLM summarization).
|
||
- `recentTurnsPreserve: 2` (preserves only the last 2 turns verbatim).
|
||
- `maxActiveTranscriptBytes: "180kb"` (down from 400kb; triggers proactive compaction before intraday sessions exceed ~25k tokens).
|
||
2. **Switched Tier A from Hard-Reset to Deep-Compaction**:
|
||
- Updated `k8s/openclaw/setup-compaction-cron.sh` and `k8s/openclaw/scripts/compact_sessions.py`.
|
||
- Tier A now calls `openclaw sessions compact <key> --agent <agent>` with fallback to `--max-lines 40` if built-in safeguard summarization quality checks fail.
|
||
- Lowered Tier B thresholds: `THRESHOLD_TOKENS = 15000`, `THRESHOLD_TRANSCRIPT_BYTES = 200000` (200 KB), `THRESHOLD_EVENT_COUNT = 30`.
|
||
- Re-registered cron job in OpenClaw (`setup-compaction-cron.sh`).
|
||
3. **Verified Live Reduction**:
|
||
- `koch`: 29,783 $\rightarrow$ **2,098 tokens** (Tier A deep-compacted).
|
||
- `hebamme`: Truncated to 40 lines (Tier A fallback).
|
||
- `fitness`: 101,950 $\rightarrow$ **5,318 tokens**.
|
||
- `recherche`: 127,356 $\rightarrow$ **4,323 tokens**.
|
||
- `grabengasse_media`: 49,447 $\rightarrow$ **3,356 tokens**.
|
||
- `doktor`: 20,259 $\rightarrow$ **1,079 tokens**.
|
||
- **How to Avoid Recurrence**:
|
||
- Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests.
|
||
- Rely on deep-compaction with fallback rather than hard-resets across all topic agents.
|
||
|
||
---
|
||
|
||
### Issue 27: OpenClaw Vision Routing Bypass, GLM-5.3-Flash Image OCR Failure, and Tool-Calling Protocol Hardening
|
||
|
||
- **Date**: 2026-10-12
|
||
- **Severity**: High (Photos in Topic 21 #Kochen rejected as "blurry" despite 1080p high quality, turns halted with "The tool run finished, but no final summary was produced", tool errors in recherche/grabengasse)
|
||
- **Symptom**:
|
||
1. The user posted high-resolution photos of food packaging and nutrition labels (e.g. IronMaxx Clear Whey Isolate Zero Kirsche, 1080x1064 px).
|
||
2. Koch agent repeatedly responded that the photo was blurry and illegible.
|
||
3. Internal chain-of-thought hallucinated that the label was a small blurry thumbnail with ~4px text reading "MIRKIC" instead of "KIRSCHE".
|
||
4. The turn completed with `stopReason: "stop"` without emitting any visible text, causing OpenClaw to emit `The tool run finished, but no final summary was produced. I did not repeat any completed actions.`
|
||
5. Ad-hoc Python image manipulation failed due to missing `PIL` (Pillow).
|
||
6. Sporadic tool call errors across topics: `tool_search` wrapped in `tool_call`, `sessions_history` negative offsets, `ask_user` flat parameters, missing `googleapiclient` without `uv run`.
|
||
- **Root Cause**:
|
||
1. **OpenClaw `imageModel` Fallback Behavior**:
|
||
OpenClaw's configuration schema defines `imageModel` as: *"Optional image model used when the primary model lacks image input."*
|
||
Because `z-ai/glm-5.3-flash` advertises `input_modalities: ["text", "image", "video"]` on OpenRouter, OpenClaw considered it image-capable and never routed images to `imageModel: "openrouter/google/gemini-3.8-flash"`.
|
||
GLM-5.3-Flash's actual vision perception on dense German labels is poor, leading it to misread text, conclude it cannot read it, and finish its thinking block without generating any output text.
|
||
2. **Empty Turn Finalization Fallback**:
|
||
When a model finishes with only thinking tokens and zero output text or tool calls, OpenClaw's delivery mirror triggers `settled-finalization-fallback`.
|
||
3. **Python Runtime Deficiencies**:
|
||
Core libraries (`pillow`, `google-api-python-client`, `pypdf`) were not installed in the persistent Python interpreter at `/home/openclaw/.openclaw/.local/python/...`.
|
||
- **Resolution**:
|
||
1. **Dedicated Vision Model for Koch**:
|
||
Configured `koch`'s primary model to `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`) both live via `openclaw config set` and in `openclaw-instance.secret.yaml`. Gemini 3.8 Flash provides SOTA OCR and dense-table extraction at $0.075 / 1M cached prompt tokens.
|
||
2. **Prompt-Hardening in `knowledge-wiki`**:
|
||
- Updated `knowledge-wiki/agents/koch.md` with strict rules: mandatory visible text output on every turn (no silent thinking stops), Vision-first protocol (transcribe all tables without claiming blurriness).
|
||
- Updated `knowledge-wiki/agents/README.md` with cross-agent tool-calling standards (`tool_search` top-level only, `sessions_history` non-negative offset + `sessionKey`, `ask_user` questions array, `uv run` for Python scripts).
|
||
3. **Pre-installed Python Libraries**:
|
||
Installed `pillow`, `google-api-python-client`, and `pypdf` into the persistent Python environment using `uv pip install --break-system-packages`, and added the command to `openclaw-instance.secret.yaml` for permanent self-healing.
|
||
4. **Retroactive Meal Intake Recovery & Compaction**:
|
||
The user prompt (`540g selbst gemachte Lasagne mit bechqmele sauce`) that was dropped during the unfinalized GLM loop was logged retroactively into DuckDB (`01M4RQT5PFNN7J4307TCRM3E80`, ~864 kcal, ~43.2 g Protein). Daily totals (both portions 540g + 430g and the 30g Clear Whey shake) updated to 3,266 kcal / 177.9 g protein, web dashboards re-rendered, and Topic 21 compacted from 49.6k tokens down to 4.6k tokens.
|
||
- **How to Avoid Recurrence**:
|
||
- For agents that routinely process user photos or document scans, assign dedicated vision models (`google/gemini-3.8-flash`) as the agent's primary model rather than relying on `agents.defaults.imageModel`.
|
||
- Maintain the response guarantee in agent prompts to prevent empty turns when reasoning models encounter uncertainty.
|
||
|
||
---
|
||
|
||
### Issue 28: Topic 20 #Sport Transcript Blindness: GLM-5.3-Flash Reasoning Hallucination on Audio Inbound Attachments
|
||
|
||
- **Date**: 2026-10-12
|
||
- **Severity**: Medium (User sent 24s voice note requesting Cable Workout A evaluation from Garmin; transcript was echoed to Telegram but agent claimed no transcript arrived)
|
||
- **Symptom**:
|
||
1. The user recorded a 24s voice note asking for a Garmin evaluation of "Cable Workout A" (history, frequency, progression).
|
||
2. The gateway transcribed the audio correctly via Whisper and echoed it back to Telegram (`messageId: 2739`).
|
||
3. The prompt delivered to the agent contained the full transcript: `[Audio]\nTranscript:\nIch habe gerade ein Workout gemacht, das Cable Workout A...`.
|
||
4. Yet after 78 tool steps and 19.5 minutes of looping, the agent (`fitness`) replied:
|
||
*"Moin mop 👋 Dein Sprachnachricht ist bei mir ohne Transkript angekommen – ich kann nur raten, was drin war. 🎙️"*.
|
||
- **Root Cause**:
|
||
1. **GLM-5.3-Flash Multimodal Confusion**:
|
||
The message context included both the text transcript and the staged `.ogg` attachment in `message.media`.
|
||
In its internal reasoning (`thinking`), GLM-5.3-Flash hallucinated:
|
||
*"Voice message from Moritz. Need transcript — it's not provided this time? The context shows a voice message but no transcript. Hmm, the inbound is voice; transcript usually would be in context. Not visible. Maybe I need to transcribe."*
|
||
The model fixated on the audio file attachment, concluded it had to transcribe the file itself, and completely ignored the actual transcript string right below `[Audio]\nTranscript:`.
|
||
2. **19.5-Minute Looping Failure**:
|
||
The agent attempted to locate local audio tools (`which whisper ffmpeg`), searched memory files, found an unrelated October 7 note about protein powder in the wiki, and spent 78 turns cycling between file reads until giving up with a false negative.
|
||
- **Resolution**:
|
||
1. **Model Upgraded to Gemini 3.8 Flash**:
|
||
Replaced `primary: "openrouter/z-ai/glm-5.3-flash"` with `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`, `qwen3.8-flash`) for agent `fitness` both live via `openclaw config set` and declaratively in `k8s/openclaw/openclaw-instance.secret.yaml`. Gemini 3.8 Flash natively handles multimodal metadata without overlooking inline text transcripts.
|
||
2. **Garmin Workout Data Synced & Evaluated**:
|
||
Executed `garmin_sync.py` to mirror the latest workouts into `data/garmin/activities/2026-10.jsonl`.
|
||
Retrieved all 3 sessions of Moritz's `SuperSetACable-55mins` (2026-09-28, 2026-10-06, 2026-10-12) and extracted the workout progression data.
|
||
- **How to Avoid Recurrence**:
|
||
- In agent system prompts, reinforce that `[Audio]\nTranscript:\n<text>` already contains the authoritative, finalized speech-to-text result and requires no local audio file re-processing.
|
||
|
||
### Issue 29: Decoupled Vision Perception — Gemini 3.8 Flash Media-Understanding with GLM-5.3-Flash Primary Chat Engine
|
||
- **Symptoms**:
|
||
- Switching the primary agent model completely to `google/gemini-3.8-flash` changed the entire conversational tone and agent persona across all text turns, whereas only high-fidelity OCR and nutrition image perception were desired.
|
||
- When `openrouter/z-ai/glm-5.3-flash` was the primary model, OpenClaw previously routed photos natively to GLM because the OpenRouter model catalog declared multimodal capability (`image`), causing GLM to fail at OCR reading of nutritional labels.
|
||
- **Root Cause & Architectural Discovery**:
|
||
1. OpenClaw checks `activeModelSupportsNativeVision` and `resolveRunModelHasVision`. If the primary model advertises vision on OpenRouter AND no explicit image understanding models are declared in `tools.media.models`, OpenClaw bypasses the `media-understanding` pipeline and passes raw images natively to the chat engine.
|
||
2. Because GLM-5.3-Flash struggled with German nutritional tables, the entire primary model had been switched to Gemini 3.8 Flash, which Moritz explicitly rejected ("Haben wir nun das Modell für den Koch agent komplett auf Gemini 3.8 flash gewechselt? Das wollte ich nicht. Nur für die Bilderkennung bitte.").
|
||
- **Resolution**:
|
||
1. **Reverted Primary Chat Models**:
|
||
Reverted `agents.entries.koch.model.primary` and `agents.entries.fitness.model.primary` back to `openrouter/z-ai/glm-5.3-flash` (with fallbacks `deepseek-v4.1-flash`, `qwen3.8-flash`) in both `openclaw.json` and `openclaw-instance.secret.yaml`.
|
||
2. **Configured Dedicated Media-Understanding Image Engine**:
|
||
Added an explicit `image` capability model under `tools.media.models`:
|
||
```yaml
|
||
tools:
|
||
media:
|
||
image:
|
||
enabled: true
|
||
preferredModel: "openrouter/google/gemini-3.8-flash"
|
||
models:
|
||
- provider: openrouter
|
||
model: "google/gemini-3.8-flash"
|
||
capabilities: ["image"]
|
||
prompt: "Beschreibe das Bild detailliert auf Deutsch. Wenn Nährwertangaben, Zutaten, Barcode, Verpackungstext oder handschriftliche Notizen zu sehen sind, transkribiere alle Zahlen, Gramm-Angaben und Nährwerte (Kalorien, Eiweiß/Protein, Kohlenhydrate, Fett) vollständig und exakt."
|
||
```
|
||
3. **Declared Text-Only Override for GLM**:
|
||
Under `models.providers.openrouter.models`, registered `z-ai/glm-5.3-flash` with `input: ["text"]`. This signals to OpenClaw that GLM must never receive raw images natively, forcing automatic routing of all image attachments through Gemini 3.8 Flash media-understanding before handing off enriched text to GLM.
|
||
4. **Updated Persisted SQLite Sessions**:
|
||
Reset `session_nodes` for `agent:koch:telegram:group:-1004326921510:topic:21` and `agent:fitness:telegram:group:-1004326921510:topic:20` to `z-ai/glm-5.3-flash`.
|
||
- **How to Avoid Recurrence**:
|
||
- Always decouple perception tasks (OCR, image description, speech-to-text) into `tools.media` or `imageModel`, keeping primary conversational models focused on reasoning and dialogue without changing the core chat persona.
|
||
|
||
|