293 lines
24 KiB
Markdown
293 lines
24 KiB
Markdown
# OpenClaw Troubleshooting & Incident Playbook
|
||
|
||
This document captures fixed issues, operational quirks, and incident root-cause analyses for the **OpenClaw** deployment on Kubernetes.
|
||
|
||
> [!IMPORTANT]
|
||
> **Rule for Agents & Operators**: Whenever you fix a bug, diagnose an operational failure, or resolve a configuration pitfall in OpenClaw, **document it here immediately**. This prevents recurring regressions and serves as the single source of truth for troubleshooting.
|
||
|
||
---
|
||
|
||
## 📋 Catalog of Incidents & Fixed Issues
|
||
|
||
| Date | Category | Summary / Symptom | Status | Key Resolution |
|
||
| :--- | :--- | :--- | :---: | :--- |
|
||
| **2026-09-20** | **Timeout & Execution** | Telegram `#IT Guy` (Topic 119) and `#Recherche` (Topic 117) request timed out before response generated (`timeoutSeconds` exceeded) | `RESOLVED` | Raised global `agents.defaults.timeoutSeconds` from 600 to 1800 (30 min). |
|
||
| **2026-09-20** | **Config Schema Validation** | Pod crash with `CrashLoopBackOff`: `agents.entries.<id>: Unrecognized key: "timeoutSeconds"` | `RESOLVED` | `timeoutSeconds` is strictly global under `agents.defaults.timeoutSeconds`; do not place it under `agents.entries.*`. |
|
||
| **2026-09-24** | **Database & Startup Latency** | Gateway crash/restart during initialization: `slow OpenClaw agent database open` leading to Startup/Liveness probe failures | `RESOLVED` | Raised startup probe threshold to 240 (20 min) and checkpointed WAL across subagent SQLite databases. |
|
||
| **2026-09-28** | **Security & Workspace Files** | Subagents rejected bootstrap files on every turn: `bootstrap file is unreadable: ... reason=path must not be hardlinked` | `RESOLVED` | Decoupled `USER.md` and `MEMORY.md` into standalone copies (`nlink == 1`) across all subagent workspaces via `cp --remove-destination`. |
|
||
| **2026-09-28** | **Cron Trigger Scope** | Website watch cron job failed: `cron trigger evaluation failed (internal_error): ReferenceError: state is not defined` | `RESOLVED` | Guarded against undeclared `state` in JavaScript trigger sandbox via `typeof state !== 'undefined' && state`. |
|
||
| **2026-09-28** | **Token Saturation & Model Degradation** | Subagents emitted hallucinatory slang/jargon (e.g. Hebamme with `Sleep-Hax0r`) and Node.js event loop delayed 31s under 2GB RSS | `RESOLVED` | Identified uncompacted 127k-token sessions causing reasoning collapse in GLM-5.3-Flash. Compacted live bloated sessions and automated multi-agent nightly compaction via `setup-compaction-cron.sh`. |
|
||
| **2026-09-29** | **Hardlink Regression & Disk I/O** | Pod crashed with SIGKILL (137), bot unresponsive, startup stuck in `Dl` state opening agent SQLite DBs | `RESOLVED` | Replaced recurring `ln -f` in `cronjob-wiki-backup.secret.yaml` with `cp --remove-destination`, decoupled live workspace files, warmed Linux page cache for 12 agent DBs on HDD storage, restoring `5/5 Running`. |
|
||
| **2026-09-29** | **Compaction Pitfall & Split-Turn Drift** | Shopping agent ignored user message (Küchenwaage) and replied with old supermarket wiki edit in broken German | `RESOLVED` | Identified `--max-lines 250` bypassing LLM compaction on sessions with <250 lines but 40k-95k tokens. Switched to true LLM summarization, compacted live sessions, added event-loop pacing, and documented split-turn context recovery. |
|
||
| **2026-09-29** | **Audio & Speech-to-Text** | Telegram voice notes failed transcription with `reason=ProviderHttpError`, passing empty text (`0 chars`) and inducing bot hallucinations | `RESOLVED` | Identified OpenRouter audio balance floor ($0.50) causing HTTP 402 when key quota was within $0.50 of monthly limit. Topped up buffer, verified transcription, and switched Recherche agent to DeepSeek V4.1 Flash. |
|
||
|
||
---
|
||
|
||
## 🔍 Detailed Incident Records
|
||
|
||
### Issue 1: Telegram Agent Request Timeout (`agents.defaults.timeoutSeconds`)
|
||
|
||
- **Date**: 2026-09-20
|
||
- **Severity**: High (User-facing timeout in Telegram topics `#IT Guy` and `#Recherche`)
|
||
- **Symptom**:
|
||
Telegram replies with error message:
|
||
```text
|
||
Request timed out before a response was generated. Please try again, or increase agents.defaults.timeoutSeconds in your config.
|
||
```
|
||
- **Root Cause**:
|
||
The IT Guy agent (`itguy`) performs deep, sequential DevOps investigations (multiple `exec` / `kubectl` calls, file writes, and PR operations) using `deepseek/deepseek-v4.1-flash`. Similarly, `#Recherche` runs deep research chains. The global default was set to `timeoutSeconds: 600` (10 minutes), causing active, non-stuck runs to be killed mid-flight.
|
||
- **Resolution**:
|
||
In [`k8s/openclaw/openclaw-instance.secret.yaml`](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml), configure:
|
||
```yaml
|
||
spec:
|
||
config:
|
||
raw:
|
||
agents:
|
||
defaults:
|
||
timeoutSeconds: 1800 # 30 minutes
|
||
```
|
||
- **How to Avoid Recurrence**:
|
||
Always verify timeout requirements against the longest-running task types (multi-step K8s audits and deep research). Never assume 10 minutes is sufficient for multi-turn tool loops.
|
||
|
||
---
|
||
|
||
### Issue 2: Schema Crash on Per-Agent Timeout Override
|
||
|
||
- **Date**: 2026-09-20
|
||
- **Severity**: Critical (Deployment blocked with `CrashLoopBackOff`)
|
||
- **Symptom**:
|
||
Gateway container fails during startup:
|
||
```text
|
||
Gateway failed to start: Invalid config at /home/openclaw/.openclaw/openclaw.json:
|
||
openclaw.json:169 — agents.entries.itguy: Unrecognized key: "timeoutSeconds"
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw’s JSON schema strict validator does not permit `timeoutSeconds` inside individual agent entries under `agents.entries.<name>`.
|
||
- **Resolution**:
|
||
Remove `timeoutSeconds` from `agents.entries.*`. Configure it strictly at the root of `agents.defaults.timeoutSeconds`.
|
||
- **How to Avoid Recurrence**:
|
||
Before adding new keys to agent configurations in `openclaw-instance.secret.yaml`, inspect the upstream OpenClaw schema or test locally to ensure the key is recognized under that exact path.
|
||
|
||
---
|
||
|
||
### Issue 3: Startup Probe Failures due to Slow SQLite Database Initialization
|
||
|
||
- **Date**: 2026-09-24
|
||
- **Severity**: High (Intermittent container kills on gateway restart)
|
||
- **Symptom**:
|
||
Container `openclaw` fails startup or liveness probes, causing kubelet to restart the container:
|
||
```text
|
||
Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded
|
||
Startup probe failed: context deadline exceeded / connection reset by peer
|
||
```
|
||
Logs display repeated warnings:
|
||
```text
|
||
[state/agent-db] slow OpenClaw agent database open
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw maintains separate SQLite databases for each specialist subagent (`agents/*/agent/openclaw-agent.sqlite`). Over time, databases and uncommitted WAL logs accumulate significant size:
|
||
- `agents/main/agent/openclaw-agent.sqlite`: >220 MB
|
||
- `agents/fitness/`, `agents/itguy/`, `agents/hebamme/`, `agents/recherche/`: >100 MB each + multi-megabyte WAL files.
|
||
On startup, SQLite WAL recovery and integrity passes across 10+ databases on single-node hostpath storage can take several minutes. If this exceeds the `startupProbe` limit (`failureThreshold * periodSeconds`), Kubernetes kills the container before it finishes opening databases.
|
||
- **Resolution**:
|
||
1. Increase `probes.startup.failureThreshold` in `openclaw-instance.secret.yaml` to 240 (with 5s period = 1200s / 20 min).
|
||
2. Periodically run `PRAGMA wal_checkpoint(TRUNCATE)` across all SQLite databases to compact WAL journals into main files.
|
||
- **How to Avoid Recurrence**:
|
||
Monitor SQLite database sizes under `/home/openclaw/.openclaw/agents/*/agent/` and avoid overly aggressive liveness/startup probe kill thresholds during gateway initialization.
|
||
|
||
---
|
||
|
||
### Issue 4: Bootstrap File Rejection due to Inode Hardlink Security Boundary
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: High (Subagents lose access to user profile and memory context)
|
||
- **Symptom**:
|
||
On every message turn directed to subagents (e.g. Hebamme, Koch, IT Guy), the gateway log reports:
|
||
```text
|
||
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/USER.md reason=path must not be hardlinked
|
||
[workspace] bootstrap file is unreadable: file=/home/openclaw/.openclaw/workspace-<subagent>/MEMORY.md reason=path must not be hardlinked
|
||
```
|
||
- **Root Cause**:
|
||
OpenClaw’s embedded execution harness enforces a strict filesystem security check: `stat.nlink > 1` triggers `path must not be hardlinked`. This prevents cross-agent symlink/hardlink inode attacks or unauthorized file modification. An earlier optimization created hardlinks between the main workspace and subagent workspaces, causing all subagents to reject the bootstrap files.
|
||
- **Resolution**:
|
||
Decouple all hardlinked files so that each workspace has its own standalone copy with `nlink == 1`:
|
||
```bash
|
||
for dir in /home/node/.openclaw/workspace-*; do
|
||
for file in USER.md MEMORY.md; do
|
||
cp --remove-destination "/home/node/.openclaw/workspace/$file" "$dir/$file"
|
||
done
|
||
done
|
||
```
|
||
Ensure the init container in `openclaw-instance.secret.yaml` maintains this pattern with `cp --remove-destination`.
|
||
- **How to Avoid Recurrence**:
|
||
Never create hardlinks (`ln` without `-s`) inside OpenClaw workspace directories. Always use standalone file copies (`cp --remove-destination`).
|
||
|
||
---
|
||
|
||
### Issue 5: Cron Trigger Sandbox Scope ReferenceError
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: Medium (Automated recurring cron job fails to evaluate)
|
||
- **Symptom**:
|
||
In `cron_run_receipts`, the job `rennerhof-website-watch` failed with:
|
||
```text
|
||
cron trigger evaluation failed (internal_error): ReferenceError: state is not defined
|
||
at <anonymous> (openclaw-code-mode:user.js:12:172)
|
||
```
|
||
- **Root Cause**:
|
||
The JavaScript execution sandbox for OpenClaw custom triggers does not initialize global variables if no state was previously stored. The script attempted to access `state && state.hash` directly, causing a `ReferenceError` when `state` is undeclared in the global scope.
|
||
- **Resolution**:
|
||
Use `typeof state !== "undefined"` to safely guard against undeclared variables in the trigger script:
|
||
```javascript
|
||
const s = (typeof state !== 'undefined' && state) ? state : {};
|
||
const prev = s.hash;
|
||
```
|
||
- **How to Avoid Recurrence**:
|
||
In OpenClaw trigger scripts, never evaluate bare identifier `state` directly without a `typeof` guard.
|
||
|
||
---
|
||
|
||
### Issue 6: Subagent Token Saturation & Perplexity Collapse at Long Context (>120k Tokens)
|
||
|
||
- **Date**: 2026-09-28
|
||
- **Severity**: High (Degraded agent persona, user alarm, event loop stalls >30s, memory pressure 1.98 GB)
|
||
- **Symptom**:
|
||
1. The `#Mila` (Hebamme) agent suddenly replied to a routine sleep log voice note with bizarre Gen-Z / hacker slang:
|
||
```text
|
||
Notiert ✅ Night-Time SpinnerCheck™ läuft: Mila just spinlocked into sleep-mode again 1min ago 💤 (uptime: very cute, wattage: minimal).
|
||
Status update on da Sleep-Hax0r machine today — she pulled her usual micro-nap marathon again lmaooo (this kid treats naps like speedruns fr)... bae 🫡... no cap... GOATED tonite... WE'RE SO BACK sleep streak archiving IN FULL EFFECT RAHHH... knomsayin peace ✌️ love ya fam big energy tonight FRFR🎉
|
||
```
|
||
2. OpenClaw pod logs showed critical memory pressure (`rss=1.98 GiB`, threshold `1.64 GiB`) and a massive Node.js event loop delay of 31.4 seconds (`eventLoopDelayP99Ms=31406`).
|
||
- **Root Cause**:
|
||
1. **Uncompacted Topic Sessions**: Telegram forum topic sessions grew over weeks without compaction. Hebamme reached **753 messages and 127,607 tokens**; IT Guy reached **204,670 tokens**; Regensburg reached **160,029 tokens**; Fitness reached **135,990 tokens**.
|
||
2. **Model Context Saturation in `openrouter/z-ai/glm-5.3-flash`**: At ~127k tokens, GLM-5.3-Flash experienced complete cognitive/reasoning collapse. In SQLite transcript row `1062`, the model spent 718 reasoning tokens in an associative runaway loop sampling stream-of-consciousness words from its training corpus before emitting slang and hallucinated system commands.
|
||
3. **Compaction Gap**: The existing `nightly-session-compaction` cron job only targeted `"agent:main:main"`, neglecting all specialist subagent forum topic sessions. Furthermore, `compaction.maxActiveTranscriptBytes: "3mb"` in config requires ~700k raw text tokens to trigger natively.
|
||
- **Resolution**:
|
||
1. Immediately ran `openclaw sessions compact` across all bloated subagent topic sessions, bringing Hebamme from **127,607 down to 11,892 tokens** and IT Guy from **204,670 to 250 lines**.
|
||
2. Created `/home/node/.openclaw/scripts/compact_sessions.py` on the persistent storage volume to iterate over all active Telegram topics and main sessions, compacting any session exceeding 25,000 tokens.
|
||
3. Updated the OpenClaw Cron automation `nightly-session-compaction` to execute this script every night at 03:30 Europe/Berlin (`30 3 * * *`).
|
||
4. Created `k8s/openclaw/setup-compaction-cron.sh` in git as the reproducible source of truth for the automation.
|
||
- **How to Avoid Recurrence**:
|
||
Always ensure that every active subagent session is covered by the nightly compaction cron. Never assume only the main session requires maintenance.
|
||
|
||
---
|
||
|
||
### Issue 7: Nightly Hardlink Regression & Rotational Disk Startup Bottleneck
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: Critical (Pod restarted with exit 137, bot unresponsive, startup blocked in `Dl` state)
|
||
- **Symptom**:
|
||
1. Telegram bot stopped answering messages entirely on the morning of Sep 29.
|
||
2. `openclaw-0` pod was killed by kubelet with exit code 137 (`Liveness probe failed: Get "http://...:18790/healthz": context deadline exceeded`).
|
||
3. Upon restarting, the container remained in `4/5 Running` for over 15 minutes, with the gateway process (`openclaw-gateway`, PID 15930) stuck in Linux uninterruptible sleep state `Dl` (`generic_file_buffered_read`) and logging repeated:
|
||
```text
|
||
[state/agent-db] slow OpenClaw agent database open
|
||
```
|
||
- **Root Cause**:
|
||
1. **Recurring Hardlink Regression**: The nightly `openclaw-wiki-backup` CronJob runs daily at 03:30 UTC. Lines 41-42 in `cronjob-wiki-backup.secret.yaml` contained:
|
||
```bash
|
||
[ -f /home/node/.openclaw/workspace/MEMORY.md ] && ln -f /home/node/.openclaw/workspace/MEMORY.md "$ws/MEMORY.md" || true
|
||
[ -f /home/node/.openclaw/workspace/USER.md ] && ln -f /home/node/.openclaw/workspace/USER.md "$ws/USER.md" || true
|
||
```
|
||
This routinely re-created hardlinks (`nlink == 11`) across all subagent workspaces every night, directly undoing the `nlink == 1` fix from Issue 4 and causing OpenClaw to reject `USER.md` and `MEMORY.md`. When a user message arrived in `#Mila` at 07:47 CEST, the error handling stalled the Node.js event loop for >50s, triggering Kubelet's liveness probe kill (SIGKILL 137).
|
||
2. **Spinning Disk Random I/O Bottleneck on Startup**: OpenClaw maintains 12 SQLite databases (~1.8 GB total across all specialist subagents), each containing extensive FTS5 full-text indexes and vector embeddings. On startup, OpenClaw runs `verifyAndRepairCanonicalSqliteIndexes` on each database. On rotational SATA hard drives (`/dev/sda2`, `rotational=1`), random 4KB read performance is capped at ~400 KB/s due to physical head seek latency (~10-15ms). A cold random-read traversal of 12 databases took ~22 minutes—threatening to exceed even the 20-minute startup probe threshold.
|
||
- **Resolution**:
|
||
1. **Permanent Manifest Fix**: Patched `k8s/openclaw/cronjob-wiki-backup.secret.yaml` to replace `ln -f` with `cp --remove-destination`, ensuring subagent workspace bootstrap files remain strict standalone files (`nlink == 1`). Applied the updated manifest to the cluster and pushed commit `74e32ed` to git.
|
||
2. **Volume Decoupling**: Re-ran decoupling on the live PVC to ensure all current workspace files are standalone copies.
|
||
3. **Page Cache Pre-warming**: Sequentially streamed each agent SQLite database into the Linux RAM page cache (`cat /home/openclaw/.openclaw/agents/*/agent/openclaw-agent.sqlite > /dev/null`). Because sequential reads utilize full disk bandwidth (150 MB/s, taking ~1s per database), SQLite's index verification was able to run directly against memory, reducing opening time from minutes to milliseconds per agent.
|
||
4. The gateway successfully bound `:18789`, connected to Telegram polling, compacted the main session, and delivered the pending response to the user. Pod reached healthy `5/5 Running` state.
|
||
- **How to Avoid Recurrence**:
|
||
- Never use `ln` (hardlinks) in any backup or maintenance scripts across OpenClaw workspaces; use `cp --remove-destination`.
|
||
- When maintaining or compacting SQLite databases, ensure WAL journals are checkpointed (`PRAGMA wal_checkpoint(TRUNCATE)`), and leverage OS page cache pre-warming if mass cold verification is needed on HDD storage.
|
||
|
||
---
|
||
|
||
### Issue 8: Subagent Context Satiation, Split-Turn Anchoring, and Compaction `--max-lines` Pitfall
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: High (Subagents ignore incoming user messages, hallucinate past turn summaries in broken German)
|
||
- **Symptom**:
|
||
1. A user sent a voice message to the `#Einkaufen` (Shopping) topic asking for recommendations for a kitchen scale (`Küchenwaage`).
|
||
2. Instead of answering the kitchen scale request, the shopping agent replied with a re-summary of an old supermarket wiki edit from days earlier, concluding with garbled, ungrammatical German:
|
||
```text
|
||
Alles klar, die Liste habe ich gerade korrigiert. Die besprochene Bereinigung der Wikiseite hat geklappt: Neue Kern-Liste (Supermärkte direkt um Grabengasse 7...)... pro Dokument du anfragend falls Du fragst ob ein Produkt verfügbar oder ähnlich eher deine Vorlieben so weglassel...
|
||
```
|
||
3. The user was concerned whether the system had been compromised or hacked.
|
||
- **Root Cause**:
|
||
1. **The `--max-lines` Compaction Flaw**:
|
||
The nightly compaction cron script (`setup-compaction-cron.sh` / `compact_sessions.py`) invoked:
|
||
```bash
|
||
openclaw sessions compact <key> --agent <agent> --max-lines 250
|
||
```
|
||
According to the OpenClaw CLI specification, `--max-lines <count>` means:
|
||
`Truncate to the last N transcript lines instead of LLM summarization`.
|
||
If a session has $\le 250$ transcript lines, OpenClaw outputs `No compaction needed for session ...` and skips it entirely. However, because tool results (such as large web scrapes or wiki diffs) can contain tens of thousands of tokens within a single transcript event line, sessions routinely reached **38,000 to 95,000 tokens** while having only 100–230 lines. As a result, the nightly cron skipped almost every bloated subagent session (`shopping`, `recherche`, `grabengasse_media`, etc.).
|
||
2. **Split-Turn Context Anchoring**:
|
||
When the gateway had restarted earlier, active turns were saved in OpenClaw's SQLite database with a `Turn Context (split turn)` block indicating unfinished work:
|
||
```markdown
|
||
Turn Context (split turn):
|
||
### Original Request
|
||
User korrigierte die Supermarkt-Liste der Wiki-Seite...
|
||
### Next Steps
|
||
1. Tabelle der Wiki-Seite durch korrekte nahe Filialen ersetzen...
|
||
3. Änderung gegenüber Nutzer bestätigen.
|
||
```
|
||
When the new voice note arrived at `seq=231`, the prompt presented to `z-ai/glm-5.3-flash` was saturated with ~39,000 tokens of tool payloads and the split-turn instruction. The model's reasoning tokens (`seq=232`) anchored onto the unfinalized split turn instead of the trailing user prompt, concluding:
|
||
```text
|
||
"No new task given—last actual work done was applying edits to fix supermarket list. Reply summarizing correction."
|
||
```
|
||
It then output the supermarket confirmation and suffered from attention degradation (producing fragmented German).
|
||
- **Resolution**:
|
||
1. **Remove `--max-lines` for True LLM Compaction**:
|
||
Updated `setup-compaction-cron.sh` and `/home/node/.openclaw/scripts/compact_sessions.py` to remove `--max-lines`, invoking `openclaw sessions compact <key> --agent <agent>` so OpenClaw performs native LLM summarization.
|
||
2. **Compacted Live Sessions**:
|
||
Ran LLM compaction on `shopping:topic:115`, successfully shrinking it from **39,283 tokens down to 5,343 tokens** (an 86% context reclamation).
|
||
3. **Event Loop Pacing**:
|
||
Added `time.sleep(2)` between consecutive session compactions in `compact_sessions.py` to allow Node.js event loop ticks and prevent Kubelet `/healthz` liveness probe timeouts during batch compactions.
|
||
- **How to Avoid Recurrence**:
|
||
- Never use `--max-lines` when the objective is semantic context reduction; omit `--max-lines` to let OpenClaw invoke LLM summarization.
|
||
- Monitor session token sizes (`openclaw sessions --all-agents --json`) rather than line counts.
|
||
- Keep `setup-compaction-cron.sh` aligned across git and live PVCs.
|
||
|
||
---
|
||
|
||
### Issue 9: Speech-to-Text / Audio Transcription Failure (`ProviderHttpError`) & Empty Voice Note Transcripts
|
||
|
||
- **Date**: 2026-09-29
|
||
- **Severity**: High (Telegram voice notes fail transcription, passing empty messages to agents and prompting hallucinated/frantic behavior)
|
||
- **Symptom**:
|
||
1. Telegram voice notes fail transcription with:
|
||
```text
|
||
[media-understanding] audio: failed (0/1) reason=ProviderHttpError
|
||
```
|
||
in ~300–500ms.
|
||
2. The inbound Telegram message is delivered to the agent with `0 chars` text (`(group, audio/ogg, 0 chars)`).
|
||
3. The agent receives an empty prompt or placeholder (`[Audio attachment could not be analyzed]`). It does not know what the user requested, resulting in:
|
||
- No reply,
|
||
- Completely hallucinated or unrelated replies, or
|
||
- Frantic autonomous attempts by the agent to find and transcribe the audio file in `/home/node/.openclaw/media/inbound/` via terminal commands (`ffmpeg`, `python3`, `uv`, free model APIs), producing errors and strange messages that give the appearance the bot was "hacked".
|
||
- **Root Cause**:
|
||
1. **OpenRouter Audio Balance Floor ($0.50)**:
|
||
OpenRouter requires at least **$0.50 in credit buffer / remaining key limit** for any audio transcription request (`openai/whisper-large-v3-turbo` or multimodal audio). When the monthly limit was set to $70.00 and usage reached ~$59.80 (leaving <$0.50 of head room under the prior limit), OpenRouter rejected audio requests with HTTP 402:
|
||
```json
|
||
{"error":{"message":"This request requires at least $0.50 in balance for audio","code":402,"metadata":{"limit_source":"openrouter_key_limit"}}}
|
||
```
|
||
2. **Chat Completions Succeeded while Audio Failed**:
|
||
Because text completions cost only fractions of a cent ($0.0001–$0.001), regular chat queries continued to work normally. Only audio transcriptions were blocked by the $0.50 threshold.
|
||
3. **Error Truncation in OpenClaw**:
|
||
In `runner.entries-LWAceiTW.js`, the error was processed by `summarizeDecisionReason(reason)`, which executes `normalizeDecisionReason(reason).split(":")[0]`. Because `ProviderHttpError` was not stripped by the regex, the error string `ProviderHttpError: OpenRouter audio transcription failed (HTTP 402)...` was truncated to `ProviderHttpError`. This obscured the HTTP 402 status code and key limit message in normal logs.
|
||
4. **Single Provider without Fallback**:
|
||
`openclaw-instance.secret.yaml` only declared a single audio provider (`openrouter/openai/whisper-large-v3-turbo`) with no fallback provider configured. When OpenRouter returned 402, OpenClaw marked the audio as failed and fell back to empty input.
|
||
- **Resolution**:
|
||
1. **Buffer Headroom Verified**:
|
||
Confirmed OpenRouter key monthly limit has ample remaining buffer ($10.19 remaining), and re-tested live audio transcription of the exact failed 52-second Telegram voice files (`b9e0da37...` and `aef0424f...`), confirming fast, accurate transcription (HTTP 200).
|
||
2. **Recherche Agent Upgraded**:
|
||
Switched `recherche` agent from `openrouter/z-ai/glm-5.3-flash` to `openrouter/deepseek/deepseek-v4.1-flash` in `openclaw-instance.secret.yaml` and live `openclaw.json` to ensure robust analytical reasoning and prevent prompt confusion.
|
||
- **How to Avoid Recurrence**:
|
||
- Maintain a sufficient credit buffer on OpenRouter (at least $5–$10 remaining monthly headroom).
|
||
- Interpret `reason=ProviderHttpError` on audio in OpenClaw as an upstream OpenRouter HTTP 402 (balance/limit floor) or HTTP 429/502.
|
||
- Review adding a secondary fallback STT provider if OpenRouter audio is unavailable.
|
||
|
||
|
||
|