moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(openclaw): upgrade fleet to DeepSeek V4.1 Flash 1017 and harden operations

Browse Source 
- Upgrade OpenClaw primary model from deepseek-v4.1-flash to deepseek-v4.1-flash-1017 across all interactive agents, heartbeats, compaction, and cron jobs.
- Configure hardened OpenRouter provider routing for 1017 (deepinfra, inference-net, parasail, cloudflare) with strict fp8/bf16 quantization and tool_choice verification.
- Retain deepseek-v4.1-flash (0910), z-ai/glm-5.3-flash, and qwen3.8-flash in the automated fallback chain.
- Update ADR 0002 with 1017 economics (30% price reduction on prompt/cache, +60% intelligence benchmark increase) and a 7-point provider qualification playbook.
- Add operational scripts and tooling for session compaction, subagent vector deduplication, gateway preflight, and triage diagnostics in k8s/openclaw.
- Update DuckDB backend engine, git-sync logic, and test suites.
- Enhance Mila tracking UI components and add container build configuration.
- Sync Terraform infrastructure state for Authentik.
This commit is contained in:
Moritz Graf
2026-10-18 10:54:59 +02:00
parent 7ce1da18ab
commit dd60a095c6
26 changed files with 1786 additions and 1301 deletions
Download Patch File Download Diff File Expand all files Collapse all files
k8s/openclaw/TROUBLESHOOTING.md
+45
View File
@@ -966,4 +966,49 @@ This document captures fixed issues, operational quirks, and incident root-cause
- **How to Avoid Recurrence**:
- Always decouple perception tasks (OCR, image description, speech-to-text) into `tools.media` or `imageModel`, keeping primary conversational models focused on reasoning and dialogue without changing the core chat persona.
---
### Issue 30: GLM-5.3-Flash Malformed Tool Call Stream Aborts (`streamlake/fp8`), Cold HDD Prompt Assembly Stall, and Fleet-Wide Migration to DeepSeek V4.1 Flash
- **Date**: 2026-10-17
- **Severity**: High (Topic 164 `#Grabengasse & Media` failed with `Provider returned an incomplete or malformed tool call`; 12 total occurrences across 5 agents in 17 days; 504.5s cold-disk event-loop stall on first morning request)
- **Symptom**:
1. At 07:11:02 CEST, Moritz sent a voice note in Topic 164 (`#Grabengasse & Media`, agent `grabengasse_media`): *"Kannst du mir auf dem Sonos in der Küche die Bibi Blocksberg Playlist Der Affe ist los abspielen?"*
2. The turn stalled for **8 minutes and 24 seconds** (`504,493 ms`) during `system-prompt` assembly (`cpuCoreRatio=0.005`, pure disk I/O wait) before sending the first LLM request at 07:19:26 CEST.
3. In Turn 1 (`SEQ 175`), `z-ai/glm-5.3-flash` called `tool_search({"query": "sonos spotify play"})`. In Turn 2 (`SEQ 177`), it called `tool_describe({"name": "mcp:spotify:spotify__search"})`.
4. In Turn 3 (`SEQ 179`, `responseId: gen-1792214444-T46fmmBLqgmCCe0YJp4X`), after generating 326 reasoning tokens (`"Search for 'Bibi Blocksberg Der Affe ist los' using mcp:spotify:spotify__search"`), the upstream OpenRouter provider (`streamlake/fp8`) aborted the stream mid-JSON while serializing nested `tool_call({"id": "mcp:spotify:spotify__search", "args": {...}})` arguments:
`stopReason: "error"`, `errorMessage: "Provider returned an incomplete or malformed tool call"`.
5. OpenClaw only retries or triggers model fallbacks on HTTP status errors (`429`, `502`, `503`, `504`) prior to stream start—not when an HTTP `200 OK` SSE stream terminates mid-generation with a malformed tool-call chunk.
- **Root Cause & Fleet Forensics**:
1. **`z-ai/glm-5.3-flash` Nested Tool-Call Fragility on `streamlake/fp8` & `siliconflow/fp8`**:
- Across October 1–17, 2026, `Provider returned an incomplete or malformed tool call` occurred **12 times** across `main` (6×), `grabengasse_media` (3×), `koch` (1×), `hebamme` (1×), and `doktor` (1×)—exclusively on `z-ai/glm-5.3-flash`, always right after `tool_describe` when emitting nested `tool_call` JSON payloads.
2. **Suboptimal OpenRouter Provider Routing**:
- `extra_body.provider.order` for `z-ai/glm-5.3-flash` listed `"cloudflare"` first (which does not host `glm-5.3-flash` on OpenRouter), falling through to `streamlake/fp8` (91.7% uptime) and `siliconflow/fp8` (74.1% uptime), and did not block `fp4` (`nvfp4`) quantizations (`phala`, `atlas-cloud`, `parasail`).
3. **Fleet-Wide Cost & Cache-Hit Economics (`DeepSeek V4.1 Flash` vs. `GLM 5.3 Flash`)**:
- Over the 17-day period (Oct 1–17, 2026), 73% to 94% of all tokens across interactive agents (`21.93M` `cacheRead` vs. `8.94M` uncached input) and background heartbeats (`22.86M` `cacheRead` vs. `7.73M` uncached input) were prompt cache reads.
- While `deepseek/deepseek-v4.1-flash` has a slightly higher base uncached prompt rate (`$0.14–$0.15/M` vs. `$0.06–$0.075/M`), its top `fp8` providers (`deepinfra` and `streamlake`, both 100% uptime and full `tool_choice` support) charge **3.5× to 10× less for `cacheRead` tokens** (`$0.0042/M` on DeepInfra, `$0.0030/M` on StreamLake vs. `$0.015–$0.030/M` on GLM 5.3 Flash).
- Consequently, running the entire fleet (interactive agents + background heartbeats) on `deepseek/deepseek-v4.1-flash` via `deepinfra`/`streamlake` costs **~$2.69 over 17 days (~$4.75/month)** vs. **$4.50 actual on `GLM 5.3 Flash` (~$7.95/month)**—a **~40% net cost reduction** alongside 0 malformed tool calls.
4. **Cold Rotational HDD Read Stall After Nightly Backups**:
- Nightly backups (`03:30`) and compaction (`04:45`) evicted the Knowledge Wiki and agent workspace `*.md` files from the Linux kernel page cache, causing the first morning turn's synchronous `system-prompt` loader to block on rotational SATA HDD seeks for 504.5s.
- **Resolution**:
1. **Fleet-Wide Migration to `openrouter/deepseek/deepseek-v4.1-flash-1017` (with `deepseek-v4.1-flash` 0910 as first fallback)**:
- Switched `agents.defaults.model.primary`, `utilityModel`, `heartbeat.model`, `subagents.model`, `memory-core.config.dreaming.model`, and all interactive agent entries (`main`, `fitness`, `koch`, `hebamme`, `travel`, `shopping`, `regensburg`, `recherche`, `doktor`, `itguy`, `grabengasse_media`, `mila_tracker`) to the newly released October 17, 2026 snapshot `openrouter/deepseek/deepseek-v4.1-flash-1017` (Artificial Analysis Intelligence Index 39.5 vs. 24.7 on 0910, and 30% cheaper on DeepInfra at `$0.098/M` prompt, `$0.294/M` completion, `$0.00294/M` `cacheRead`), with `openrouter/deepseek/deepseek-v4.1-flash`, `openrouter/z-ai/glm-5.3-flash`, and `openrouter/qwen/qwen3.8-flash` as fallbacks (`nutrition_tracker` remains on `openrouter/google/gemini-3.8-flash`).
2. **Preserved Decoupled Vision Perception (Issue 29 Compatibility)**:
- Registered both `deepseek/deepseek-v4.1-flash-1017` and `deepseek/deepseek-v4.1-flash` with `input: ["text"]` under `models.providers.openrouter.models` in `openclaw-instance.secret.yaml` so OpenClaw continues routing all image attachments through `tools.media.image` (`google/gemini-3.8-flash`) for OCR before passing enriched text to DeepSeek V4.1 Flash.
3. **Hardened OpenRouter Provider & Quantization Routing (`extra_body.provider`)**:
- Configured `openrouter/deepseek/deepseek-v4.1-flash-1017` with `order: ["deepinfra", "inference-net", "parasail", "cloudflare"]`, `quantizations: ["fp8", "bf16", "unknown"]`, and `ignore: ["morph", "baseten", "deepseek", "open-inference", "decart", "sail-research", "wafer"]` (live routing tests showed `morph/fp8` is rejected by OpenRouter's `Filter by Tool Compatibility` when `tools` are passed, `deepseek` is blocked by account training-opt-out guardrails, and `baseten` returns HTTP 429).
- Configured fallback `openrouter/deepseek/deepseek-v4.1-flash` (`0910`) with `order: ["deepinfra", "streamlake", "inference-net", "digitalocean", "venice", "nebius"]` and `quantizations: ["fp8", "bf16", "unknown"]`.
- Cleaned up `openrouter/z-ai/glm-5.3-flash` fallback routing (`order: ["morph", "near-ai", "digitalocean", "venice", "streamlake"]`, `quantizations: ["fp8", "bf16", "unknown"]`, blocking `fp4`).
4. **Direct MCP Tool Fastpaths & 1-Turn Sonos Playbook for `grabengasse_media` (Topic 164)**:
- Added exact MCP tool IDs (`mcp:spotify:spotify__search`, `mcp:spotify:spotify__play`, `mcp:spotify:spotify__transfer_playback`, `mcp:jelly:jelly__jelly_search`, `mcp:jelly:jelly__jelly_request`, `mcp:homeassistant:homeassistant__ha_get_state`, `mcp:homeassistant:homeassistant__ha_call_service`) as well as all verified Sonos entity IDs (`media_player.bad`, `media_player.wohnzimmer`, `media_player.kuche_sonos`, `media_player.kuche_ikea`, `media_player.dining_room`) and Sonos `source_list` favorites (`"Fusion 2026 Sets"`, `"ChilloutTechno"`, `"deephouse"`, `"OberDITSCHI"`, etc.) directly to `openclaw-instance.secret.yaml`, `knowledge-wiki/agents/grabengasse_media.md`, and `/home/openclaw/.openclaw/workspace-grabengasse_media/AGENTS.md`.
- Eliminates both the 2 MCP discovery turns (`tool_search` + `tool_describe`) and the 2 file-read turns (`read("USER.md")`, `read("MEMORY.md")`, `read("SKILL.md")`).
5. **Extended Linux Page-Cache Pre-Warming to Knowledge Wiki & Workspace Markdown Files**:
- Updated `k8s/openclaw/scripts/gateway-preflight.mjs` and `k8s/openclaw/scripts/compact_sessions.py` (which now also runs `prewarm_page_cache()` at the end of the 04:45 nightly run) to pre-read all Knowledge Wiki (`*.md`, `*.jsonl`) and `workspace*/` `*.md` files into RAM, preventing morning `system-prompt` disk freezes.
6. **SQLite `session_nodes` Trigger (`entry_valid`) & Subagent Vector Re-Indexing Fix (`extraPaths: []`)**:
- Discovered that SQLite `session_nodes` has an `AFTER UPDATE OF entry_json` trigger (`session_nodes_entry_valid_after_entry_update`) that resets `entry_valid = 0` whenever `entry_json` is modified, throwing `SessionCanonicalKeyMigrationRequiredError` until `UPDATE session_nodes SET entry_valid = 1` is executed.
- Hardened `k8s/openclaw/scripts/recover_sessions.py` to always enforce `PRAGMA foreign_keys = ON` before deleting trailing error turns, set `entry_valid = 1` across all rows, and reset `attempts = 0` on pending `channel_ingress_events` in `state/openclaw.sqlite`.
- Explicitly set `memory.search.extraPaths: []` at the top level in `openclaw-instance.secret.yaml` and `dedup_subagent_vectors.py` so `mergeMode: merge` does not leave `extraPaths` active globally in `openclaw.json`, preventing subagents from re-embedding 4,059 wiki chunks on their first turn after startup.
- **How to Avoid Recurrence**:
- Always enforce `quantizations: ["fp8", "bf16"]` in OpenRouter `extra_body.provider` for tool-calling agents so OpenRouter never routes to degraded `fp4` (`nvfp4`) endpoints.
- Whenever updating `entry_json` in `session_nodes` via SQLite, always follow up with `UPDATE session_nodes SET entry_valid = 1` in a separate statement (after the `AFTER UPDATE OF entry_json` trigger fires), and always set `PRAGMA foreign_keys = ON` before deleting from `transcript_events`.
- Keep Knowledge Wiki and workspace Markdown files in the nightly `prewarm_page_cache()` pass so morning turns hit RAM (`<50ms`) instead of cold rotational disks.