moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(openclaw): upgrade fleet to DeepSeek V4.1 Flash 1017 and harden operations

Browse Source 
- Upgrade OpenClaw primary model from deepseek-v4.1-flash to deepseek-v4.1-flash-1017 across all interactive agents, heartbeats, compaction, and cron jobs.
- Configure hardened OpenRouter provider routing for 1017 (deepinfra, inference-net, parasail, cloudflare) with strict fp8/bf16 quantization and tool_choice verification.
- Retain deepseek-v4.1-flash (0910), z-ai/glm-5.3-flash, and qwen3.8-flash in the automated fallback chain.
- Update ADR 0002 with 1017 economics (30% price reduction on prompt/cache, +60% intelligence benchmark increase) and a 7-point provider qualification playbook.
- Add operational scripts and tooling for session compaction, subagent vector deduplication, gateway preflight, and triage diagnostics in k8s/openclaw.
- Update DuckDB backend engine, git-sync logic, and test suites.
- Enhance Mila tracking UI components and add container build configuration.
- Sync Terraform infrastructure state for Authentik.
This commit is contained in:
Moritz Graf
2026-10-18 10:54:59 +02:00
parent 7ce1da18ab
commit dd60a095c6
26 changed files with 1786 additions and 1301 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0002-openclaw-model-selection-tiering-and-economics.md
+107 -36
View File
@@ -1,5 +1,5 @@
---
status: "accepted (trial: GLM-5.3-Flash)"
status: "accepted (fleet-wide primary: DeepSeek-V4.1-Flash-1017; updated 2026-10-17)"
date: 2026-09-13
decision-makers: ["Moritz Graf", "Antigravity"]
consulted: []
@@ -28,7 +28,7 @@ How do we architect the full-spectrum model selection across **all modalities**
## Decision Drivers
* **State-of-the-Art Recency**: Exclusively employ the latest generation models (e.g. Gemini 3.1, GLM 5.3, Whisper V3 Turbo, Sonar Pro). Completely decommission legacy generations (e.g. Gemini 2.5).
* **State-of-the-Art Recency**: Exclusively employ the latest generation models (e.g. DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, Sonar Pro). Completely decommission legacy generations (e.g. Gemini 2.5).
* **Conversational & Reasoning Quality**: High emotional intelligence, nuanced German fluency, personal context recall, and strict adherence to tool-calling protocols (ADR 0001).
* **Strict Modality Decoupling**: Each capability must route to its dedicated, best-in-class engine:
- Text & Agentic Reasoning
@@ -38,9 +38,9 @@ How do we architect the full-spectrum model selection across **all modalities**
- Text-to-Speech / Audio Output (synthesizing briefings and podcasts)
- Web Search & Retrieval (grounded search)
- Semantic Embeddings (vector memory)
* **Inference Economics**: Low per-token costs for large conversational contexts (15k–80k tokens), zero background bleed, and predictable billing.
* **Inference Economics**: Low per-token costs for large conversational contexts (15k–80k tokens), high prompt-cache discount efficiency (`cacheRead`), zero background bleed, and predictable billing.
* **Latency & Turn Responsiveness**: Conversational turns on Telegram must return within **4–10 seconds**, avoiding provider queuing or massive prefill bottlenecks.
* **Resilience & Fault Tolerance**: Automated fallback chains across independent providers on OpenRouter.
* **Resilience & Fault Tolerance**: Automated fallback chains and strict quantization/provider pins (`fp8`/`bf16`, `tool_choice` support) across independent providers on OpenRouter.
---
@@ -60,22 +60,23 @@ How do we architect the full-spectrum model selection across **all modalities**
│ Tier 1 │ │ Tier 2 │ │ Tier 3 │ │ Tier 4 │ │ Tier 5 │
│ • Low Latency │ │ • Deep Analysis │ │ • target: none │ │ • Nano Banana 2 │ │ • Whisper V3 │
│ • Tool Routing │ │ • Autonomous │ │ • Zero Bleed │ │ • Dedicated │ │ • Gemini 3.1 │
│ • Native Vision │ │ • Isolated Run │ │ • 30m Polling │ │ Raster Head │ │ Flash TTS │
│ • Decoupled OCR │ │ • Isolated Run │ │ • 30m Polling │ │ Raster Head │ │ Flash TTS │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Primary (Trial):│ │ Primary (Trial):│ │ Primary: │ │ Primary: │ │ STT: │
│ z-ai/ │ │ z-ai/ │ │ z-ai/ │ │ google/ │ │ openai/ │
│ glm-5.3-flash │ │ glm-5.3-flash │ │ glm-5.3-flash │ │ gemini-3.1- │ │ whisper-large- │
│ │ │ │ │ │ │ flash-image │ │ v3-turbo │
│ Fallbacks: │ │ Fallbacks: │ │ Fallbacks: │ │ │ │
│ 1. deepseek- │ │ 1. deepseek- │ │ 1. deepseek- │ │ Fallbacks: │ │ TTS: │
│ v4.1-flash │ │ v4.1-flash │ │ v4.1-flash │ │ 1. gemini-3.1- │ │ google/ │
│ 2. qwen3.8-flash│ │ 2. qwen3.8-flash│ │ 2. qwen3.8-flash│ │ flash-preview│ │ gemini-3.1- │
│ │ │ │ │ │ │ 2. gemini-3-pro-│ │ flash-tts │
│ Vision Fallback:│ │ │ │ │ │ image │ │ (FB: MiniMax 2.8│
│ google/ │ │ │ │ │ │ │ │ Speech HD) │
│ Primary: │ │ Primary: │ │ Primary: │ │ Primary: │ │ STT: │
│ deepseek/ │ │ deepseek/ │ │ deepseek/ │ │ google/ │ │ openai/ │
│ deepseek-v4.1- │ │ deepseek-v4.1- │ │ deepseek-v4.1- │ │ gemini-3.1- │ │ whisper-large- │
│ flash-1017 │ │ flash-1017 │ │ flash-1017 │ │ flash-image │ │ v3-turbo │
│ │ │ │ │ │ │ │ │
│ Fallbacks: │ │ Fallbacks: │ │ Fallbacks: │ │ Fallbacks: │ │ TTS: │
│ 1. v4.1-flash │ │ 1. v4.1-flash │ │ 1. v4.1-flash │ │ 1. gemini-3.1- │ │ google/ │
│ 2. glm-5.3-flash│ │ 2. glm-5.3-flash│ │ 2. glm-5.3-flash│ │ flash-preview│ │ gemini-3.1- │
│ 3. qwen3.8-flash│ │ 3. qwen3.8-flash│ │ 3. qwen3.8-flash│ │ 2. gemini-3-pro-│ │ flash-tts │
│ │ │ │ │ │ │ image │ │ (FB: MiniMax 2.8│
│ Vision (OCR): │ │ │ │ │ │ │ │ Speech HD) │
│ google/ │ │ │ │ │ │ │ │ │
│ gemini-3.8- │ │ │ │ │ │ │ │ │
│ flash │ │ │ │ │ │ │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
@@ -86,27 +87,28 @@ How do we architect the full-spectrum model selection across **all modalities**
## Modality Specifications & Cost-Benefit Analysis
### 1. Interactive Conductor & Agentic Reasoning (Tier 1)
* **Primary (Active Trial)**: `openrouter/z-ai/glm-5.3-flash`
- **Pricing**: \$0.075 / 1M prompt tokens; \$0.300 / 1M completion tokens. This is 50–75% more cost-effective than DeepSeek V4.1-Flash and offers a 1.3M token context window. Under daytime context headroom (~25k–60k tokens), daily cost is only **~\$0.01–\$0.02 / day**.
- **Performance & Reasoning**: SOTA Chinese & English reasoning, competitive coding benchmarks, and native vision perception.
- **Forensic Diagnosis & Tool-Calling Protocol Fix**:
- *Incident*: On September 14, 2026, GLM-5.3-Flash emitted raw `<tool_call>supermarket__supermarket_search...</tool_call>` XML into Telegram instead of executing tools.
- *Root Cause Identified*: In Zhipu AI's pre-training tokenizer, function calls are emitted as special XML tokens. When models are queried through OpenRouter's OpenAI-compatible API, OpenRouter only intercepts and translates `<tool_call>` tokens into JSON if the tool `name` exists in the request's registered `tools: [...]` array. OpenClaw uses dynamic Tool Search (`tools.toolSearch.mode: "directory"`) where only 3 control tools (`tool_search`, `tool_describe`, `tool_call`) are registered in `tools`. When GLM received deferred schemas (e.g., `supermarket__supermarket_search`), its pre-training prior prompted it to call the tool directly. Because `supermarket__supermarket_search` was not in OpenRouter's `tools` array, OpenRouter passed the raw XML through as message text. Once that text entered the session history (events SEQ 297 & 306), GLM was in-context primed to repeat the XML pattern.
- *Remediation & Verification*: We introduced an explicit deferred tool execution protocol into `AGENTS.md` instructing the model that deferred tools must NEVER be called directly and MUST be routed via `tool_call({ id: 'tool_name', args: { ... } })`. Direct OpenRouter API probes confirmed that with this rule, GLM-5.3-Flash calls `tool_call` with 100% compliant JSON and zero XML leakage. Session events were sanitized.
* **Primary (Fleet-Wide Standard as of 2026-10-17)**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
- **1017 Snapshot Upgrade (Pareto Jump in Quality & Price)**: Released on October 17, 2026, `deepseek/deepseek-v4.1-flash-1017` boosts the Artificial Analysis Intelligence Index from **24.7 to 39.5** (+60%) via Reinforcement Learning with Self-Feedback (RLSF) while cutting DeepInfra `fp8` pricing by an additional **30%** (`$0.098/M` prompt, `$0.294/M` completion, and **`$0.00294/M` `cacheRead`** vs. `$0.140/$0.420/$0.0042` on the 0910 baseline).
- **Provider Pinning & Tool-Compatibility Verification**:
- Pinned `order: ["deepinfra", "inference-net", "parasail", "cloudflare"]`, `quantizations: ["fp8", "bf16", "unknown"]`, and `ignore: ["morph", "baseten", "deepseek", "open-inference", "decart", "sail-research", "wafer"]`.
- Live OpenRouter routing forensics verified that `morph/fp8` is filtered out by OpenRouter's `Filter by Tool Compatibility` when `tools` are passed, `deepseek` (official) is filtered by account privacy guardrails (`Paid model training violation`), and `baseten` returns `HTTP 429`. Meanwhile, `deepinfra`, `inference-net`, `parasail`, and `cloudflare` deliver 100% tool-calling reliability and 2–5s latency.
- **Post-Trial Evaluation of `z-ai/glm-5.3-flash` (2026-09-13 to 2026-10-17)**:
- While `z-ai/glm-5.3-flash` was trialed as primary conductor, 17-day production telemetry (Oct 1–17, 2026) revealed **12 mid-stream tool-call aborts** (`stopReason: "error"`, `errorMessage: "Provider returned an incomplete or malformed tool call"`) across 5 agents (`main`, `grabengasse_media`, `koch`, `hebamme`, `doktor`) when generating nested `tool_call({ id, args })` payloads after `tool_describe`, as well as occasional thinking-only whitespace stalls (Issue 23) and long-context attention drift (Issues 22 & 25).
- Promoting `deepseek/deepseek-v4.1-flash-1017` to primary across all agents eliminates malformed tool-call stream aborts AND lowers total monthly cost by >50% compared to GLM 5.3 Flash.
- **Decoupled Vision Override**: `deepseek/deepseek-v4.1-flash-1017`, `deepseek/deepseek-v4.1-flash`, and `z-ai/glm-5.3-flash` are registered with `input: ["text"]` under `models.providers.openrouter.models` so that OpenClaw always routes user photos through `tools.media.image` (`google/gemini-3.8-flash`) for high-precision German OCR before handing enriched text to the chat model (Issue 29).
* **Fallbacks**:
1. `openrouter/deepseek/deepseek-v4.1-flash`: Rock-solid JSON tool-calling and sub-3s response times via dedicated OpenRouter ASIC cluster, available immediately if GLM experiences any upstream provider queuing or issues.
2. `openrouter/qwen/qwen3.8-flash`: Provider diversity.
1. `openrouter/deepseek/deepseek-v4.1-flash` (`0910` baseline across 25+ providers, pinned to `deepinfra`, `streamlake`, `inference-net`, `digitalocean`, `venice`, `nebius`).
2. `openrouter/z-ai/glm-5.3-flash`: Secondary fallback pinned to `fp8`/`bf16` providers (`morph`, `near-ai`, `digitalocean`, `venice`, `streamlake`).
3. `openrouter/qwen/qwen3.8-flash`: Provider diversity.
### 2. Autonomous Subagents (Tier 2)
* **General Subagents**: `openrouter/z-ai/glm-5.3-flash`
- **Execution**: Spawned via `sessions_spawn` with a 300-second timeout. Inherits the `tool_call` protocol guardrails from `AGENTS.md`.
* **Deep Analysis & IT Specialists (`itguy`, `recherche`)**: `openrouter/deepseek/deepseek-v4.1-flash`
- High-precision reasoning, robust nested tool loops without syntax drift, and immune to XML leakage on extensive research queries or CLI diagnostics.
* **Fallback**: `openrouter/deepseek/deepseek-v4.1-flash` / `openrouter/qwen/qwen3.8-flash`.
* **All Subagents & Specialists**: `openrouter/deepseek/deepseek-v4.1-flash-1017`
- **Execution**: Spawned via `sessions_spawn` with a 300-second timeout. Provides high-precision reasoning, rock-solid nested JSON tool loops (`tool_search` -> `tool_describe` -> `tool_call`), and zero syntax drift.
* **Fallback**: `openrouter/deepseek/deepseek-v4.1-flash` / `openrouter/z-ai/glm-5.3-flash` / `openrouter/qwen/qwen3.8-flash`.
### 3. Background Daemon & Heartbeats (Tier 3)
* **Configuration**: `heartbeat.model: "openrouter/z-ai/glm-5.3-flash"`, `heartbeat.target: none`, `heartbeat.lightContext: true`.
* **Rationale**: Decoupled completely from user conversation transcripts. Polling occurs against an isolated, minimal prompt (~400 tokens), dropping weekly heartbeat token burn from 21,000,000 to <150,000 tokens (<\$0.02 / week).
* **Configuration**: `heartbeat.model: "openrouter/deepseek/deepseek-v4.1-flash-1017"`, `heartbeat.target: none`, `heartbeat.lightContext: true`.
* **Rationale**: Because background heartbeats have a 75%+ `cacheRead` share (`22.86M` `cacheRead` vs. `7.73M` uncached input over 17 days), DeepSeek V4.1 Flash 1017's \$0.00294/M cache read rate on DeepInfra reduces 17-day heartbeat cost to <\$0.90 while keeping prompt cache warm on the primary provider.
### 4. Vision Input vs. Text-to-Image Generation (Tier 4)
* **Crucial Distinction**:
@@ -171,14 +173,83 @@ To align on remaining edge cases before final sign-off:
---
## OpenRouter Provider Qualification Criteria & Re-Evaluation Playbook
Whenever a new model snapshot (such as `deepseek/deepseek-v4.1-flash-1017`) is evaluated or re-checked as new hosting providers join OpenRouter, every candidate provider MUST pass the following **7-point qualification gate** before being added to `extra_body.provider.order` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml):
### 1. Hard Technical Gates (Must Pass All)
1. **Quantization Precision (`fp8` / `bf16` Only)**:
- **Rule**: Must run `fp8` or `bf16` (or verified full-precision `unknown` from trusted cloud hosts like Cloudflare).
- **Reject**: Any `fp4`, `nvfp4`, `mxfp4`, or `int4` quantization (`decart`, `sail-research`, `wafer`, `open-inference`, `phala` fp4). 4-bit quantizations severely degrade nested JSON tool-call syntax and German grammar.
2. **OpenRouter Routing Funnel & Guardrail Compatibility (`tools` + `tool_choice: "auto"` Pass)**:
- **Rule**: Must not be stripped by OpenRouter's internal routing filters (`Filter by Tool Compatibility` or `Filter by Guardrails`) when a request includes `tools: [...]` and `tool_choice: "auto"`.
- **Why**: Metadata in `/api/v1/models/.../endpoints` can claim `supports_tool_choice: true` even when OpenRouter's live router blocks the provider on actual tool calls (e.g. `morph/fp8` is stripped by `Filter by Tool Compatibility`, and `deepseek` official is stripped by `Paid model training violation` account privacy settings).
3. **Nested MCP `tool_call` Serialization Integrity (2/2 Live Pass)**:
- **Rule**: Must cleanly emit `finish_reason: "tool_calls"` with valid, parseable nested JSON arguments (`{"id": "mcp:...", "args": {...}}`) across back-to-back test runs without mid-stream SSE aborts (`Provider returned an incomplete or malformed tool call`) or HTTP `429` upstream rate limits.
4. **Context Window & Completion Headroom**:
- **Rule**: `context_length >= 262,144` (1,048,576 preferred) and `max_completion_tokens >= 131,072`.
- **Why**: Active daytime sessions with system prompts and tool schemas reach 20k–80k tokens.
5. **Uptime & Reliability**:
- **Rule**: `uptime_last_1d >= 95%` and `uptime_last_30m == 100%`.
### 2. Economic & Latency Ranking (How to Order Passing Providers)
Among providers passing all 5 technical gates, rank them in `extra_body.provider.order` by **effective production cost** and **turn latency**:
1. **Implicit Prompt Cache Read Rate (`input_cache_read`) — Highest Weight**:
- Because **73%–94%** of OpenClaw's token volume (both interactive multi-turn chats and 30m background heartbeats) consists of cached prompt reads (`cacheRead`), a provider with `supports_implicit_caching: true` and a **95%–98% cache discount** (`<= $0.005 / 1M` tokens, e.g. `deepinfra` at `$0.00294/M` or `streamlake` at `$0.0030/M`) is vastly cheaper in net monthly spend than a provider with slightly cheaper uncached input but weak or missing `cacheRead` discounts (e.g. `nebius` with `cache_read: null` or `modal`/`primeintellect` at `$0.030/M`).
2. **Base Prompt & Completion Pricing**:
- Target `prompt <= $0.18 / 1M` and `completion <= $0.65 / 1M` for top-priority slots; allow higher-priced providers (`<= $0.30 / $1.20`) only in lower fallback slots (#3/#4) for outage resilience.
3. **End-to-End Tool-Call Latency**:
- Target **1.0s – 5.5s** for a ~300-token reasoning + tool-call turn.
### 3. One-Shot Live Verification Command (For Future Checks)
Any agent (including a fast/lightweight model) can audit all currently available OpenRouter endpoints for `deepseek/deepseek-v4.1-flash-1017` in one step by running:
```bash
# 1. Inspect current endpoint metadata, pricing, cache_read, and 24h uptime:
curl -s https://openrouter.ai/api/v1/models/deepseek/deepseek-v4.1-flash-1017/endpoints | \
jq -r '.data.endpoints[] | "\(.provider_name) | tag=\(.tag) | quant=\(.quantization) | ctx=\(.context_length) | prompt=\(.pricing.prompt) | comp=\(.pricing.completion) | cache=\(.pricing.input_cache_read) | uptime1d=\(.uptime_last_1d)"'
# 2. Live-test candidate providers inside openclaw-0 with real nested tool_call payloads:
kubectl exec openclaw-0 -n openclaw -c openclaw -- python3 -c '
import os, json, time, urllib.request
api_key = os.environ["OPENROUTER_API_KEY"]
tools = [{"type": "function", "function": {"name": "tool_call", "parameters": {"type": "object", "properties": {"id": {"type": "string"}, "args": {"type": "object"}}, "required": ["id", "args"]}}}]
candidates = ["deepinfra", "streamlake", "inference-net", "venice", "digitalocean", "novita", "parasail", "cloudflare", "nebius"]
for prov in candidates:
req = urllib.request.Request(
"https://openrouter.ai/api/v1/chat/completions",
data=json.dumps({
"model": "deepseek/deepseek-v4.1-flash-1017",
"messages": [{"role": "user", "content": "Call tool_call with id=\"mcp:homeassistant:homeassistant__ha_get_state\" and args={\"entity_id\": \"media_player.bad\"}"}],
"tools": tools, "tool_choice": "auto", "max_tokens": 250,
"provider": {"order": [prov], "allow_fallbacks": False}
}).encode(),
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
)
t0 = time.time()
try:
with urllib.request.urlopen(req, timeout=20) as r:
b = json.loads(r.read().decode())
tc = b["choices"][0]["message"].get("tool_calls")
args = tc[0]["function"]["arguments"] if tc else "NONE"
print(f"PASS | {prov:15s} | {time.time()-t0:4.2f}s | actual={b.get(\"provider\")} | args={args}")
except urllib.error.HTTPError as e:
print(f"FAIL | {prov:15s} | HTTP {e.code} | {e.read().decode()[:140]}")
except Exception as e:
print(f"FAIL | {prov:15s} | ERR {e}")
'
```
---
## Consequences
### Positive
* **Unified Latest-Generation Stack**: Completely eliminates legacy Gemini 2.5 models; standardizes exclusively on Gemini 3.1, GLM 5.3, Whisper V3 Turbo, and Sonar Pro.
* **Unified Latest-Generation Stack**: Completely eliminates legacy Gemini 2.5 models; standardizes exclusively on DeepSeek V4.1 Flash 1017, Gemini 3.8/3.1, GLM 5.3, Whisper V3 Turbo, and Sonar Pro.
* **100% Reliable Image Generation**: Correctly routes image generation requests to Nano Banana 2 (`google/gemini-3.1-flash-image`), completely eliminating HTTP 404 generation errors.
* **Cohesive Voice & Audio Pipeline**: Clear architectural documentation of the Whisper V3 Turbo and Gemini 3.1 Flash TTS integration for dossiers and podcasts.
* **Guaranteed Cost Ceiling**: Drops weekly expenditure from **\$12.55 to ~\$1.20–\$1.80** (85–90% savings) through decoupled heartbeats and nightly compaction.
* **Guaranteed Cost Ceiling**: Drops weekly expenditure from **\$12.55 to ~\$0.80–\$1.40** (~90% savings) through DeepSeek V4.1 Flash 1017's `$0.00294/M` cache reads, decoupled heartbeats, and nightly compaction.
### Negative & Mitigations
* **OpenRouter Provider Variability**: GLM-5.3-Flash prefill throughput varies across OpenRouter hosting providers.
* *Mitigation*: Daily compaction bounds transcript size; failover to DeepSeek V4.1-Flash is available at zero architectural switching cost.
* **OpenRouter Provider Rollout Lag on Day-0 Model Releases**: Newly released snapshots initially launch with 5–10 providers before the full fleet (25+ providers) joins over subsequent days.
* *Mitigation*: Codified the 7-point provider qualification gate and one-shot test script above, and configured automatic model fallback to the `0910` snapshot (`openrouter/deepseek/deepseek-v4.1-flash`).