# ADR 0002: OpenClaw Full-Spectrum Model Selection, Multimodal Tiering, and Cost Governance
## Context and Problem Statement
OpenClaw serves as the 24/7 personal AI concierge, family assistant, and operational orchestrator for Moritz Graf and his family. It interacts across Telegram, voice notes, webhooks, and background automations, bridging to personal knowledge (Obsidian wiki), smart home systems (Home Assistant, Sonos), fitness tracking (Garmin Connect), and cluster monitoring (Prometheus, Gitea).
During early September 2026, OpenClaw consumed **143.26 million tokens** over 7 days, accumulating **\$12.55+** in raw OpenRouter API costs. Forensic analysis revealed that over **80% of this spend was architectural waste**:
1.**Unbounded Context Re-transmission**: An uncompacted session (`agent:main:main`) reached 381k tokens (2,650 events, 5.7 MB JSON). Every minor turn re-ingested this entire history, costing \$12.20 (97.2% of total spend).
2.**Heartbeat Amplification Loop**: Background heartbeats firing every 30 minutes targeted the active user session (`target: last`), re-evaluating the 381k context 48 times/day simply to output `NO_REPLY` (\$5.16 wasted).
3.**Expensive & Deprecated Model Allocations**: Frontier models (`claude-3.7-sonnet` and `deepseek-v4-pro`) were configured without clear cost-benefit boundaries, with subagents frequently timing out (>300s). In addition, legacy model generations (e.g., Gemini 2.5) lingered in configuration despite superior, newer generations being available.
4.**Modality Conflation (Image Input vs. Image Generation)**: Vision perception (analyzing an uploaded photo) was conflated with text-to-image synthesis (drawing an image). Configuring chat models like `z-ai/glm-5.3-flash` for image synthesis caused runtime HTTP 404 failures because pure LLMs lack diffusion/raster generation heads.
Moritz established two foundational requirements for the repository:
> 1. *"I want high quality, but it should not be too expensive."*
> 2. *"We always want to use the newest models (as they usually are the best)."*
How do we architect the full-spectrum model selection across **all modalities** (text, subagents, heartbeats, vision, image generation, audio transcription, text-to-speech, web search, and vector embeddings) to deliver premium quality, sub-10s latency, and a total operating budget of **<\$2.00 / week**?
* **Conversational & Reasoning Quality**: High emotional intelligence, nuanced German fluency, personal context recall, and strict adherence to tool-calling protocols (ADR 0001).
* **Strict Modality Decoupling**: Each capability must route to its dedicated, best-in-class engine:
- Text & Agentic Reasoning
- Native Multimodal Vision (reading user photos)
- Text-to-Image Generation (synthesizing new pictures)
- **Pricing**: \$0.075 / 1M prompt tokens; \$0.300 / 1M completion tokens. This is 50–75% more cost-effective than DeepSeek V4.1-Flash and offers a 1.3M token context window. Under daytime context headroom (~25k–60k tokens), daily cost is only **~\$0.01–\$0.02 / day**.
- **Performance & Reasoning**: SOTA Chinese & English reasoning, competitive coding benchmarks, and native vision perception.
- *Incident*: On September 14, 2026, GLM-5.3-Flash emitted raw `<tool_call>supermarket__supermarket_search...</tool_call>` XML into Telegram instead of executing tools.
- *Root Cause Identified*: In Zhipu AI's pre-training tokenizer, function calls are emitted as special XML tokens. When models are queried through OpenRouter's OpenAI-compatible API, OpenRouter only intercepts and translates `<tool_call>` tokens into JSON if the tool `name` exists in the request's registered `tools: [...]` array. OpenClaw uses dynamic Tool Search (`tools.toolSearch.mode: "directory"`) where only 3 control tools (`tool_search`, `tool_describe`, `tool_call`) are registered in `tools`. When GLM received deferred schemas (e.g., `supermarket__supermarket_search`), its pre-training prior prompted it to call the tool directly. Because `supermarket__supermarket_search` was not in OpenRouter's `tools` array, OpenRouter passed the raw XML through as message text. Once that text entered the session history (events SEQ 297 & 306), GLM was in-context primed to repeat the XML pattern.
- *Remediation & Verification*: We introduced an explicit deferred tool execution protocol into `AGENTS.md` instructing the model that deferred tools must NEVER be called directly and MUST be routed via `tool_call({ id: 'tool_name', args: { ... } })`. Direct OpenRouter API probes confirmed that with this rule, GLM-5.3-Flash calls `tool_call` with 100% compliant JSON and zero XML leakage. Session events were sanitized.
* **Fallbacks**:
1.`openrouter/deepseek/deepseek-v4.1-flash`: Rock-solid JSON tool-calling and sub-3s response times via dedicated OpenRouter ASIC cluster, available immediately if GLM experiences any upstream provider queuing or issues.
* **Rationale**: Decoupled completely from user conversation transcripts. Polling occurs against an isolated, minimal prompt (~400 tokens), dropping weekly heartbeat token burn from 21,000,000 to <150,000 tokens (<\$0.02 / week).
### 4. Vision Input vs. Text-to-Image Generation (Tier 4)
* **Crucial Distinction**:
- **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled natively by `z-ai/glm-5.3-flash`, with `google/gemini-3.1-flash-image` as fallback.
- **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here.
- **Capabilities**: Fast, highly accurate transcription of German and English Telegram voice notes, with local gateway fallback.
* **Text-to-Speech (Audio Dossiers & Podcasts)**:
- **Engine**: `google/gemini-3.1-flash-tts-preview` via `text-to-audio` skill.
- **Voices**:
- German Solo/Host: `Puck` (warm, natural, dynamic).
- German Deep/Calm: `Charon`.
- German Co-Host / Expert: `Aoede`.
- English: `Zephyr`.
- **Fallback**: `minimax/speech-2.8-hd` for studio-grade multi-speaker podcasts when Google TTS is throttled.
### 6. Grounded Web Search & Vector Embeddings
* **Live Search**: `perplexity/sonar-pro` via OpenRouter (replacing raw web scraping with grounded citations).
* **Memory Embeddings**: `text-embedding-3-small` via OpenRouter for the Obsidian personal knowledge wiki.
---
## Context Lifecycle & Daytime Headroom Policy
To balance the user's desire to retain full conversation context across the entire day while avoiding the 70-second latency penalty that uncompacted transcripts impose on OpenRouter providers:
Executes in **2.1 seconds** using **0 LLM tokens**, ensuring every morning begins with a lean, fast ~15k token transcript.
---
## Open Clarifying Questions for Moritz
To align on remaining edge cases before final sign-off:
1. **Active Trial with GLM-5.3-Flash as Primary Conductor**:
- As requested, `z-ai/glm-5.3-flash` is now deployed as the Primary Conductor for interactive chat and autonomous subagents, equipped with the prompt guardrails against direct tool calls and XML leakage.
- `deepseek/deepseek-v4.1-flash` stands ready as the zero-latency primary fallback.
- We will observe real-world performance, tool adherence, and cost over the next several days.
2. **Thinking / Reasoning Mode on Telegram**:
- Currently, `thinking: medium` is enabled, emitting 100–300 reasoning tokens before generating conversational text.
- *Question*: Would you like to disable thinking mode (`thinking: none`) for standard Telegram chat turns (for near-instant replies) and keep reasoning active only for subagents?
3. **On-Demand Frontier Model**:
- With all standard workloads handled by Flash-tier models, do you ever want an explicit escape hatch to a frontier model (e.g. `claude-3.7-sonnet` or `gemini-3.1-pro-preview` triggered exclusively via `/expert <query>`)?
***Cohesive Voice & Audio Pipeline**: Clear architectural documentation of the Whisper V3 Turbo and Gemini 3.1 Flash TTS integration for dossiers and podcasts.
***Guaranteed Cost Ceiling**: Drops weekly expenditure from **\$12.55 to ~\$1.20–\$1.80** (85–90% savings) through decoupled heartbeats and nightly compaction.
@@ -34,6 +34,7 @@ To perform modifications or rotate api keys:
2.**Declarative Base Config**: Edit fields under `spec.config.raw` in [openclaw-instance.secret.yaml](file:///Users/moritz/src/infrapuzzle/k8s/openclaw/openclaw-instance.secret.yaml).
3.**Apply & Rollout**: Apply the files. The operator tracks config hashes and performs an automated rolling update of the StatefulSet automatically (no manual restart needed!):
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Moritz Graf