moritz/infrapuzzle
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
4ae16b26402afb996f630d3e54248bb7d3af3c95
infrapuzzle/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md
T

16 KiB
Raw Blame History

status, date, decision-makers, consulted, informed
status date decision-makers consulted informed
proposed 2026-09-13
Moritz Graf
Antigravity

ADR 0002: OpenClaw Full-Spectrum Model Selection, Multimodal Tiering, and Cost Governance

Context and Problem Statement

OpenClaw serves as the 24/7 personal AI concierge, family assistant, and operational orchestrator for Moritz Graf and his family. It interacts across Telegram, voice notes, webhooks, and background automations, bridging to personal knowledge (Obsidian wiki), smart home systems (Home Assistant, Sonos), fitness tracking (Garmin Connect), and cluster monitoring (Prometheus, Gitea).

During early September 2026, OpenClaw consumed 143.26 million tokens over 7 days, accumulating $12.55+ in raw OpenRouter API costs. Forensic analysis revealed that over 80% of this spend was architectural waste:

  1. Unbounded Context Re-transmission: An uncompacted session (agent:main:main) reached 381k tokens (2,650 events, 5.7 MB JSON). Every minor turn re-ingested this entire history, costing $12.20 (97.2% of total spend).
  2. Heartbeat Amplification Loop: Background heartbeats firing every 30 minutes targeted the active user session (target: last), re-evaluating the 381k context 48 times/day simply to output NO_REPLY ($5.16 wasted).
  3. Expensive & Deprecated Model Allocations: Frontier models (claude-3.7-sonnet and deepseek-v4-pro) were configured without clear cost-benefit boundaries, with subagents frequently timing out (>300s). In addition, legacy model generations (e.g., Gemini 2.5) lingered in configuration despite superior, newer generations being available.
  4. Modality Conflation (Image Input vs. Image Generation): Vision perception (analyzing an uploaded photo) was conflated with text-to-image synthesis (drawing an image). Configuring chat models like z-ai/glm-5.3-flash for image synthesis caused runtime HTTP 404 failures because pure LLMs lack diffusion/raster generation heads.

Moritz established two foundational requirements for the repository:

  1. "I want high quality, but it should not be too expensive."
  2. "We always want to use the newest models (as they usually are the best)."

How do we architect the full-spectrum model selection across all modalities (text, subagents, heartbeats, vision, image generation, audio transcription, text-to-speech, web search, and vector embeddings) to deliver premium quality, sub-10s latency, and a total operating budget of <$2.00 / week?

Decision Drivers

  • State-of-the-Art Recency: Exclusively employ the latest generation models (e.g. Gemini 3.1, GLM 5.3, Whisper V3 Turbo, Sonar Pro). Completely decommission legacy generations (e.g. Gemini 2.5).
  • Conversational & Reasoning Quality: High emotional intelligence, nuanced German fluency, personal context recall, and strict adherence to tool-calling protocols (ADR 0001).
  • Strict Modality Decoupling: Each capability must route to its dedicated, best-in-class engine:
    • Text & Agentic Reasoning
    • Native Multimodal Vision (reading user photos)
    • Text-to-Image Generation (synthesizing new pictures)
    • Speech-to-Text / Audio Input (transcribing voice messages)
    • Text-to-Speech / Audio Output (synthesizing briefings and podcasts)
    • Web Search & Retrieval (grounded search)
    • Semantic Embeddings (vector memory)
  • Inference Economics: Low per-token costs for large conversational contexts (15k–80k tokens), zero background bleed, and predictable billing.
  • Latency & Turn Responsiveness: Conversational turns on Telegram must return within 4–10 seconds, avoiding provider queuing or massive prefill bottlenecks.
  • Resilience & Fault Tolerance: Automated fallback chains across independent providers on OpenRouter.

Full-Spectrum Model Architecture & Concrete Allocations

┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                       OpenClaw Ingress Gateway                                         │
└──────────────────────────────────────────────────┬─────────────────────────────────────────────────────┘
                                                   │
         ┌───────────────────┬─────────────────────┼─────────────────────┬───────────────────┐
         ▼                   ▼                     ▼                     ▼                   ▼
┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
│ Text Conductor  │ │ Subagent Engine │   │ Background &    │   │ Image Synthesis │ │ Voice & Audio   │
│ & Vision Input  │ │ (sessions_spawn)│   │ Heartbeat       │   │ (Text-to-Image) │ │ (STT & TTS)     │
├─────────────────┤ ├─────────────────┤   ├─────────────────┤   ├─────────────────┤ ├─────────────────┤
│ Tier 1          │ │ Tier 2          │   │ Tier 3          │   │ Tier 4          │ │ Tier 5          │
│ • Low Latency   │ │ • Deep Analysis │   │ • target: none  │   │ • Nano Banana 2 │ │ • Whisper V3    │
│ • Tool Routing  │ │ • Autonomous    │   │ • Zero Bleed    │   │ • Dedicated     │ │ • Gemini 3.1    │
│ • Native Vision │ │ • Isolated Run  │   │ • 30m Polling   │   │   Raster Head   │ │   Flash TTS     │
└────────┬────────┘ └────────┬────────┘   └────────┬────────┘   └────────┬────────┘ └────────┬────────┘
         │                   │                     │                     │                   │
         ▼                   ▼                     ▼                     ▼                   ▼
┌─────────────────┐ ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐ ┌─────────────────┐
│ Primary:        │ │ Primary:        │   │ Primary:        │   │ Primary:        │ │ STT:            │
│ deepseek/       │ │ deepseek/       │   │ z-ai/           │   │ google/         │ │ openai/         │
│ deepseek-v4.1-fl│ │ deepseek-v4.1-fl│   │ glm-5.3-flash   │   │ gemini-3.1-     │ │ whisper-large-  │
│                 │ │                 │   │                 │   │ flash-image     │ │ v3-turbo        │
│ Fallbacks:      │ │ Fallbacks:      │   │ Fallbacks:      │   │                 │                   │
│ 1. glm-5.3-flash│ │ 1. glm-5.3-flash│   │ 1. deepseek-    │   │ Fallbacks:      │ │ TTS:            │
│ 2. qwen3.8-flash│ │ 2. qwen3.8-flash│   │    v4.1-flash   │   │ 1. gemini-3.1-  │ │ google/         │
│                 │ │                 │   │ 2. qwen3.8-flash│   │    flash-preview│ │ gemini-3.1-     │
│ Vision Fallback:│ │                 │   │                 │   │ 2. gemini-3-pro-│ │ flash-tts       │
│ google/         │ │                 │   │                 │   │    image        │ │ (FB: MiniMax 2.8│
│ gemini-3.1-     │ │                 │   │                 │   │                 │ │  Speech HD)     │
│ flash-image     │ │                 │   │                 │   │                 │ │                 │
└─────────────────┘ └─────────────────┘   └─────────────────┘   └─────────────────┘ └─────────────────┘

Modality Specifications & Cost-Benefit Analysis

1. Interactive Conductor & Agentic Reasoning (Tier 1)

  • Primary: openrouter/deepseek/deepseek-v4.1-flash
    • Pricing: $0.150 / 1M prompt ($0.300 peak); $0.600 / 1M completion. Under our nightly compaction regime (15k tokens baseline), daily cost is only **$0.02–$0.04 / day**.
    • Tool-Calling Reliability: 100% native OpenAI JSON tool-calling compliance. Never leaks internal XML tokens into chat.
    • Latency: Sub-3-second TTFT and fast streaming via DeepSeek's dedicated OpenRouter ASIC cluster, completely immune to the 20–77s prefill queuing observed with third-party GLM providers.
  • Fallbacks:
    1. openrouter/z-ai/glm-5.3-flash: Fallback when DeepSeek is rate-limited or degraded.
    2. openrouter/qwen/qwen3.8-flash: Provider diversity.

Warning

Production Defect in GLM-5.3-Flash (Tool Call XML Leakage): In live production testing on September 14, 2026, z-ai/glm-5.3-flash exhibited a critical tool-calling failure in OpenClaw's Tool Search architecture. When executing multi-turn tool dispatches (e.g. searching supermarkets, Geizhals, eBay, or live gold prices), GLM-5.3-Flash bypassed the tool_call wrapper and leaked its internal Chinese pre-training syntax (<tool_call>name<arg_key>...</arg_key></tool_call>) directly into the Telegram text stream. OpenClaw treated this as conversational text, dispatching raw XML to the user instead of executing the tools. DeepSeek V4.1-Flash strictly adheres to JSON tool calls and resolves this completely.

2. Autonomous Subagents (Tier 2)

  • Primary: openrouter/deepseek/deepseek-v4.1-flash
    • Reliability & Execution: Consistently compiles complex JSON payloads for external systems (e.g. Garmin Connect workouts, multi-source price aggregation) without XML leakage.
    • Execution: Spawned via sessions_spawn with a 300-second timeout.
  • Fallback: openrouter/z-ai/glm-5.3-flash for non-tool reasoning tasks.

3. Background Daemon & Heartbeats (Tier 3)

  • Configuration: heartbeat.model: "openrouter/z-ai/glm-5.3-flash", heartbeat.target: none, heartbeat.lightContext: true.
  • Rationale: Decoupled completely from user conversation transcripts. Polling occurs against an isolated, minimal prompt (~400 tokens), dropping weekly heartbeat token burn from 21,000,000 to <150,000 tokens (<$0.02 / week).

4. Vision Input vs. Text-to-Image Generation (Tier 4)

  • Crucial Distinction:
    • Vision Input (imageModel): Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled natively by z-ai/glm-5.3-flash, with google/gemini-3.1-flash-image as fallback.
    • Text-to-Image Synthesis (mediaModels.image): Creating new pictures upon user request (e.g. "Zeichne mir ein Profilbild von Octobot"). Chat LLMs return 404 here.
  • Image Synthesis Engine:
    • Primary: openrouter/google/gemini-3.1-flash-image (Nano Banana 2).
      • Cost: $0.00006 per generated image + $0.50/1M prompt.
      • Latency: ~4.5 seconds.
    • Fallbacks:
      1. openrouter/google/gemini-3.1-flash-image-preview
      2. openrouter/google/gemini-3-pro-image (Nano Banana Pro)
    • Decommissioned: Gemini 2.5 is completely removed from all image configurations.

5. Voice, Audio & Podcast Synthesis (Tier 5)

  • Speech-to-Text (Audio Input):
    • Engine: openrouter/openai/whisper-large-v3-turbo
    • Capabilities: Fast, highly accurate transcription of German and English Telegram voice notes, with local gateway fallback.
  • Text-to-Speech (Audio Dossiers & Podcasts):
    • Engine: google/gemini-3.1-flash-tts-preview via text-to-audio skill.
    • Voices:
      • German Solo/Host: Puck (warm, natural, dynamic).
      • German Deep/Calm: Charon.
      • German Co-Host / Expert: Aoede.
      • English: Zephyr.
    • Fallback: minimax/speech-2.8-hd for studio-grade multi-speaker podcasts when Google TTS is throttled.

6. Grounded Web Search & Vector Embeddings

  • Live Search: perplexity/sonar-pro via OpenRouter (replacing raw web scraping with grounded citations).
  • Memory Embeddings: text-embedding-3-small via OpenRouter for the Obsidian personal knowledge wiki.

Context Lifecycle & Daytime Headroom Policy

To balance the user's desire to retain full conversation context across the entire day while avoiding the 70-second latency penalty that uncompacted transcripts impose on OpenRouter providers:

  1. Daytime Headroom (maxActiveTranscriptBytes: "3mb"): The active transcript is permitted to retain the day's back-and-forth interactions so OpenClaw recalls earlier topics and instructions naturally.
  2. Deterministic Nightly Compaction: Scheduled daily at 03:30 AM Europe/Berlin (ce39bbf8-b83f-453e-be53-04e27db27b78): 
    openclaw sessions compact "agent:main:main" --max-lines 250
    Executes in 2.1 seconds using 0 LLM tokens, ensuring every morning begins with a lean, fast ~15k token transcript.

Open Clarifying Questions for Moritz

To align on remaining edge cases before final sign-off:

  1. Conductor Allocation Confirmed:
    • Following the tool-calling XML leakage diagnosed on September 14, deepseek/deepseek-v4.1-flash has been promoted to Primary Conductor for interactive chat and autonomous subagents. Its dedicated ASIC cluster provides instant tool execution and 2–4s response times at ~$0.02–$0.04/day.
    • z-ai/glm-5.3-flash remains active for vision input and lightweight background heartbeats.
  2. Thinking / Reasoning Mode on Telegram:
    • Currently, thinking: medium is enabled, emitting 100–300 reasoning tokens before generating conversational text.
    • Question: Would you like to disable thinking mode (thinking: none) for standard Telegram chat turns (for near-instant replies) and keep reasoning active only for subagents?
  3. On-Demand Frontier Model:
    • With all standard workloads handled by Flash-tier models, do you ever want an explicit escape hatch to a frontier model (e.g. claude-3.7-sonnet or gemini-3.1-pro-preview triggered exclusively via /expert <query>)?

Consequences

Positive

  • Unified Latest-Generation Stack: Completely eliminates legacy Gemini 2.5 models; standardizes exclusively on Gemini 3.1, GLM 5.3, Whisper V3 Turbo, and Sonar Pro.
  • 100% Reliable Image Generation: Correctly routes image generation requests to Nano Banana 2 (google/gemini-3.1-flash-image), completely eliminating HTTP 404 generation errors.
  • Cohesive Voice & Audio Pipeline: Clear architectural documentation of the Whisper V3 Turbo and Gemini 3.1 Flash TTS integration for dossiers and podcasts.
  • Guaranteed Cost Ceiling: Drops weekly expenditure from $12.55 to ~$1.20–$1.80 (85–90% savings) through decoupled heartbeats and nightly compaction.

Negative & Mitigations

  • OpenRouter Provider Variability: GLM-5.3-Flash prefill throughput varies across OpenRouter hosting providers.
    • Mitigation: Daily compaction bounds transcript size; failover to DeepSeek V4.1-Flash is available at zero architectural switching cost.
Reference in New Issue View Git Blame Copy Permalink