diff --git a/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md b/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md index b845c7d..5e302a4 100644 --- a/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md +++ b/docs/adr/0002-openclaw-model-selection-tiering-and-economics.md @@ -110,7 +110,7 @@ How do we architect the full-spectrum model selection across **all modalities** ### 4. Vision Input vs. Text-to-Image Generation (Tier 4) * **Crucial Distinction**: - - **Vision Input (`imageModel`)**: Reading photos sent to Telegram (e.g. food ingredients, plants, receipts). Handled by `google/gemini-3.8-flash`, with `z-ai/glm-5.3-flash` as fallback. + - **Vision Input (`imageModel` vs. Dedicated Primary)**: Reading photos sent to Telegram (e.g. food ingredients, nutrition tables, plants, receipts). Note: OpenClaw only invokes `imageModel` if the primary model *lacks* image input modality. Since `z-ai/glm-5.3-flash` advertises image modality on OpenRouter, OpenClaw does not auto-divert to `imageModel`. Therefore, vision-intensive agents (specifically `koch`) must explicitly configure `primary: "openrouter/google/gemini-3.8-flash"` to leverage Google's SOTA OCR and dense-table recognition. - **Text-to-Image Synthesis (`mediaModels.image`)**: Creating new pictures upon user request (e.g. *"Zeichne mir ein Profilbild von Octobot"*). Chat LLMs return 404 here. * **Image Synthesis Engine**: - **Primary**: `openrouter/google/gemini-3.1-flash-image` (Nano Banana 2). diff --git a/k8s/openclaw/TROUBLESHOOTING.md b/k8s/openclaw/TROUBLESHOOTING.md index a1e9144..80140b5 100644 --- a/k8s/openclaw/TROUBLESHOOTING.md +++ b/k8s/openclaw/TROUBLESHOOTING.md @@ -870,4 +870,38 @@ This document captures fixed issues, operational quirks, and incident root-cause - Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests. - Rely on deep-compaction with fallback rather than hard-resets across all topic agents. +--- + +### Issue 27: OpenClaw Vision Routing Bypass, GLM-5.3-Flash Image OCR Failure, and Tool-Calling Protocol Hardening + +- **Date**: 2026-10-12 +- **Severity**: High (Photos in Topic 21 #Kochen rejected as "blurry" despite 1080p high quality, turns halted with "The tool run finished, but no final summary was produced", tool errors in recherche/grabengasse) +- **Symptom**: + 1. The user posted high-resolution photos of food packaging and nutrition labels (e.g. IronMaxx Clear Whey Isolate Zero Kirsche, 1080x1064 px). + 2. Koch agent repeatedly responded that the photo was blurry and illegible. + 3. Internal chain-of-thought hallucinated that the label was a small blurry thumbnail with ~4px text reading "MIRKIC" instead of "KIRSCHE". + 4. The turn completed with `stopReason: "stop"` without emitting any visible text, causing OpenClaw to emit `The tool run finished, but no final summary was produced. I did not repeat any completed actions.` + 5. Ad-hoc Python image manipulation failed due to missing `PIL` (Pillow). + 6. Sporadic tool call errors across topics: `tool_search` wrapped in `tool_call`, `sessions_history` negative offsets, `ask_user` flat parameters, missing `googleapiclient` without `uv run`. +- **Root Cause**: + 1. **OpenClaw `imageModel` Fallback Behavior**: + OpenClaw's configuration schema defines `imageModel` as: *"Optional image model used when the primary model lacks image input."* + Because `z-ai/glm-5.3-flash` advertises `input_modalities: ["text", "image", "video"]` on OpenRouter, OpenClaw considered it image-capable and never routed images to `imageModel: "openrouter/google/gemini-3.8-flash"`. + GLM-5.3-Flash's actual vision perception on dense German labels is poor, leading it to misread text, conclude it cannot read it, and finish its thinking block without generating any output text. + 2. **Empty Turn Finalization Fallback**: + When a model finishes with only thinking tokens and zero output text or tool calls, OpenClaw's delivery mirror triggers `settled-finalization-fallback`. + 3. **Python Runtime Deficiencies**: + Core libraries (`pillow`, `google-api-python-client`, `pypdf`) were not installed in the persistent Python interpreter at `/home/openclaw/.openclaw/.local/python/...`. +- **Resolution**: + 1. **Dedicated Vision Model for Koch**: + Configured `koch`'s primary model to `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`) both live via `openclaw config set` and in `openclaw-instance.secret.yaml`. Gemini 3.8 Flash provides SOTA OCR and dense-table extraction at $0.075 / 1M cached prompt tokens. + 2. **Prompt-Hardening in `knowledge-wiki`**: + - Updated `knowledge-wiki/agents/koch.md` with strict rules: mandatory visible text output on every turn (no silent thinking stops), Vision-first protocol (transcribe all tables without claiming blurriness). + - Updated `knowledge-wiki/agents/README.md` with cross-agent tool-calling standards (`tool_search` top-level only, `sessions_history` non-negative offset + `sessionKey`, `ask_user` questions array, `uv run` for Python scripts). + 3. **Pre-installed Python Libraries**: + Installed `pillow`, `google-api-python-client`, and `pypdf` into the persistent Python environment using `uv pip install --break-system-packages`, and added the command to `openclaw-instance.secret.yaml` for permanent self-healing. +- **How to Avoid Recurrence**: + - For agents that routinely process user photos or document scans, assign dedicated vision models (`google/gemini-3.8-flash`) as the agent's primary model rather than relying on `agents.defaults.imageModel`. + - Maintain the response guarantee in agent prompts to prevent empty turns when reasoning models encounter uncertainty. + diff --git a/k8s/openclaw/openclaw-instance.secret.yaml b/k8s/openclaw/openclaw-instance.secret.yaml index c562e21..cb39e93 100644 Binary files a/k8s/openclaw/openclaw-instance.secret.yaml and b/k8s/openclaw/openclaw-instance.secret.yaml differ diff --git a/knowledge-wiki b/knowledge-wiki index e83bf3c..8bfdede 160000 --- a/knowledge-wiki +++ b/knowledge-wiki @@ -1 +1 @@ -Subproject commit e83bf3cbb197092b5c21188f277ba1d6d22b212c +Subproject commit 8bfdede7c8550756db9b4c7b0573a1dcdc1b8c36