fix(openclaw): allocate Gemini 3.8 Flash to koch agent, harden vision and tool protocols
This commit is contained in:
@@ -870,4 +870,38 @@ This document captures fixed issues, operational quirks, and incident root-cause
|
||||
- Keep `keepRecentTokens: 3000` and `maxActiveTranscriptBytes: 180kb` in declarative manifests.
|
||||
- Rely on deep-compaction with fallback rather than hard-resets across all topic agents.
|
||||
|
||||
---
|
||||
|
||||
### Issue 27: OpenClaw Vision Routing Bypass, GLM-5.3-Flash Image OCR Failure, and Tool-Calling Protocol Hardening
|
||||
|
||||
- **Date**: 2026-10-12
|
||||
- **Severity**: High (Photos in Topic 21 #Kochen rejected as "blurry" despite 1080p high quality, turns halted with "The tool run finished, but no final summary was produced", tool errors in recherche/grabengasse)
|
||||
- **Symptom**:
|
||||
1. The user posted high-resolution photos of food packaging and nutrition labels (e.g. IronMaxx Clear Whey Isolate Zero Kirsche, 1080x1064 px).
|
||||
2. Koch agent repeatedly responded that the photo was blurry and illegible.
|
||||
3. Internal chain-of-thought hallucinated that the label was a small blurry thumbnail with ~4px text reading "MIRKIC" instead of "KIRSCHE".
|
||||
4. The turn completed with `stopReason: "stop"` without emitting any visible text, causing OpenClaw to emit `The tool run finished, but no final summary was produced. I did not repeat any completed actions.`
|
||||
5. Ad-hoc Python image manipulation failed due to missing `PIL` (Pillow).
|
||||
6. Sporadic tool call errors across topics: `tool_search` wrapped in `tool_call`, `sessions_history` negative offsets, `ask_user` flat parameters, missing `googleapiclient` without `uv run`.
|
||||
- **Root Cause**:
|
||||
1. **OpenClaw `imageModel` Fallback Behavior**:
|
||||
OpenClaw's configuration schema defines `imageModel` as: *"Optional image model used when the primary model lacks image input."*
|
||||
Because `z-ai/glm-5.3-flash` advertises `input_modalities: ["text", "image", "video"]` on OpenRouter, OpenClaw considered it image-capable and never routed images to `imageModel: "openrouter/google/gemini-3.8-flash"`.
|
||||
GLM-5.3-Flash's actual vision perception on dense German labels is poor, leading it to misread text, conclude it cannot read it, and finish its thinking block without generating any output text.
|
||||
2. **Empty Turn Finalization Fallback**:
|
||||
When a model finishes with only thinking tokens and zero output text or tool calls, OpenClaw's delivery mirror triggers `settled-finalization-fallback`.
|
||||
3. **Python Runtime Deficiencies**:
|
||||
Core libraries (`pillow`, `google-api-python-client`, `pypdf`) were not installed in the persistent Python interpreter at `/home/openclaw/.openclaw/.local/python/...`.
|
||||
- **Resolution**:
|
||||
1. **Dedicated Vision Model for Koch**:
|
||||
Configured `koch`'s primary model to `openrouter/google/gemini-3.8-flash` (fallbacks: `glm-5.3-flash`, `deepseek-v4.1-flash`) both live via `openclaw config set` and in `openclaw-instance.secret.yaml`. Gemini 3.8 Flash provides SOTA OCR and dense-table extraction at $0.075 / 1M cached prompt tokens.
|
||||
2. **Prompt-Hardening in `knowledge-wiki`**:
|
||||
- Updated `knowledge-wiki/agents/koch.md` with strict rules: mandatory visible text output on every turn (no silent thinking stops), Vision-first protocol (transcribe all tables without claiming blurriness).
|
||||
- Updated `knowledge-wiki/agents/README.md` with cross-agent tool-calling standards (`tool_search` top-level only, `sessions_history` non-negative offset + `sessionKey`, `ask_user` questions array, `uv run` for Python scripts).
|
||||
3. **Pre-installed Python Libraries**:
|
||||
Installed `pillow`, `google-api-python-client`, and `pypdf` into the persistent Python environment using `uv pip install --break-system-packages`, and added the command to `openclaw-instance.secret.yaml` for permanent self-healing.
|
||||
- **How to Avoid Recurrence**:
|
||||
- For agents that routinely process user photos or document scans, assign dedicated vision models (`google/gemini-3.8-flash`) as the agent's primary model rather than relying on `agents.defaults.imageModel`.
|
||||
- Maintain the response guarantee in agent prompts to prevent empty turns when reasoning models encounter uncertainty.
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
Reference in New Issue
Block a user