I tried adding eyes and ears to DeepSeek V4 Flash, which doesn't support vision, using Qwen-MM-Plugins

Introduction

Hello, I'm Morishige from Classmethod's Manufacturing Business Technology Department.

Qwen has released Qwen-MM-Plugins — a collection of plugins that "make any agent harness multimodal."

https://github.com/QwenLM/Qwen-MM-Plugins

The DeepSeek V4 Flash-0731 I currently use day-to-day is fast and cheap, but it doesn't accept images. Sending a screenshot returns a 400. That said, switching the main model to a larger one just for images feels unappealing from a cost and speed perspective. When I saw this release in that context, I thought it might fill that gap — that's where this started.

I've written two articles about running this model on DGX Spark.

https://dev.classmethod.jp/articles/dgx-spark-deepseek-v4-flash-0731-llama-cpp/

https://dev.classmethod.jp/articles/dgx-spark-2node-deepseek-v4-flash-0731/

Incidentally, I've also been seeing an idea lately for filling the same gap on the model side. It involves placing a small VLM and shim in front of DeepSeek that converts images to text before passing them along, making it appear to the harness as a "DeepSeek that accepts images."

https://github.com/tonyd2wild/DeepSeek-v4-Flash-0731-Vision-DSpark-1M-NVFP4-KV-2x-DGX-Spark

If you're self-hosting DeepSeek itself, that's another viable option. The reason I chose the plugin approach this time is that it's designed without locking in the main model. Even if I switch to a different model later, the same configuration can be reused — and as I'll mention later, there's room to leverage it in combination with vision-capable models as well.

To state the conclusion upfront: the gap was filled. Images, audio, and video all pass through. However, the plugin's flagship "feed content directly to the model" tools don't work with text-only models — what was usable was the separate category. And eyes and ears on my local DGX Spark are sufficient.

What I'm using for eyes and ears is Nemotron 3 Nano Omni, which I previously ran on this blog. The story of running it standalone is summarized in an earlier article.

https://dev.classmethod.jp/articles/dgx-spark-nemotron3-nano-omni-multimodal-launch-bench/

In this article, I'll cover how to build a configuration where a text-only model remains the main actor while eyes and ears are attached externally, all the way to actually showing it a screenshot and having it fix a UI bug. I hope this resonates with people who want to use local LLMs in practice but are struggling with multimodal support.

Qwen-MM-Plugins Extends Harnesses with Skills and MCP

Qwen-MM-Plugins is a collection of plugins that extend the agent runtime, not the model itself. Released under Apache-2.0, the repository itself was created on July 29.

The structure is simple: each feature is distributed as a pair of a "skill" and an "MCP server." The skill tells the model "there's a tool like this," while the MCP server handles the actual processing.

Features are divided into 7 categories.

Claude Code, Codex, Qoder, OpenClaw, and Qwen Code can be installed via the plugin marketplace. opencode and Gemini CLI are configured by writing manually into a config file. Since I use opencode, I take the latter approach.

For this session, I'll be using core and omni-av.

Tools Are Split into Two Categories

Now to the main topic. What caught my attention while reading the documentation was how the tools were described. In the core skill, tools are written in two groups. Here's an excerpt:

Native reading (feeds content directly to you):
- See a file (PDF, Office, CSV, code, notebook, 3D, ...) → visualize
- Read an image with dynamic resolution → read_image
- Read a video (extract frames) → read_video

External API calls (DashScope):
- Call an external VLM about images/videos → vision_chat
- Extract text from an image → ocr
- Detect/locate objects in an image → grounding

It says "feeds content directly to you" — meaning it passes the content directly to you. Curious, I went to look at the implementation of read_image, and the last line was this:

return [text(summary), image(b64, mime)]

It's simply returning the resized image itself. In other words, the native reading category assumes that the model running inside the harness can read images. For a model that can't, it's like receiving a package it can't open.

On the other hand, vision_chat and ocr call a separate vision model via API and pass back the returned text. These can be used even with text-only models.

Let me verify this distinction in practice. When you send an image directly to DeepSeek V4 Flash-0731, this happens:

POST /v1/chat/completions  + image_url
→ 400 {"message":"This model does not support image inputs"}

A clear refusal. Adding the plugin doesn't change this fact. Only the external VLM call category works; the native reading category remains unusable.

What Happens with Native Reading for a Vision-Capable Model?

I also verified the other side just to be sure. I changed the router target to vision-capable minimax-m3 and had it call the same read_image. The instruction was "use only read_image."

⚙ read_image(image_path=…/ppe-sample.jpg, budget="normal")

- White helmet (safety helmet) — woman on the left
- Yellow helmet (safety helmet) — man on the right
- Yellow-green high-visibility safety vest (with reflective stickers) — woman on the left
- Dark navy work coveralls (jumpsuit) — man on the right
The man on the right is holding what appears to be a walkie-talkie, but this is a communication device, not protective equipment.
Safety shoes and gloves could not be confirmed from the image as the legs and hands are hidden by the wooden fence and clothing.

If the model can read images, it naturally works without issue. Moreover, for the same image, it provides more information than what you get via vision_chat later. Whether reflective stickers are present, the judgment that a walkie-talkie is not protective equipment, the disclaimer about parts that are hidden and can't be confirmed — all of this comes back.

The reason is clear: vision_chat works by "asking a question and getting an answer," so things you didn't ask about won't come back. With native reading, the image itself arrives in the model's hands, so the model can look at it as much as it wants by its own judgment.

In other words, native reading is not a useless feature — it's the legitimate approach for models that can read images. It's just that text-only models can't use it.

read_image also has a budget parameter for selecting resolution, allowing you to adjust how much detail is visible in exchange for token count. However, it doesn't appear to be universally effective — in my light testing, numeric labels on graphs of modest size were readable even at the smallest budget, but an 11px table in a 4K screenshot couldn't be read even with a higher budget. When I cropped it and tried again, the model confused columns and gave wrong answers, so it seems worth checking compatibility with your material.

Hearing that only half is usable might feel like a loss, but in practice it's not that much of a problem. If you "want to know what's in an image," it's sufficient to receive the answer as text. Plus, since images don't enter the context, the main model stays lightweight.

Placing Eyes and Ears on DGX Spark

So what do we put in the "external" part of the external VLM call category? Reading the documentation, it seems to assume using Alibaba's DashScope. However, reading the source, the top of shared/api_openai.py stated:

Targets any OpenAI-compatible endpoint (DashScope's compatible-mode is only the default base_url).

Any OpenAI-compatible endpoint works; DashScope is just the default. Furthermore, the API key resolution section has this note:

api_key falls back to "EMPTY" so local/self-hosted servers that ignore auth still work

It's designed with self-hosted servers in mind. The official documentation only describes DASHSCOPE_BASE_URL as "for proxies and gateways," so this is something you wouldn't notice without reading the code.

So, I'll set up eyes and ears on DGX Spark. I'll use the FP8 version of Nemotron 3 Nano Omni. Since it can handle images, audio, video, and text in a single model, I don't need to set up eyes and ears separately.

uv venv --python 3.12 && source .venv/bin/activate
uv pip install "vllm[audio]==0.20.0"

vllm serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --served-model-name nemotron-omni qwen3.7-plus qwen3.5-omni-plus \
  --host 0.0.0.0 --port 8000 \
  --gpu-memory-utilization 0.7 \
  --max-model-len 131072 \
  --media-io-kwargs '{"video":{"num_frames":256,"fps":2}}' \
  --video-pruning-rate 0.5 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_v3 \
  --trust-remote-code

Adding Aliases in --served-model-name

The plugins decide on the model names they call themselves. vision_chat and ocr use qwen3.7-plus, and omni-av series use qwen3.5-omni-plus. Since there's no way to change these names via environment variables, you'd get a 404 as-is.

So I add name aliases on the vLLM side. Since --served-model-name accepts multiple values, listing the names the plugins want to call means the actual model can be Nemotron but it will still be accepted. This way, the plugin side requires no modifications and no additional arguments.

Set --max-model-len Generously

The other issue is context length. omni-av sends requests with max_tokens defaulting to 65536, so if the server's --max-model-len is smaller than that, you'll get a 400 error saying max_tokens=65536 cannot be greater than max_model_len. This has nothing to do with the size of the file you sent. Setting it to 131072, with room for the input as well, lets video summaries pass through.

Testing All Three Modalities

Only two environment variables need to be set.

DASHSCOPE_BASE_URL=http://<dgx-spark>:8000/v1
DASHSCOPE_API_KEY=EMPTY

In this state, calling tools via the MCP server worked for images, audio, and video respectively.

For video, the return looks like this:

## Storyline

00:00.000 - 00:04.000
The video opens with a high-angle aerial shot sweeping over a steep, grassy mountainside …

Audio Required a One-Line Patch

Only audio's omni_asr returned a 400 out of the box, so I checked what the plugin was sending and manually tried 3 patterns.

Raw base64 works. The cause is a mismatch in wrapping: the plugin wraps audio in data: format following DashScope documentation, but according to the OpenAI specification, input_audio.data is just the base64 string itself. vLLM implements the spec correctly, so when passed in wrapped form, it fails on decoding. In other words, vLLM is correct per the spec, and the plugin is aligned with DashScope's proprietary extension.

The fix is effectively one line that removes the wrapping only for self-hosted setups. After applying the patch locally, transcription came back in 11.0 seconds. This fix has been proposed upstream as issue #8 and PR #9.

Putting It Behind a Router to Make Destinations Switchable

At this point, the MCP server was directly hitting DGX Spark. It was working, but the destination was hardcoded in the plugin's environment variables.

In my environment, normal model calls go through a router called NeMo Switchyard. It's a mechanism for routing between cheaper and smarter models depending on the use case — I also wrote an article about this previously.

https://dev.classmethod.jp/articles/nvidia-nemo-switchyard-first-touch/

Since I have a router, putting eyes and ears there too consolidates destination switching and logging in one place. I tried it.

[llm_clients.local_omni]
format = "openai_chat"
base_url = "http://<dgx-spark>:8000/v1"
api_key_env = "FIREWORKS_API_KEY"

[targets.omni]
id = "nemotron-omni"
llm_client = "local_omni"

[routes.omni-vision]
id = "qwen3.7-plus"
type = "passthrough"
target = "omni"

[routes.omni-av]
id = "qwen3.5-omni-plus"
type = "passthrough"
target = "omni"

As a small trick, I set the route IDs to the exact model names the plugins want to call. This is the same thing I did earlier with vLLM's --served-model-name, but now done on the router side. With this in place, the plugin side only needs to point DASHSCOPE_BASE_URL at the router.

Here's the overall picture of what's been assembled so far.

Two paths diverge at the router. The blue line on the left goes to the text-only brain; the orange on the right goes to the eyes and ears that receive media. Media never reaches the brain — only the text returned by the eyes and ears arrives there.

I had two concerns. Would the router pass through requests containing media unchanged, and if so, how much latency would it add?

The first was not a problem. Images, audio, and video all pass through transparently. Since the router runs in a Docker container on a separate machine from DGX Spark, network connectivity between them was confirmed at the same time.

For the second, I measured alternating 3 times each between direct calls and router-routed calls.

Even for a video request containing 7.6MB of base64, the increase is only 0.03 seconds. The first time I measured, it looked over a second slower — but that was just variance. I'm glad I measured several times.

Logging also worked as expected, with media calls appearing in the router log with token counts.

Incidentally, the tier column was empty for these rows. Since the route is a passthrough, it means the classifier that determines which model to route to was not invoked. This is desirable — having images and audio flow into the classifier would incur unnecessary costs.

Disabling Reasoning Made It 8x Faster

With the configuration assembled, I measured with actual tasks. I prepared two tasks where correctness could be verified mechanically. One was identifying protective equipment from a construction site photo; the other was reading four values from a graph. Since the graph has numbers printed on it, there's no ambiguity — either you could read them or you couldn't. With no room for fudging, it's well-suited for testing vision quality.

Both tasks scored perfect accuracy. However, latency was 16–19 seconds, which is too long to wait when using it as eyes. The culprit was the long reasoning Nemotron-series models emit before answers by default. Constraining max_tokens sometimes caused the reasoning alone to exhaust the budget, leaving the answer empty. In a previous blog post handling a different Nemotron, I worked around this by disabling reasoning with chat_template_kwargs.

However, the plugin doesn't send that parameter. It's not in the tool arguments either. I thought I'd have to patch the plugin side — but then I remembered the router was in the middle. Adding one line to the target is all it takes.

[targets.omni]
id = "nemotron-omni"
llm_client = "local_omni"
extra_body = { chat_template_kwargs = { enable_thinking = false } }

This applies uniformly to all calls. Here are the re-measured results.

Median accuracy remained perfect. Strictly speaking, in 1 out of 3 runs without reasoning, the vest was missed. No changes were made to the plugin.

Having the router in place was originally about being able to switch destinations, but it also served as a layer for injecting model-specific behaviors from the outside. Personally, this was the most pleasant discovery of this whole exercise.

Can Cloud-Based Eyes Serve as a Substitute?

Here a natural question arises: is it actually necessary to keep the eyes locally? I have DGX Spark for myself alone, but distributing it to a team doesn't mean everyone can have one on their desk.

For my team's environment, I use open-weight models via Fireworks. I also wrote an article about that configuration.

https://dev.classmethod.jp/articles/open-weight-team-ai-environment/

If Fireworks has vision-capable models, all I'd need to do is swap the route destination. I tried it.

The swap itself was 3 lines. Add one target and change the route destination. Nothing touched in MCP settings or plugins. It works as intended.

The challenge was figuring out which model accepts what. I exhaustively tried every model available in my account, and here are the results.

Not a single model accepted audio. Three pass for images.

The reason I put △ for minimax-m3 on video is that the error was different from the others. While other models returned errors indicating no video support, this one said "pass video as an HTTP URL; base64 data URLs are not supported." It's just saying it's a delivery method issue, so when I retried with a publicly accessible URL, it read it fine. However, since the plugin converts local files to base64 before sending, this doesn't fit that path. Making it work would require placing the video somewhere externally accessible — which isn't realistic for internal footage.

What About Hosted Versions of the Same Omni?

We're not limited to Fireworks. The very Nemotron 3 Nano Omni I'm using for eyes is hosted on OpenRouter and NVIDIA's NIM API.

https://openrouter.ai/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free

Since both can be tested by just changing the route destination, I hit all three modalities in the format the plugin sends.

In addition to images, video passes through as data URLs. This is the first time a hosted option has appeared that can read local video files. It's the same model, so that makes sense — but it fills a gap that had gone completely unmet in the Fireworks exhaustive test.

Audio didn't work on either hosted option. And not just that — rather than an error, both returned 200 with a response saying "please upload an audio file." The usage.prompt_tokens only reflected the text portion, meaning the audio part was silently stripped in the middle. Since no error is raised, this is easy to miss, so it's worth keeping in mind.

Free-tier constraints also require attention. NIM returned 503 due to concurrency limits during my testing, and OpenRouter's :free has request count limits. For team-wide regular use, paid tiers seem to be a prerequisite.

I also compared speeds. The median values from the same 2 tasks in the reasoning chapter, measured with only the route destination changed.

A single DGX Spark running a 30B model beat serverless cloud on latency. Accuracy was perfect across all arms with no differences. As long as it's being used as eyes, there's no apparent disadvantage to keeping it local.

From here, where to place it depends on how widely you're distributing it.

Regarding PDFs, read_image from the plugin can't open them. However, when I actually tried it, the model found its own workaround. It used save_view to export one page as PNG and then passed it to ocr. If the PDF has a text layer, pdftotext is faster and more accurate, so image conversion would be needed for scanned documents and cases where figures themselves are the answer.

Having It View a Screenshot and Fix CSS

Up to here, the discussion was about "whether the eyes can see." Finally, let me test whether it can complete a full loop as an agent.

The subject is a common flex bug. A card has an input field and a submit button laid out side by side, but because the input field has min-width: 280px, the button overflows the card.

With a card width of 360px, the combined minimum width of the input field and button plus gap exceeds the card, causing the blue "登録する" button to break through the right edge.

I prepared a small shell script that invokes headless Chrome to take screenshots, and had the model take them itself. The instruction was just this:

The appearance of index.html is broken. Take a screenshot with ./shot.sh,
identify the cause, and fix the CSS. After fixing, take another screenshot
and visually confirm whether it's fixed.
Since you can't read images yourself, use the available tools to check.

The loop I wanted it to follow looks like this:

All four steps are executed by the same model. The first-pass verification uses vision_chat; the final judgment uses coordinate comparison via grounding.

It started moving. First it took a screenshot, then sent a question to vision_chat.

⚙ vision_chat(images=[shot.png],
   text="Is the blue '登録する' button overflowing past the right edge of the white card?
         Answer YES or NO, then briefly describe the position of the relevant area in one line.")

Based on the answer, it drilled down to the cause.

I've identified the cause.
- What's broken: The blue "登録する" button is overflowing past the right edge of the white card
- Cause: .field has flex: 1; min-width: 280px; set. The card's content width is
  360px - 24px×2 = 312px, but min-width 280px + gap 12px + button (nowrap, ~90px)
  requires ~382px, causing the flex row to exceed the card width and overflow to the right.

The numbers are correct. The fix was the one-liner min-width: 0, the standard solution for flex items that won't shrink.

What followed was interesting. It took another screenshot and sent a verification to vision_chat, but upon seeing the answer, it said:

VLM verification is unstable, so I'll verify rigorously with coordinates.

⚙ grounding(image_path=shot.png, prompt="the blue button with text 登録する")
⚙ grounding(image_path=shot.png, prompt="the white card container")

Verification complete with coordinates (normalized coordinates 0-1000):
- Card right edge: x=655
- Button range: x=567〜639
Button right edge 639 < 655, so it's neatly contained within the card. Fix complete.

Rather than asking "is it contained?" in natural language, it switched to getting coordinates via object detection and judging with an inequality. I didn't instruct it to do this.

After changing to min-width: 0. The input field shrinks and the button fits within the card's border.

It's actually fixed.

Throughout this loop, the main model never once received an image. All it received was the text returned by vision_chat and the coordinates returned by grounding. A model that returns 400 when you send it an image directly ended up fixing a UI bug.

Incidentally, I also did an A/B comparison of whether installing the official plugin skill made a difference. Testing 13 times each, having the skill installed raised the rate of choosing grounding from 15% to 46%. Since the skill says "for detecting or locating objects, use grounding," the "please list" instruction seems to have been pulled that direction. I also tried rewriting the skill myself, but didn't see notable improvements. If you want to stabilize tool selection, naming the tool explicitly in your prompt seems more reliable than tweaking the skill wording.

Summary

This was about adding eyes and ears to a text-only model. The result: images, audio, and video all pass through. However, the approach differs from what the plugin officially envisions — the flagship native reading tools remain unusable. What works is only the category that calls an external vision model and returns text.

As for the core question of "can we run 0731 as the main model in practice," my assessment based on what I measured here is promising. Vision responses with reasoning disabled come back in 1–2 seconds, faster than serverless cloud, without degraded accuracy, and the router adds only a few percent of overhead. The brain stays cheap and fast, and the loop of showing a screenshot and fixing CSS closed in one pass. The key advantage of this configuration is that since observations come back as short text, the slow-long-generation weakness of local 30B models doesn't surface. That said, since accuracy was perfect across the board because the tasks were easy, I intend to continue evaluating quality on complex tables and nuanced judgment calls as I actually use it.

Three adjustments were needed, two of which only required server startup options. Setting --max-model-len generously, and listing the names the plugins want to call in --served-model-name. The remaining one was the base64 wrapping of audio, which required a one-line patch.

What changed my thinking after doing this is that the configuration isn't just "a workaround for vision-unsupported models." Keep the main model locked to something cheap and fast, and delegate only modalities to specialized models. Since what the eyes return is text and coordinates, the main model's context isn't burdened with images. No longer needing a large multimodal model at the center — thinking of it that way, there seem to be quite a few applications.

On Qwen-MM-Plugins itself: reading the internals, there's nothing that's a new invention. Both skill and MCP are existing mechanisms, and vision_chat is essentially just a short piece of code that calls an OpenAI-compatible API. The value lies more in the packaging — organizing over 30 tools into 7 features and making them distributable across multiple harnesses. And that straightforwardness paid off directly this time. If it had been locked behind proprietary protocols, pointing the destination to local and placing it under a router would both have been impossible.

Next, I'd also like to test how far video-memory is usable when this configuration is pointed at long videos.

Reference Links

• QwenLM/Qwen-MM-Plugins — Apache-2.0. Distributed as pairs of skills and MCP servers
• Qwen-MM-Plugins Installation Guide — Configuration examples for harnesses requiring manual registration such as opencode / Gemini CLI
• nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8
• Nemotron 3 Nano Omni (free) on OpenRouter — Hosted. Images and video pass through, but audio is silently dropped (see main text)
• Nemotron 3 Nano Omni on NIM API — Provides the same model via OpenAI-compatible API at integrate.api.nvidia.com
• Run Highly Efficient Multimodal Agentic AI with NVIDIA Nemotron 3 Nano Omni Using vLLM — Explanation from the vLLM side. DGX Spark is also listed as a supported target
• Fireworks Video & audio inputs guide — Models supporting audio and video require dedicated deployment
• Running NVIDIA Nemotron 3 Nano Omni on DGX Spark (Article from 2026-04-29)
• Running 284B DeepSeek V4 Flash-0731 on Two DGX Spark Units
• Trying Out NVIDIA's New LLM Routing Infrastructure NeMo Switchyard
• Team AI Environment Covering Everything from Development to Business Use with Open-Weight Models