The Best Way to Make Your (Small) Vision Language Model Smarter<br>← back to homepage<br>The Best Way to Make Your (Small) Vision Language Model Smarter

Vision language models (VLMs) are expensive. Open-source VLMs like Qwen are easy to run for free on consumer hardware, but are also notoriously unreliable.

What's the best way to boost the accuracy of open-source vision language models? Average the outputs over multiple calls? Use another VLM judge to verify the output of the original model? Give the model a zoom tool to better inspect the image?

To find out, I ran 13 open-source VLMs for over 500 GPU hours with different test-time compute methods.

TLDR: the best way to boost a VLM's accuracy is by using another VLM from a different model family to judge the original model's answers, but you're better off just using the biggest VLM you can fit on your GPU.

When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifiers, which studies the best ways to use LLM judges on text-only tasks. -->

Method

I tested models across 4 different families: Gemma at E2B, E4B and 12B, Qwen3-VL at 2B, 4B and 8B, InternVL3.5 at 1B, 2B, 4B, 8B and 14B, LLaVA-1.5 at 7B and 13B. Each model was run locally through vLLM on a single RTX 3090.

Benchmarks

To test both simple perception and visual reasoning, I evaluated every model on both CharXiv and CountBenchQA. CharXiv is a popular benchmark that tests the model's ability to logically interpret real-world scientific figures. CountBenchQA is a simple dataset where the model has to count the number of objects in an image. Despite the simplicity of the questions, smaller VLMs can surprisingly struggle at counting everyday objects.

CharXiv: "At epoch 60, which training method has a higher adversarial accuracy?"<br>Ground truth : Accelerated DAT

CountBench: "How many spoons are there in the image?"

Ground truth: 6

Test-Time-Compute Methods

The simplest method to increase the accuracy of VLMs is to run the same model N times and take the response that is generated most often (Maj@N). Similarly, you can ask the same question to multiple different models and take the most common answer across them for a cross-model majority vote.

Another popular technique for improving VLMs is to use a separate VLM to judge the original model's answers. Following Lu et al., I ran the VLM judge and the generation model in a loop where if the judge rejects an answer as incorrect, the generation model produces a fresh response to try again (with a maximum of 5 attempts). I tested the VLM judge under three different settings:

Self - model judges its own answers.

Cross - judge from a different family.

Intra - judge from the same family at a different size.

Finally, I also tried giving the model a tool to zoom into the image. There's surprisingly little information out there about how to run open-source models with this type of zoom tool and how much it changes the accuracy. To test this, I ran each model with a zoom tool multiple times with different maximum budgets of 2, 4 and 8 total crops and report the best accuracy. LLaVA was not trained to have multi-turn conversations with multiple images, so I did not test the zoom tool calls with it.

Results

Solver ModelBase<br>AccZoom<br>ToolAvg Cross<br>Family judgeMaj@5Qwen3-VL-2B-Instruct0.54+4%+5%+9% Qwen3-VL-4B-Instruct0.63-4%+2%+6% Qwen3-VL-8B-Instruct0.65-4%+2%+4% InternVL3.5-1B0.42-8%+9%+11% InternVL3.5-2B0.52+3%+5%+10% InternVL3.5-4B0.61-5%+3%+4% InternVL3.5-8B0.63-31%+2% +1%InternVL3.5-14B0.64-1%+2%+4% gemma-4-E2B-it0.44+8%+5%+8% gemma-4-E4B-it0.54+4%+2%+5% gemma-4-12B-it0.77+1%+0%+1% llava-1.5-7b-hf0.25NA+26% +6%llava-1.5-13b-hf0.27NA+26% +12%average0.53 &plusmn; 0.2-3 &plusmn; 10.4%+7 &plusmn; 8.8%+6 &plusmn; 3.6%

Relative change in accuracy over the base model, averaged across CharXiv and CountBenchQA.

Baseline - Raw Accuracy

In general across model sizes, Gemma is the best with Qwen and InternVL closely behind. I included LLaVA as a sanity baseline and because it is one of my favorite VLMs. However, its accuracy is... not great.

-->

Zoom Tool

Small models aren't good at tool usage. Qwen and InternVL frequently mess up the format of the tool call, and even when they do it correctly, they zoom in to irrelevant parts of the image. Gemma benefits the most from the zoom tool1, but the performance gains are minimal.

Click to see an example of the zoom tool<br>In this example, InternVL3_5-8B zooms in to the incorrect region in the image. Then it hallucinates that the crop shows the correct "hopper:stand" sublpot and confidently gives the wrong answer

Question: "For the subplot labeled hopper:stand, what is the maximum episode return approximately achieved at 10h?"

Model reasoning<br>Okay, let's take a closer look at the 'hopper:stand' subplot to determine the maximum episode return at around 10 hours. I'll start by examining the graph where the x-axis represents learner walltime and the y-axis shows episode return.

Let me zoom in on the...