We are surprisingly close to prompt-to-model in computer vision

In computer vision, neural networks have become a commodity. You are a Google search away from downloading near-SOTA neural networks that are free, open source, have a permissive license, and are capable enough for most use cases.

Take Rekor, ZeroEyes, Intenseye, and Tiliter. I am not certain how their products are really implemented, but at least in theory, they could be replicated just by taking whatever recent YOLO flavour you like and training it for the specific task.

The only real MOAT these companies have, at least from the technical POV, is the dataset. These companies had to spend their capital on capturing real-world footage and photos and labelling this footage and these photos.

Depending on the specific task, this can be more or less capital intensive. For certain use cases, this can be easily and cheaply outsourced. At other times, it requires trained or specialized personnel to capture the footage or annotate it.

This moat is getting shallower. Thanks to diffusion models it is now possible to generate realistic images and use them to train the models. A few weeks ago I came across this paper which used a diffusion model to generate images of apples in an orchard, then trained some YOLO models on this dataset. I love it!<br>Still, you would need to annotate the images, but that can also be streamlined with a segmentation model, like SAM3. This was proposed in this paper.

At the end of the day this is just using highly capable models, which are (somewhat) heavy to run, to generate and label the data that small, efficient models learn from. I find this kind of knowledge transfer neat! I like the idea so much that I decided to check if this pipeline could be replicated on consumer hardware. The task I picked: detecting wrenches in real-world photos. This is arguably an easy task, wrenches are common enough that both image generation models and segmentation models can detect them. Solutions exist if the object you care about is not a common one.

The whole pipeline.

The hardware I used to generate the images and train the model. In 2026 this is by no means a beefy box.

I procedurally generated the prompts to include diverse context, background and lighting, then I used Flux Klein 9B to actually generate a total of just under 1000 images of wrenches.

Studio shot

Garage bench

Wrench set

Pile of wrenches

My machine is not beefy enough to load the text encoder (a Qwen3 LLM that turns the prompt into the embeddings the diffusion model conditions on) and the actual model at the same time. So I had to first load just the text encoder, generate the inputs for the model, and then load the actual diffusion model and stream the inputs to it. Even so, the 9B model is way too big for my 6GB of VRAM, so it spilled to the 16GB of RAM and a bit of the SSD swap. I ran the model without quantization. This configuration resulted in a generation time of around two minutes per image. Quite slow but still reasonable, ~33 hours for the whole training dataset.

Then I used SAM-3 to detect the wrenches in the images. SAM-3 works incredibly well and fits comfortably in my GPU.

Approved

Approved

Discarded, the red wrench escaped from area 51

Discarded, the gpu was so hot that it melted the wrenches :O

As you can see in the previous images, not all the generated photos were good enough. Another failure scenario is that SAM3 sometimes labeled random objects as wrenches or failed to label some of the wrenches. So I had to manually cherry-pick the images that contain good looking wrenches and are properly annotated by SAM3. I ended up with 515 approved images out of the ~1000 I reviewed.

I vibe coded a web interface to approve/discard the images and reviewing the whole dataset took around 15 minutes.

Before training, I applied albumentations (blurs, downscaling, sharpening) on the photos in an attempt to reduce the gap between artificial and real images. About a third of the training images contain no wrench at all: hard negatives with confuser objects (pliers, breaker bars, screwdrivers) and plain wrench-free scenes, some of them sampled from COCO. These keep the false positives in check.

To measure how much of this transfers to the real world, I also built an evaluation dataset by hand. I downloaded 137 real photos from the web and manually annotated them with bounding boxes (48 photos with wrenches, 30 hard negatives with wrench-like tools, 59 wrench-free scenes). I used them as the validation split during training, so they are not a strict holdout: they influenced checkpoint selection.

I used this setup to fine tune a YOLO 26n. After 200 epochs, evaluated on this validation set, the model scores:

precision: 0.92

recall: 0.71

mAP@0.5: 0.82

mAP@0.5:0.95: 0.64

Training curves. The validation split is made of real photos, so these numbers measure the synthetic-to-real gap directly.

And this is what the model (blue boxes) predicts on real...