More models, more choice: Comparing 11 different AI models

Help

Elad Rosenheim Principal Product Manager

In this post

We just launched a partnership with OpenRouter that lets us offer two new pieces of functionality:

First, your projects can use any model on OpenRouter through our AI Gateway. That means that if your own web app offers AI inference-based features to your end users, you now have a much wider selection of models to fit any task and budget.

Second, we’re extending the selection of frontier coding models available for use via Agent Runners. Agent Runners is the chat prompt box you get within Netlify, which lets you build new projects from scratch or iterate on an existing one. The selection of models now includes much-hyped recent open models such as Kimi K3, GLM 5.2, and DeepSeek V4, available to everyone.

We call it Agent Runners because we run a full coding agent inside, not a pared-down one. Until now, we’ve supported Claude Agent, OpenAI Codex, and Gemini CLI which are optimized to run models from these providers.

We provide these agents with extra skills, and context about the current project, so that the agent will know exactly which Netlify capabilities are available for use (e.g., Netlify Database, the AI Gateway, or Identity), when to use them, and how. But to effectively drive a whole variety of new models, we’ve added the popular open-source OpenCode as a new choice of agent.

But with more choice come the inevitable questions: How do I know which model is right for me? Am I missing out on something that’s materially better, or more cost-effective (so I can do more with my credits), or is going to blow my mind like the internet says? There’s a lot of FOMO going around these days.

To provide you with some insights, here’s what we learned when running identical prompts across a range of models… all of which are now available for you to use today on Netlify.

You can see the results of all the models we tested on this site we created with the full report.

What we tested

Internally at Netlify, we use AXIS for automatically evaluating models, a tool that we’ve recently open-sourced.

We provide AXIS with a variety of test cases: prompts for building a new site and then iterating on it. We instruct AXIS on which agents and models to test these prompts, and define the checks that AXIS should then perform and score the generated site with.

These checks are very much focused on correct functionality of the generated site rather than its design, e.g.: does it use a database when a user’s needs call for it? Does it properly use Netlify Database in that case? In those cases where a simple static site will do, we also ensure that the generated site is not over-engineered, and no database is set up.

If a certain model is behind on its test scores, we don’t offer it in Agent Runners. If models too often fail at correctly applying one of our skills, or things do work but the credit cost seems inflated, then the problem is probably with the skill (in which case we optimize that skill).

But this time, we want to provide you with something much more immediately useful: when you go and build your dream using different models that each use wildly different amounts of credits, what do you get? What do the result look like?

We tested three relatively straightforward use-cases:

A site for a local coffee shop. LLMs just love making sites for local coffee shops! The initial prompt is simple, and a static site with no fancy database or the like will do. Then we do a follow-up prompt that asks for a simple option to reserve seats, and check how the model handled that.

A simple to-do list web app in which multiple users can view and add tasks. This calls for a simple design, but requires a shared database from the get-go. Then we ask to support an optional photo upload per item, and check if the model used the proper Netlify primitive.

A “What can I cook” web app that lets users enter what ingredients they have at home, and suggests a recipe using AI. The site itself is rather simple, but we want to check that the generated site correctly uses our AI Gateway to generate a recipe for the user.

For each of these cases, we’ll show you the look of the generated sites, comment on notable issues, and compare how many credits each took to generate. Of course, this is going to be a much more subjective test than our internal test suites, but it’s also going to be a very fun one. We’d love to know your opinion of the results!

All models were run with their default settings on Netlify. One notable mention is that we currently run GPT 5.6 Sol speicifically on low effort by default, giving you a more economical alternative to Opus that still provides pretty darn good results (as you’ll see below). However, the effort setting is now under your control, and our defaults may change with time.

This post is going to cover only the very first scenario: the static page for a coffee shop, while follow-up posts...