Measuring and improving AI-generated UI design | Softlight

My girlfriend has never written a line of code, but woke up one morning and built a bot to secure hard-to-get Resy reservations. The bot worked great. Her first complaint: the UI looked bad. It just felt AI-generated.

She is not alone. Some people call it “AI slop.” Others say the models “lack taste.” Software creation has been democratized. But so far, it seems like good UI design hasn’t been.

I keep coming back to the same question: can I get my agents to do better UI design work, so I don’t have to?

This is a hard problem, and I can’t say I have solved it. But I made some progress. The following is what I learned through the process.

Measuring the quality of AI design

Most people treat design as subjective. I think it’s more measurable than that.

Good design helps users and businesses accomplish goals. If you understand what a feature means to accomplish, it is possible to judge the design.

Take the following example I worked on at Uber. At the time, Uber was trying to cross-sell riders into other services: Uber Eats, Grocery, Reserved Rides, etc. So, the company made a design change to the home page to emphasize those services more:

Is this a “good” design change? It really depends: do more users try Uber Eats? Do fewer users request a ride because they lose certainty from the cars on the map? Given the constraints of the user & business goals, it is possible to guess whether a design is “better”.

This was the first big challenge - how do I create a wide variety of AI-generated design changes where I understand the user & business context, so I can correctly judge the design output?

To do so, I took open source apps, and created new features I wanted to add.

Building a sample open source feature that solved a problem for me. This was for Twenty, an open source CRM.

I manually generated and then reviewed 100s of new features. I annotated the design issues I found and had an LLM summarize the issues. The vast majority of issues bucketed into the following:

Prioritization: Great designers prioritize ruthlessly in the UX. Only the core information is shown, the core path through the product is extremely apparent, and nothing else gets in a user’s way. The models seemed to think all user actions are equally important, and as a result users are left with clutter.

Typography, spacing, and color: The model overuses color and clutters the UI with no consistent spacing. This all links back to #1 - great designers use color, fonts, and spacing to draw attention to specific user actions. The model is not prioritizing which actions are important.

UI “bugs”: Simple things that a human designer would never mess up (alignment of components is off, text wrapping creates weird looking components, etc.)

Design system adherence: Introducing different components, button radii, fonts, etc. leads to an experience that does not feel unified.

Testing improvements to generate better designs

I experimented with 20+ different agent setups to improve the design outputs of coding agents. I then ran the same open source feature creation tests with those agent setups, and manually evaluated whether the design improved. This led to over 500 screens that needed to be reviewed, so I created an evaluation tool to quickly compare and score the outputs from different agents for the same design change.

What I found worked the best:

Self-improvement loops. A model can improve the design if it “sees” it. I found that showing the model the UI it generated, and asking it to fix issues led to meaningful gains. Not surprising, but was the biggest lever.

In the self-improvement loop, a rubric based on the common issues I flagged with AI design improved the output: Sharing the common flaws I identified when generating the features initially was the most helpful in the review rubric.

Injecting relevant examples of “good design” from well-known apps improves the output: I previously built a corpus of 500k+ screens from well designed apps (think Stripe, Airbnb, etc). Showing the model examples of those in the rubric further improved the outputs.

Some of the things that surprisingly didn’t work:

Creating a rubric based on “common design best practices”: This didn’t really help. My guess is the models already have these in their training data. The real gain came when I took the common flaws I noticed in AI design, and emphasized those in the rubric.

Negative examples of classic AI design slop signals (e.g. purple gradients, oversized rounded cards, emoji-bullet feature lists): Showing it the common patterns of AI design didn’t help much. It was more productive to show it images of “good design”, rather than the poor traditional signals of “AI design”.

Evaluation tool I built to quickly judge different...