What AI Says About Your Business: We Ran the Test

Skip to content

Newsletter<br>Contact

Search

Featured, LLM

Marketing Intelligence

Do Language Models Flatten Your Business? We Tested Four Real Ones to Find Out

By<br>Yajush Gupta

July 27

14–22 minutes

Listen to this article

18 min read

Editor’s note: We decided the methodology of the test before we saw any result, so we couldn’t bend the method to fit a better story. Every AI answer in the results section is shown exactly as it came out, including the ones that made our neat theory messier. The academic research we cite is peer reviewed, and the stat that comes from a company selling a related product is flagged as such.

TL;DR

What AI says about your business is now something customers read before they buy. More and more of them ask ChatGPT, Gemini or Google’s AI to describe a company before they walk in or check out.

A popular claim in marketing circles says these tools sometimes sand off what makes a business distinctive. They instead show a generic version.We wanted to see what happens when AI describes a real business whose facts already exist online.

We built the Flattening Test with four real businesses and differentiators we verified from public sources first, to see if the distinctive fact survived in the AI responses.

We found that the tasty, visible, quotable stuff (a signature dish, a secret menu) survived about 83% of the time. The facts about how the business is run (family owned, never franchised, only two locations) survived zero times. One answer even made up details that were wrong.

The takeaway is that the danger is not that AI forgets you. It is that AI remembers your gimmick and forgets your backbone . There is a simple, free way to check your own business, below.

Table of Contents<br>TL;DR<br>The Echo Chamber Claim<br>The Evidence, and Its Edge<br>The businesses with the most to lose<br>The test: four businesses, three questions, one ruleOur scoring logic<br>Findings

What to do about it, in one afternoon<br>Where this test is weak<br>FAQ<br>Related reading

The Echo Chamber Claim

Think of a customer who has heard your name and wants a quick take before they commit. A few years ago they Googled you. Today, more and more of them open ChatGPT or Gemini and simply ask, "tell me about this place." Whatever the AI says back is now your introduction. What AI says about your business has quietly become your first impression, and you are not in the room, you did not write it, and you rarely ever see it.

The problem is that introduction comes out sanded down. The forty year decision to never franchise becomes "a beloved local institution." The kitchen built entirely around one bold idea becomes "quality seasonal ingredients." The single fact a happy customer would actually repeat to a friend goes missing, and what is left could describe any competitor on your street.

The reasoning behind the concern sounds convincing. Large language models work by predicting the most likely next few words, and the most likely words are the average ones, so the theory says any description drifts toward the middle, toward what a typical business in your category sounds like. A whole industry has grown up on top of that idea. Vendors sell audits of your "representation accuracy," meaning whether AI describes you correctly, and the sales pitch only works if the flattening is really happening.

So we went looking for the study that proves it. The biggest piece of research on how AI talks about brands, an analysis of more than 100,000 AI answers across over 100 brands, measured how often brands get mentioned and whether the tone was positive or negative. Genuinely useful, but it did not check the specifics or verifiable facts that make a business worth picking over others. Neither did anyone else we could find. So we tested it.

The Evidence, and Its Edge

To be fair to the theory, it stands on better ground than most marketing claims. One of the most reliably repeated findings about AI today is that it makes writing blander as a group, even while making each piece look better. In a study published in Science Advances, 293 people wrote short stories, some with AI help and some without. Judges rated the AI assisted stories as more creative and more enjoyable. Those same stories were also 10.7% more similar to one another. Better on their own, more alike as a set.

Another carefully controlled experiment presented at a major AI conference, ICLR 2024, found the same shape in argument essays. People writing with AI help produced work that was measurably more samey, and the sameness came from the AI’s contributions, not the humans’. Then came the largest test so far, covering 2,200 college admissions essays. Each new human written essay brought fresh ideas at two to eight times the rate of each AI written one, and that gap held up no matter how the researchers tried to coax the AI into being more original.

There is even an explanation for why. A paper accepted at another top AI...