Agentic AI Business Risk: 4 Real Cases

Skip to content

Newsletter<br>Contact

Search

Agentic AI, Featured

AI Tools & Workflows, Sales

When Your Chatbot Is So Eager to Help, It Forgot Whose Side It’s On

By<br>Yajush Gupta

August 1

9–14 minutes

Listen to this article

12 min read

It started with a bank telling customers how to avoid its own fees. It ends with a Chevy bot recommending a Ford.

TL;DR:

Companies are putting AI into the customer-facing layer of their product to be maximally helpful.

Sometimes AI is so good at its job that it points customers toward the cheaper, faster, or fee-free path, one that skips the workflow the company built its revenue on. This is Agentic AI Business Risk.

It’s already showing up in retail, travel, banking, and auto sales. The businesses handling it well decided, on purpose, which leaks they can live with.

Table of Contents

The Unspoken Launch Variable

A support chatbot has one job description on paper: to resolve your customer’s problem. Nobody writes “protect the commission structure” or “preserve the overdraft fee” into that job description, because it would sound bad in a press release. But those are the things that most businesses need the interaction to protect, and even a genuinely capable AI doesn’t know that unless someone tells it to care.

That’s the setup. A large language model, unlike a scripted bot, will reach for whatever answer best satisfies the question asked. If a customer asks a bank’s assistant how to avoid a fee, or asks a retailer’s assistant to find the cheapest version of a product, an AI trained to be helpful will often find it, even if the honest answer routes the customer away from the thing that pays for the assistant in the first place. It’s the same underlying behavior we found when we tested how language models describe real brands: the model optimizes for the best answer to the question in front of it, not for whoever is paying to host the conversation.

Bain & Company has a name for the far end of this: agents could “entirely disintermediate multibrand retailers and turn direct-to-consumer brands into indirect ones,” reducing some retailers to little more than commoditized fulfillment. This here is the boardroom-level version of the problem. Down at the level of a single chat window, it looks smaller and stranger, and it’s happening.

Four Real Cases

Amazon’s Rufus Sends Shoppers Off Amazon, on Purpose

Amazon’s in-app assistant, Rufus, is built to keep people inside Amazon’s checkout. But Amazon also built a feature called Buy for Me directly into it. If a shopper asks for something Amazon doesn’t sell, Rufus will find it on a brand’s own website and complete the purchase there, using the customer’s Amazon payment details, without the shopper ever leaving the app. It’s part of a broader shift toward agentic checkout, where the AI doesn’t just recommend a product, it completes the transaction.

The numbers around Rufus are pretty huge. Amazon says shoppers who use Rufus are more than 60% more likely to finish a purchase during that session, and the assistant is now credited with close to $12 billion in incremental annualized sales. And some of that money is flowing to brands that don’t sell on Amazon’s own marketplace at all.

It’s a strange choice for a company that built its empire on owning the transaction, until you consider the alternative: shoppers increasingly start their search in ChatGPT or Gemini instead of Amazon’s search bar, and a version of Amazon that says “we don’t have that” loses the shopper entirely. Amazon appears to have decided that losing a slice of margin on an off-platform sale beats losing the shopper’s attention altogether. That calculus also shows up in how these systems weigh signals that used to matter most to shoppers; we found something similar when we looked at why AI shopping agents barely react to star ratings the way a human buyer would.

However, merchants have complained to Modern Retail that Buy for Me listed their products without asking first, forcing them to opt out after the fact rather than opt in. Amazon designed its tool in order to keep shoppers engaged, and at the same time, training its own AI to treat the rest of the internet as its inventory.

Hotels Are Trying to Use AI to Bypass the Middleman, and So Is the Middleman

Online travel agencies built a business on being the place travelers search first, in exchange for a commission that industry sources put at 15% to 25% per booking. Hotels have wanted out of that arrangement for two decades and never had the leverage to make it stick. Conversational AI just joined the chat.

RateGain’s UNO Booking Engine was among the first to hook a hotel’s live rates directly into AI assistants through the Model Context Protocol, letting a traveler book a room inside the conversation itself, no OTA in the loop. Startups built entirely around this idea, pushing hotel rates straight into ChatGPT and Claude conversations, have already signed...