Why You Need to Test AI Brand Recommendations Beyond Broad Category Prompts

Asking an AI assistant to name five brands in your category is a fine place to start an audit, but it is a terrible place to stop—especially if your business is built for a specific type of buyer.

A brand completely missing from a generic category list might easily show up the moment a user adds a real-world use case to their prompt. Or, it might stay hidden until the prompt feeds the model unusually specific, textbook clues about the brand's identity. Those two scenarios tell very different stories about your visibility.

A recent study looking at repeated AI brand recommendations gives us a solid framework for pulling these scenarios apart. However, we have to keep one major boundary in mind from the start: the researchers ran all their tests with web search and other retrieval tools turned off.

Map out your competitors before you run the prompts

In their research, Edward Malthouse and his team tested six commercial models across five product categories using fresh, stateless API sessions. They ran each broad, category-only prompt 40 times per model.

Crucially, the researchers defined their competitor lists before looking at the AI's answers. If you are running your own audit, steal this habit. If you only track the brands the model decides to bring up on its own, a competitor that gets completely ignored will never even make it onto your spreadsheet.

Start with a justifiable list of competitors for your category and write down why each one belongs there. Then, track which ones the model names—and log the zeros for the ones it skips. Never let an AI's first answer dictate the boundaries of your market.

You also need to clearly label your testing environment. Answers from a bare model with search disabled only tell you how that specific setup behaves. They won't tell you what a live, personalized consumer app connected to the web is going to pull up and recommend.

Don't let a single metric hide how often you show up

The paper leans on two main metrics. The first is Brand Recommendation Probability at five, which is simply the percentage of generated lists where a brand cracks the top five.

The second is Mean Reciprocal Rank at five. This metric scores a brand 1 for first place, 1/2 for second, 1/3 for third, and so on, while handing out a zero whenever the brand is missing entirely. Because it averages these scores across all generated lists, it mashes together how often a brand shows up with where it ranks.

To see why this matters, imagine a brand that ranks first in one list and is completely missing from the next three. Its average reciprocal rank is 0.25. Now imagine a second brand that ranks fourth in all four lists. Its score is also 0.25. One brand has a 25% presence rate, while the other is at 100%, yet the combined score makes them look identical.

Note: BLURSOR previously referred to this metric as a brand's average position "when named." That was a mistake on our part—lists where a brand is absent still count as zeros in the average. Always report simple presence right alongside reciprocal rank so people reading your report can tell the difference between a rare #1 pick and a consistent #4.

Layer in real buyer needs before you panic over a zero

The researchers took a closer look at two brands that got completely shut out of the broad category prompts: Craftsman drills and L.L.Bean hiking jackets. When the team fed the models detailed prompts based on specific buyer needs—without naming the brands—Craftsman popped into the top five in 35.4% of its 240 lists, while L.L.Bean showed up in 5.4% of its own 240 lists. You can dig into the needs-based data in Section 4.3 of the paper.

That gap is telling. Adding realistic buyer context didn't rescue both brands equally.

When you set up your own audit, build prompts around the actual problems your business solves: the specific job to be done, tight constraints, budget limits, or tough operating environments. Stick to language a real customer would actually type, and keep these needs-based numbers separate from your broad category results.

If you write a prompt that quietly parrots your brand's unique marketing copy, you aren't testing normal customer visibility anymore. You are running a diagnostic probe that makes the guessing game much easier for the model.

Use insider clues to diagnose the model, not to pad your stats

When the researchers used those kinds of heavily revealing positioning probes, both Craftsman and L.L.Bean shot past 80% presence. That suggests the earlier omission wasn't simply an inability to produce the brand names under any condition.

What it doesn't prove is that normal shoppers will ever type those clues into a chat box. And it definitely doesn't mean that stuffing those exact phrases onto your website will boost your AI recommendations—remember, web retrieval was turned off for this study, and this specific diagnostic test only looked at two brands using a tiny set of handwritten prompts.

The smartest way to structure an audit is to keep three distinct buckets: broad category prompts, realistic buyer needs, and clearly labeled diagnostic probes. Putting them side by side helps you figure out why a brand is going missing. Just resist the urge to blend all three into one vanity metric that rewards you for writing prompts that spoon-feed the answer to the AI.

Finally, the study's early attempts to correlate these results with broader marketplace visibility don't give us a magic budget lever to pull. Instead, the best question to ask next is much simpler: is your brand missing from the specific buying moments it was built for, or is it just getting left out of vague prompts where the buyer hasn't explained what they actually need?