Why Rephrasing Buying Questions Changes AI Brand Recommendations

When evaluating how often an AI assistant recommends a brand, it is easy to conflate three distinct sources of variation: what happens when you run the exact same prompt twice, what happens when you rephrase the prompt without changing its meaning, and what happens when you add a new buyer constraint. Before trying to interpret a brand-visibility score, these three factors need to be separated. Each one answers a completely different diagnostic question.

Consider the prompts: “Which CRM would suit a small consultancy?” and “Recommend a CRM for a small consulting firm.” For all practical purposes, they express the exact same buying intent. However, appending “with data hosted in the EU” alters the underlying requirement.

A rigorous AI visibility audit cannot treat these variations as equivalent. Comparing the first two prompts tests sensitivity to phrasing while holding intent constant. Introducing the EU hosting requirement tests whether the model appropriately adapts its recommendations to a new customer constraint.

A recent analysis of AI commercial recommendations explored this dynamic by running repeated prompts and paraphrases across roughly 12,000 executions on OpenAI and Anthropic model configurations. The researchers observed notable variance even when repeating identical prompts, and substantially greater divergence when questions were reworded.

These findings highlight a practical question for anyone relying on visibility reports: how well does a fixed set of prompts actually represent broader buying intent? At the same time, it is important not to overinterpret the data. The study does not prove that website quality is irrelevant, nor does it mean that every fluctuation in a fixed-prompt benchmark is mere noise.

Establish a baseline for ordinary variation

To understand how much recommendations shift on their own, the researchers executed identical prompts multiple times within a single day. They measured the similarity of the resulting brand lists using Jaccard overlap, which ranged from 0.50 to 0.61 across the tested model configurations.

To extract brand names from the responses, the study used two AI judges. A brand was only counted as recommended if both judges agreed. Because this conservative extraction method can miss valid mentions, the reported figures specifically reflect the output of that filtering process.

It also helps to be precise about what Jaccard similarity actually measures. The metric divides the number of shared items by the total number of unique items across both sets—it is not the simple percentage of agreement between two answers. For instance, if two responses each recommend four brands and share three in common, there are five distinct brands in total. Three divided by five yields a Jaccard score of 0.60.

Crucially, Jaccard overlap only measures set similarity; it says nothing about whether the recommendations are accurate, well-reasoned, or appropriate for the buyer. Two diverging lists might both offer solid options, while a highly stable list could consistently recommend a poor fit.

In the context of this study, the identical-prompt results simply establish an internal baseline. They do not represent a universal ceiling on stability for all AI systems, and a single-day snapshot cannot tell us how recommendations drift over weeks or months.

Separate phrasing sensitivity from shifting constraints

When the researchers moved beyond identical prompts, they separated cosmetic paraphrases—like swapping synonyms or restructuring sentences—from edits that introduced or altered buyer constraints. The resulting Jaccard overlap dropped to roughly 0.288 for cosmetic rewrites and 0.135 when constraints were added.

The drop to 0.288 demonstrates that, within this experimental setup, minor wording tweaks can yield substantially different brand lists even when the underlying question remains the same. The 0.135 figure, however, requires a different interpretation: a new geographic region or technical requirement may legitimately call for different suppliers.

The study also looked at differences across model providers. Within the bounds of the experiment, point estimates indicated that rephrasing a prompt sometimes caused greater divergence than switching from one provider to another. While that is a striking observation for this specific dataset, it should not be extrapolated into a universal rule that prompt wording always outweighs model selection.

Furthermore, because the researchers did not experimentally manipulate the quality of the underlying web content, the study cannot tell us how prompt sensitivity compares to the impact of improving a brand's own documentation and site authority.

Balance panel stability with intent coverage

Using a fixed panel of prompts is entirely valid when your goal is longitudinal tracking. Holding the wording constant lets you compare how a specific question is answered over time. If you rewrite your test prompts every week, you confound the measurement.

The limitation of a fixed panel is that it only measures itself. If you want to understand how a brand performs across a broader buying intent, you need a secondary evaluation layer that tests plausible alternative phrasings—while keeping equivalent rewrites strictly separated from prompts that introduce new constraints.

A well-designed prompt log should document the core customer need, the exact phrasing, any explicit constraints, and the rationale for including the query. Whenever possible, these variations should be grounded in real customer research and actual sales inquiries, rather than synthetically generating dozens of near-duplicate sentences and assuming they represent real-world behavior.

This research does not establish a magic number of paraphrases required for a valid audit. The necessary breadth depends entirely on how diverse your customers' questions are and what business decisions the report is meant to inform. Simply adding more prompts achieves very little if they all cluster around the same narrow interpretation of buyer intent.

Contextualize score changes

When a brand's visibility score shifts, the first step is to verify whether the model version, search mode, prompt panel, or extraction logic changed. Once methodology changes are ruled out, compare the score movement against both ordinary rerun variance and the spread observed across equivalent phrasings.

Finally, always archive the full text of the responses alongside their cited sources, rather than just tallying brand mentions. A company might be named consistently while its actual product offerings are misrepresented. Conversely, a brand might vanish from a generic phrasing while remaining the top recommendation for the specific constraint that actually drives conversions.

Ultimately, effective evaluation requires distinguishing between stability on a single question and coverage across a broader intent. A tracking report can deliver the former without pretending to offer the latter. Requesting visibility into both makes it much easier to understand what the data can—and cannot—tell you.

Methodology note: The underlying study relied on single-day commercial prompts, an English-language dataset focused on US, UK, and EU markets, a predefined brand catalog, and provider-native web search (where retrieved context can vary between runs). The authors did not test content-optimization interventions or prescribe a universal sample size for prompt variants.