I have been exploring whether businesses can reliably measure their visibility in AI-generated supplier recommendations.
A common testing method asks questions containing the company’s name. However, this mainly establishes whether the model can retrieve or describe that company when prompted. It does not establish whether the model would independently select it.
I developed a blind-first method that creates 20 realistic customer questions based on the business, industry and location, while excluding the business name from the initial recommendation prompts.
The same questions are tested independently across ChatGPT, Google Gemini and Anthropic Claude, producing 60 responses.
The analysis then separates:
• Supplier-selection and comparison answers
• General informational answers
• Business mentions and first-position recommendations
• Named competitors
• Differences between providers
Only answers in which the AI actually selects or compares suppliers are treated as recommendation opportunities.
I have implemented this approach in a platform called AI‑Tracker, but I am primarily interested in discussing the methodology here.
The difficult questions include model variability, geographical interpretation, prompt neutrality and how frequently testing must be repeated before trends become meaningful.
Do you think blind prompting produces a more useful measurement than brand-named prompting? What additional controls would you introduce?
Blind prompting provides a far more meaningful measurement of commercial visibility because it evaluates unprompted recommendation probability rather than simple parametric recall. Asking a model about a brand by name injects target tokens into its context window, forcing its attention mechanism toward that entity. Blind prompting accurately simulates real-world buyer discovery behavior, measuring whether an LLM independently selects your business as a solution.
Brand-named prompting is only useful for auditing entity accuracy, sentiment, and hallucinations, not market share or discoverability.
For controls you can consider a Multi-Run Stochastic Sampling, as LLM responses are non-deterministic. Running a prompt once yields a point-in-time snapshot, not a reliable trend. Run each of the 20 prompts 5 to 10 times per provider at a fixed temperature setting. This transforms binary results into a statistical Recommendation Rate (%) and establishes a confidence interval.
Thanks, this is an excellent distinction, particularly between measuring discovery and auditing brand/entity accuracy.
The current AI-Tracker scan is deliberately presented as a dated evidence snapshot rather than a stable ranking or statistically certain recommendation probability. The same 20 blind questions are tested across three providers, allowing provider differences to be measured, but each provider/question combination currently produces one response.
Repeated stochastic sampling would strengthen the measurement considerably. The practical trade-off is scale: five to ten runs would increase one report from 60 responses to 300–600, with corresponding processing time, cost and report complexity.
A repeated-sampling or research-grade mode is therefore something I’m considering. Monthly scans also provide longitudinal evidence, although they do not replace multiple runs taken under the same conditions.
I also agree that transparency is essential: question set, provider, sample size and scan date should remain visible, and the result should never be presented as a universal or permanent “AI ranking”.
Would you consider three runs per provider/question a meaningful first step, or do you think five is the minimum useful sample?
I’d lean toward three runs as a useful starting point, as long as the methodology stays consistent across providers and prompts. Five runs would provide stronger evidence, but even repeated smaller samples could help identify whether a recommendation is consistent or just a one-off response.
I also think there’s an interesting connection between AI visibility and the quality of the underlying business presence. For example, a service such as WPAegis focuses on WordPress security, backups, malware protection, updates, and performance. Those website-quality signals may indirectly affect how confidently a business is represented or discovered online.
It would be interesting to test whether improvements in website reliability and structured business information actually produce measurable changes in AI recommendation visibility over several months.