A developer built a system that watches brand visibility across the major AI-powered search tools—ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews. The project shows that a handful of API calls can’t tell a brand whether a user actually sees its name, because the model’s raw answer often differs from the polished result shown in the consumer app and the answers fluctuate from one query to the next.

Why brand visibility in AI search matters

AI chat assistants are becoming the default front-end for web discovery.

The two gaps that turned a simple scrape into an engineering marathon

1. The API is not the product. Calling the OpenAI API (or any other provider’s endpoint) returns the model’s internal answer. The ChatGPT app, however, adds its own retrieval system, system prompts and web-search augmentations before the user sees the final reply. The same happens with Claude, Gemini and the other services. An API call tells you what the model knows; the consumer app shows what the customer actually sees. Ignoring the extra layers means you’re measuring a different product.

2. Answers are not stable. Run the same prompt five times and you’ll likely get five different replies. Brand mentions shift, ordering changes, and source URLs move. A single API response is effectively a coin flip, not reliable data for a visibility report. One study cited in the development notes found that more than half of cited sources change from one day to the next.

What the 2026 APIs actually give you

  • Perplexity: The most builder-friendly. Its API returns citations in a straightforward format, but the citation list still diverges from what the Perplexity web app displays.
  • Google Gemini: Offers grounding through Vertex AI, yet the ranking and presentation logic in the consumer Gemini app differ from the raw API output.
  • OpenAI and Anthropic: Provide solid models with optional web-search tools, but they do not expose the exact ranking algorithm used in ChatGPT or Claude’s public interfaces.
  • Google AI Overviews: No public API exists. Tracking requires scraping the search result pages, a fragile and costly workaround.

Visibility is a rate, not a rank

Most existing tools reduce the problem to a binary “mentioned / not mentioned” check. That approach ignores the probabilistic nature of LLM outputs. Because each query can yield a different answer, you need a statistical view rather than a single snapshot.

A practical rule that emerged from the system’s design is to treat visibility as a rate: run the same prompt multiple times, across geographies and device types, then calculate a confidence interval. If a brand appears in most of the runs, you can report a high visibility rate with an appropriate margin of error, rather than a definitive “yes” or “no”.

How the monitoring tool was built

  • Multiple runs per prompt: At least seven to eight executions to smooth out randomness.
  • Geographic and device sampling: Queries issued from different regions and simulated on mobile versus desktop to capture platform-specific personalization.
  • Change-detection focus: Rather than a fixed schema that assumes static answers, the system flags shifts in mention frequency, source diversity and ranking position over time.

What to watch next

  • Platform transparency: Until providers publish the exact retrieval and ranking logic used in their consumer apps, third-party monitors will continue to rely on approximations.

Takeaway

Tracking brand presence in AI-driven search is not a matter of pulling a single API response. It requires repeated, geographically diverse queries, an understanding that the API is only a piece of the consumer product, and a statistical reporting model that captures visibility as a rate. Until AI search providers expose their full consumer-facing logic, anyone needing reliable brand-visibility data will have to keep building careful approximations.