AI Search Integrity Study

Why this study

Our June study showed that AI recommendations change between identical asks. This study asked how much they move over two weeks of normal use, how far the five sources agree with each other, and whether Google's own results are stable enough to use as a baseline.

How it was run

28 unaided buyer questions across seven industries, each asked 3 times a day, 8 hours apart, from 21 October to 3 November 2026, to ChatGPT, Gemini, Perplexity, Claude and Google. 42 complete scheduled runs: 4,704 AI questions asked (28 questions, 4 engines, 42 runs), 4,679 answered. Two trial runs on 20 October are not included. Brands and cited websites were extracted from every answer. Intervals are bootstrap 95% intervals across runs. Some questions come from client projects, so results are reported by industry only and no client is named.

Finding 1: One check gives the wrong picture most of the time

In 76% of answers, the first brand an engine recommends is not the brand it recommends most often across all runs (95% interval 75% to 77%). The top three brands differ in 90% of answers.

Finding 2: A number needs many runs before it means anything

For brands that appear in at least 30% of answers, a reading from 3 runs can be off by about ±46 points, from 7 runs by ±31, from 14 runs by ±25, from 28 runs by ±17, and from all 42 runs by ±15. Only 2 of 230 such brand readings came within ±10 points, and only with all 42 runs.

Finding 3: Perplexity is the steadiest engine, Gemini the least steady

A single check names a different top brand 60% of the time on Perplexity, 79% on Claude, 80% on ChatGPT and 85% on Gemini.

Finding 4: The engines draw on different websites

For the same question, two engines share only 15% of the websites they use. ChatGPT names its sources in 45% of its answers; the other three always do.

Finding 5: Google's AI Overviews look beyond the top results

An AI Overview appeared on 77% of searches, and only 34% of the sites it cites are in Google's top 10 ordinary results.

Finding 6: Google results are not a fixed baseline either

31% of Google results were flagged as unusual (95% interval 28% to 34%, 797 results): fewer than half of the question's usual websites appeared. When a flagged result was looked up again about 7 minutes later, 53% were still flagged. 16% of Google lookups failed because the data supplier could not retrieve the page.

By industry

By buying stage, 'best X in a city' questions are the least stable (83% different top brand); comparison and cost questions are steadier (69% and 66%). Google figures per industry rest on 96 to 136 results each, so they carry margins of about ±8 to ±10 points.

What a real person sees (side test)

Five times between 28 October and 2 November, 15 searches were run in a normal browser on a Los Angeles home connection and through the study's collector at the same moment. On searches the study flagged, the two pages shared only about 15% of their websites; on never-flagged searches, about 33%. The person's own page also changed a lot between rounds (15% to 38% of the same websites), so much of the movement is Google changing results for everyone. This is one browser in one city with a few control searches per round: it points a direction, it is not proof.

What this means for marketers

• Do not judge AI visibility from one check. Track the same questions repeatedly and read the range, not the single number.
• Expect engines to disagree; measure each one separately.
• Treat Google's ordinary rankings and its AI Overview as two different surfaces.
• When a figure moves, ask whether it moved more than its normal variation before acting.