I’ve asked the same question five times, and the cited sources move dimitrisdrakatos.com

AI Search excites me and while I’ve been spending a lot of my last year working in-house & clients on that, I decided to run a series of experiments to try to understand better how LLMs decide what to cite and mention. Here’s my first one: I asked four AI systems the same five questions, five times each, through their APIs with web search switched on. That is 100 answers. I wrote the questions before collecting anything and never changed them. What I wanted to know was simple: ask the same question twice, do you get the same sources back?

Key findings:
– Across the 20 question-and-provider combinations, the median overlap in cited domains between two repeats of the same question was 42.4%.
– 14 of those 20 sat at or below 50%.
– Topic mattered more than I expected. In one OpenAI setup, photo books came back at 20% overlap and email marketing at 100%. Same model, same settings, same afternoon.
– On that 100%: it is a median of ten comparisons, so at least half cited an identical set of domains, not all ten.

What it means:
If you are checking whether a brand shows up in an AI answer, one query is a sample of one. Check it more than once before you report it as a result.

What it does not tell you: anything about accuracy, source quality or how good the recommendations were, and nothing about how the consumer apps behave. This is cited domains, four provider APIs, one day.

What is next:
First of a series, with more to come. The evidence package is on the article page if you would rather check the numbers than take them from me, and there is a contact form if I have got something wrong.