Methodology

Three commitments

  1. Intervals, not points. Every rate carries a 95% interval and its sample size.
  2. ≥7 runs per core prompt per day, pooled over 2–4 week windows, with the effective sample size reported per prompt cluster. Below that we label results “snapshot” or “insufficient” and draw no trend conclusions. Long-tail prompts run weekly and are only reported at window level.
  3. We disclose our own biases first.

How intervals are computed

Runs of the same prompt are correlated, so we use a two-stage (prompt-clustered) bootstrap: resample prompts with replacement, then resample runs within each chosen prompt; 2,000 resamples, fixed seed. The estimate is the equal-weight average over prompts. In simulation, treating every run as independent produced intervals about half as wide as they should be, covering the true value only 51–72% of the time instead of 95%. We also publish the intraclass correlation (ICC): when it is high, uncertainty comes from which questions were asked, and more prompts help more than more repeats.

Why our intervals are wider than “±5 points”

Some tools quote tight margins by treating every run as an independent sample. Runs of the same question are not independent, so those margins are too narrow — in our simulation they covered the true value only about half to three quarters of the time. Ours are wider because they are honest about where the uncertainty comes from: mostly from which questions were asked. When we compare you with a competitor we compare you on the same answers (paired differences), which cancels much of that shared noise.

Mentioned is not recommended

We report whether an answer names you, cites a page about you, and recommends you — separately, plus whether it describes you correctly against a fact sheet you sign. In our sandbox experiment, a large share of “mentions” of a new brand were the assistant telling people to wait for reviews.

Sample sufficiency

Trend-grade = ≥14 days of data and a median of ≥7 runs per prompt per day. Snapshot = ≥7 runs per prompt but fewer days: shows structure, not change. Below that we do not draw conclusions.

Stability metrics

Published baselines: day-to-day citation Jaccard 0.34–0.42; same-moment repeats 0.32–0.43. When the phrasing overlap is lower than the repeat overlap, “ask it differently and you get a different market” — the largest, most ignored source of risk.

Known biases (ours included)

What we will not do

Common questions, answered with this method: FAQ.