Is Your AI Visibility Data Reliable?
AI responses are nondeterministic. A single run tells you almost nothing. Explore how prompt volume, run frequency, and model coverage affect the statistical confidence of your visibility data, and see the hidden cost of tracking this manually.
Why one AI run tells you almost nothing
Large language models are nondeterministic: ask ChatGPT, Gemini, or Claude the same question twice and the brands it recommends can change. Measuring AI visibility from a single run — or a handful of prompts — produces a number with a confidence interval wide enough to be meaningless. This calculator models how prompt volume, run frequency, and model coverage tighten that interval, using the binomial statistics behind a reliable share-of-voice estimate.
The math is unforgiving at small samples. Detecting a real shift in mention rate — the Minimum Detectable Change — requires enough independent observations to separate signal from run-to-run noise. The tool shows the sample size needed for a given confidence level, and the hidden hours of manual prompting that a scheduled, multi-model tracker replaces.
Key takeaways
- AI responses are nondeterministic — a single run's share of voice carries a confidence interval too wide to act on.
- Reliability scales with prompt volume × run frequency × model coverage, not with any one of them alone.
- The Minimum Detectable Change sets how big a shift you can even see at a given sample size — small samples hide real movement.
- Confidence comes from independent observations; a scheduled tracker across ChatGPT, Gemini, and Claude reaches it without manual prompting.
Worst-case uncertainty is at 50%. Adjust to match your observed data.
Prompts grouped by persona, buyer stage, JTBD, etc.
Each model is tracked independently. Selecting more multiplies total run count and manual effort.
At 10 prompts per brand tracked across 3 models, daily tracking gives you 300 monthly observations per model with a margin of error of ±4.7%. If a brand appears in 50% of responses, you can state with 90% confidence it's between 45.3% and 54.7%. You can detect real competitive shifts of 6.7% or more.
Every 3 days (±8.2%) is solid for monthly trending. You're building a meaningful picture over time even if individual snapshots are less precise. Weekly (±11.6%) is best used as an early-warning signal for large changes, not for tight competitive comparisons.
With 10 prompts, be cautious about reporting at the sub-cluster level. Each segment may only have 2-5 prompts, which is too few for reliable standalone conclusions. Consider reporting clusters in aggregate.
Margin of Error vs. Total Monthly Observations
How precisely can you state your brand's mention rate? The curve shows worst-case error. Vertical markers show where each frequency lands. Lower is better.
Minimum Detectable Change
What's the smallest real shift in brand mention rate you can reliably detect? This matters for competitive comparisons.
Margin of Error by Prompts & Frequency
Each cell shows ±margin of error at your selected confidence level and observed mention rate. Green = tight & reliable. Yellow = directional. Red = exploratory only.
| Frequency ↓ / Prompts → | 5 | 10 | 15 | 20 | 25 | 30 | 40 | 50 |
|---|---|---|---|---|---|---|---|---|
| Daily (30×/mo) | ±6.7% | ±4.7% | ±3.9% | ±3.4% | ±3.0% | ±2.7% | ±2.4% | ±2.1% |
| Every 3 Days (10×/mo) | ±11.6% | ±8.2% | ±6.7% | ±5.8% | ±5.2% | ±4.7% | ±4.1% | ±3.7% |
| Weekly (5×/mo) | ±16.4% | ±11.6% | ±9.5% | ±8.2% | ±7.4% | ±6.7% | ±5.8% | ±5.2% |
The Hidden Cost of Manual Tracking
Tracking AI visibility by hand means opening each model, entering each prompt, reading the response, recording whether the brand appears, and doing it again for every model you care about. We estimate ~2.5 minutes per prompt per model (opening the tool, typing the prompt, waiting for the response, recording the result). That's a conservative estimate.
Daily
Every 3 Days
Weekly
At 90% confidence with a 50% observed mention rate, you'd need 68 runs per prompt to stay within a ±10% margin of error. Across 3 models and 10 prompts, that's 2,040 total prompt runs — every month.85 hrs/month · 19.8 hrs/weekof manual work, just to collect the data — before any analysis, reporting, or action. That's not a workflow; that's a full-time job.
This math is why Spyglasses runs prompts on a daily schedule.
Daily Prompt Tracking →