What outcome should I test for AI visibility?
Start by choosing one measurable outcome and writing the change you expect before collecting new observations. A useful outcome might be whether your brand is named, whether a page is cited, the cited position, or the share of tested answers containing the brand. Do not combine these into one score at the start, because a brand can gain mentions while losing citations or position.
Write a directional hypothesis such as, “The updated comparison page will increase the proportion of matched answers that name our brand.” Define the minimum change that would matter to the business separately from the statistical test. Statistical significance asks whether random variation is a plausible explanation. Practical significance asks whether the observed movement is worth the work or commercial attention.
Keep the question set stable while testing one intervention. A change to pricing information, page structure, or factual wording can be tested, but changing the prompt set at the same time makes the result difficult to interpret. Record the intervention date, affected URLs, target queries, intended audience, and primary outcome. This record becomes the boundary between a measurement exercise and a defensible experiment.
Build a fixed baseline across the seven engines
Create the baseline from the same buyer questions asked across ChatGPT, Perplexity, Gemini, Claude, Grok, Google AI Overviews and Google AI Mode. Use questions that represent the category, comparison, problem, and recommendation intents you actually want to influence. Preserve the exact wording, location settings, language, date, and other available context for every prompt.
The baseline should capture more than a yes or no result. Record whether the answer names the brand, cites a page, gives a position, names competitors, and uses the intended source. Save the answer text or an auditable extract, because a brand mention can be incidental while a citation can show that the engine used the changed page.
Do not quietly remove difficult prompts after the change. Removing prompts that remain negative creates survivorship bias. Do not add new prompts to the treatment period unless they are also added to the baseline and analyzed as a separate cohort. If answer volatility is high, collect more baseline observations over time rather than treating one scan as a stable starting point. The separate guidance on how often AI answers change can help set that observation schedule.
Pair each post-change observation with its baseline
Use the prompt and engine combination as the primary paired unit whenever possible. For example, the ChatGPT result for one exact buyer question before the change should be compared with that same question on ChatGPT after the change. Repeat the pairing for every engine and prompt, rather than treating all answers as interchangeable observations.
Pairing matters because different prompts have different difficulty levels and engines have different answer behavior. A simple average across all results can hide a loss on high-value questions behind gains on easy ones. For binary outcomes such as named versus not named, calculate the change within each pair and then summarize the paired changes. For position or citation outcomes, define how missing results are handled before analysis, since assigning an arbitrary position can distort the result.
If the prompt wording, engine, market, or language changes, treat the observation as a new stratum rather than forcing it into the original pair. A spreadsheet can handle a small, stable test if every row has an immutable prompt ID, engine, date, baseline result, post-change result, and change label. The most common measurement error is losing the pairing when results are copied into a summary dashboard.
Collect repeated observations without changing the test
Run the baseline and post-change measurements often enough to observe ordinary answer variation, while keeping the prompt set and collection conditions constant. One answer from one engine is evidence of what happened once, not strong evidence that a site change caused the movement. Repeated observations reveal whether the result persists across runs or disappears on the next collection.
Separate the intervention window from the measurement window. If a page was edited during the collection period, mark observations taken before and after the edit instead of treating the whole period as post-change. Record engine updates, indexing events, outages, unusual answer formats, and manual collection errors. A result that appears only during an engine incident should not be presented as a durable visibility gain.
Automation improves consistency, but it does not remove the need for quality checks. Review a sample of captured answers for truncation, duplicate responses, missing citations, and incorrect prompt routing. Cituna asks the seven named engines the questions a brand’s buyers ask every day and records names, citations, positions, competitors, and replacement pages. That kind of structured collection can reduce transcription risk, but the statistical design still depends on the prompt set, pairing, and intervention record.
Calculate the effect and its uncertainty
Calculate the observed change first, then calculate how uncertain that change is. For a naming outcome, report the proportion of matched observations that changed from not named to named, the proportion that moved in the opposite direction, and the net change. For citations, report gains and losses separately, because a net average can conceal turnover between pages.
Use confidence intervals or a paired resampling method to show the range of effects compatible with the observations. A paired bootstrap resamples prompt and engine pairs together, preserving the fact that each post-change result belongs to a particular baseline result. A paired permutation test can assess whether the direction of changes is stronger than expected if the intervention had no effect. The method should match the outcome type and be documented in plain language.
Do not calculate uncertainty as though every response from every engine were an independent draw from one identical population. Engines, prompts, and collection times are clustered sources of variation. At minimum, show results by engine and by prompt group alongside the overall result. If the data set is small or heavily unbalanced, label the estimate as provisional and avoid presenting a narrow interval as proof of precision.
Separate statistical significance from useful impact
Call a change statistically significant only when the prespecified test provides enough evidence against the no-change assumption under the chosen threshold. The threshold is a decision rule, not a guarantee that the intervention caused the result. A statistically significant movement can still be too small to justify more engineering, content work, or commercial attention.
Set a practical significance rule before reading the result. Examples include gaining citations for priority prompts, improving the median cited position without increasing misleading answers, or achieving a sustained gain in a defined engine and intent group. Keep the rule tied to the business decision, not to a convenient percentage selected after the outcome is visible.
Avoid running many unplanned tests and reporting only the favorable one. Testing naming, citations, position, seven engines, several intent groups, and multiple date windows creates many opportunities for a random positive result. State which outcome is primary, label other cuts as exploratory, and use a correction or stricter decision rule when many comparisons are unavoidable. Report the effect size, interval, sample definition, test method, and practical decision together. Leaders need to know both whether the movement is credible and whether it changes what the team should do.
Diagnose the result by engine, intent and failure mode
Break a significant or promising result into engine, prompt intent, citation status, and affected page before deciding what to change next. A broad gain may come from one engine or one easy query group, while a real business problem remains on recommendation prompts. Compare ChatGPT, Perplexity, Gemini, Claude, Grok, Google AI Overviews and Google AI Mode separately, then show the combined result with its weighting clearly stated.
Inspect reversals as well as gains. A page may become more visible for product questions but lose citations for comparisons after an edit. A competitor may appear because the answer changed its interpretation, because the source is unavailable, or because the changed page no longer supplies a clear fact. These are different fixes, and a single visibility score cannot identify them.
Use the answer text and cited URLs to classify the failure. Missing brand names suggest a discovery or relevance problem. Missing citations suggest a source selection or page clarity problem. Wrong facts suggest content governance or conflicting source material. Position changes may matter differently by intent. If a result differs by engine, do not average away the difference. Choose the fix that addresses the dominant failure mode, then define a new test rather than changing several page elements at once.
Choose a measurement method and make the next decision
Choose a spreadsheet for a small, stable test, a script or research workflow for repeatable custom analysis, or Cituna, which publishes this guide, when you want collection and visibility-fix work in one platform. Cituna asks ChatGPT, Perplexity, Gemini, Claude, Grok, Google AI Overviews and Google AI Mode about buyer questions every day, records names and citations, and generates fixes such as schema, FAQ markup, llms.txt and page changes. A manual approach may suit a narrowly scoped experiment, while a platform suits teams that need recurring scans and an operational path from gap to change.
Whichever method you choose, require an exportable observation table, stable prompt identifiers, engine labels, timestamps, answer evidence, and a documented test method. Treat automation as a way to reduce collection inconsistency, not as a substitute for a hypothesis or a significance calculation. Review the result with the person who owns the page change and the person who owns the business outcome.
Make one of three decisions: keep the change and monitor it, revise the change and run a new test, or stop because the observed impact is too small or unreliable. For executive communication, use the separate guide on how to report AI visibility changes to leaders. Link the result to the next action, not just to a score.
Related reading
Sources consulted
- Google Search Central (developers.google.com)
- OpenAI API documentation (platform.openai.com)
- Perplexity API documentation (docs.perplexity.ai)
- Google Search Console Help (support.google.com)
Drafted with AI assistance from our own research and Search Console data, and reviewed by Rahul A before publishing. Rules and prices change; check the linked official source before you act.