How I Verify What Actually Works in AI Search

Start with the decision, not the dashboard

Before I open a report, I write down the decision. Are we trying to repair crawler access, improve a source page, fix product data, understand a drop in referrals, or test whether an agent can finish a task? If the decision is vague, the report will become a pile of numbers looking for a story.

Then I choose one primary measure that can change that decision. Citation recurrence can help decide whether a source page deserves work. Qualified sessions and revenue answer a commercial question. Neither one replaces the other, and a blended score usually hides that distinction.

• Name the person who will act on the result.
• Write the decision date and the pages, market, and products in scope.
• Choose one primary measure and keep diagnostic measures clearly secondary.

Keep the measurement layers separate

I keep technical eligibility, answer exposure, citation or brand mention, referred visits, assisted conversions, and completed business outcomes separate. A page can improve at one layer and do absolutely nothing at the next. That is normal. It only becomes a problem when the report pretends the layers are one funnel with perfect tracking.

Official documentation tells us how a platform says a feature works. Property data tells us what the platform recorded for that site. A prompt test is one bounded observation. A business record tells us what completed. Any connection between those records is an interpretation unless the system exposes a direct link.

• Label platform statements, site data, observations, and interpretations separately.
• Keep the original dimensions and filters attached to exported data.
• Do not present correlation as a causal chain.

Save the environment behind every answer

Generated answers can change by product surface, time, location, language, device, account state, memory, personalization, model, and prompt wording. A screenshot without those conditions is decoration. It is not a reproducible record.

I save the exact prompt, timestamp and time zone, country, language, device class, signed-in state, and the product or model label visible in the interface. I also save the complete answer, the source URLs, and the specific claim each source appears to support.

• Keep the verbatim prompt and any declared paraphrases.
• Record destination URLs, not only source domains.
• Capture errors, omissions, and wrong brand facts with favorable mentions.

Repeat the test before you call it a pattern

One generated response is one observation. It is not a rank. I set the prompt panel, platforms, repeat count, collection dates, and controlled conditions before I look at the result. That makes it much harder to quietly discard the runs that ruin a good story.

For each prompt family, I report citation incidence, recurring URLs, factual accuracy, and the range across runs. I keep each engine and product surface separate because they expose different data and can use different retrieval and answer systems.

• Declare the sample before reviewing the result.
• Include failed runs and missing answers.
• Show the numerator and denominator behind every reported rate.

Connect visibility to the business without forcing the connection

AI visibility can influence discovery without producing an immediate click. It is worth tracking, but it is not the finish line. I keep referral sessions, engaged sessions, qualified actions, assisted conversions, completed sales, and revenue in separate columns.

When a referral tag exists, I connect the session to the landing page and next action. When it does not, I say so. Branded search, direct visits, and sales changes can add context. They do not automatically belong to an AI citation just because the dates look convenient.

For agent actions, verify the final receipt

Crawler access is only the first gate. A browser agent also has to understand the controls, preserve state, handle errors, request approval, and reach a clear final state. Reaching the checkout or filling a form is not completion.

For any action that matters, I confirm the result in the authoritative business system. That can be the order record, booking, sent form, published page, or payment receipt. It is the same rule I use everywhere else: the proof surface must belong to the outcome.

• Repeat the same meaningful task more than once.
• Capture the exact failure point and interface state.
• Check the final action outside the agent whenever a business record exists.