How the AI Secret Shopper
actually tests your site.
We run a fixed protocol of prompts against ChatGPT, Claude, Perplexity, and Google’s AI tools — then we score what they see using a public 7-category rubric. Here’s exactly what we look at, what we measure, and what we don’t claim.
An AI screenshot isn’t proof. A protocol is.
AI tools are non-deterministic. Two people running the same prompt against the same model can get different answers — sometimes hours apart, sometimes from different geographic IPs, sometimes because the model version changed quietly behind the scenes.
That means a single “we asked ChatGPT” screenshot is interesting, but it isn’t evidence. The owner can’t reproduce it. A competitor can correctly call it cherry-picked. And nothing on the screenshot tells the owner what to actually fix.
Caidance’s posture is the opposite. We run a fixed protocol against named models on a dated cadence, and we publish the rubric that turns those raw outputs into a Caidance Discovery Index (CDI) band. Same prompts. Same scoring rules. Same disclosure of what changed.
If we can’t show our work, you shouldn’t trust the score. So we show it.
Seven categories. Two scores. One business.
Categories 1–6 roll up into your CDI (website intelligence — how AI tools see your front door). Category 7 rolls up into your Alignment Score (systems intelligence — whether the tools behind your website are organized enough for AI to help with daily work). Both scores use the same 0–60 scale and the same four bands: Starter, Builder, Operator, Leader.
Findability
Rolls up to CDICan an AI tool name your business in response to a real buying-intent query for your category, your location, or the specific problem you solve?
We run discovery prompts against ChatGPT, Claude, and Perplexity using the same category-and-location phrasing a buyer would actually type. We log whether your business appears, where in the response, and which sources the AI cites for the recommendation.
Clarity
Rolls up to CDIOnce an AI tool finds you, can it correctly state what you do, who you serve, and what you cost — without making something up?
We ask the AI to summarize your business in plain language and we compare its summary to your actual offering. Hallucinations, omissions, and confident-but-wrong claims all count against Clarity.
Trust Signals
Rolls up to CDIDoes the AI surface third-party validation — reviews, licensing, credentials, press, citations — before it recommends you?
We log every source the AI cites when describing your business. Strong Trust Signals look like a clean mix of your own pages, third-party reviews, professional registries, and authoritative press. Weak Trust Signals look like only your own marketing copy echoed back.
Recommendation Readiness
Rolls up to CDIWill the AI actually recommend you to a buyer — or will it hedge, list you alongside competitors with no preference, or quietly skip you?
Hedging is the silent killer most owners never see. The AI doesn’t criticize the business; it just declines to commit. We measure the difference between “here are three options including yours” and “for this need, here’s the one I’d call.”
Risk / Compliance Flags
Rolls up to AlignmentIs there anything on your site, in your reviews, or in your stated practices that would cause an AI tool to refuse to recommend you for safety, legal, or regulatory reasons?
For regulated industries (healthcare, legal, financial services, real estate, nonprofit, education, manufacturing) we run compliance-aware prompts that probe HIPAA, IOLTA, SEC Marketing Rule, FINRA 4511, Fair Housing, Schedule B donor privacy, FERPA, ITAR/EAR, and PCI DSS surface signals. Even one red flag can push an AI tool from “recommend” to “decline.”
Conversion Path
Rolls up to CDIOnce the AI recommends you, can a buyer (or an AI shopping agent acting on their behalf) actually take the next step — find your phone, book a meeting, get a quote, place an order?
We test the path from AI recommendation to next action. Broken contact info, hidden booking flows, forms behind logins, phone numbers a voice assistant can’t parse — all count against Conversion Path.
System Alignment
Rolls up to AlignmentBehind the front door: are your CRM, accounting, ecommerce, and operations tools organized enough for AI tools to help with daily work without breaking something?
A 21-question Alignment Assessment maps your declared stack against 154 stack-specific rules, 14 compound risk patterns (including AI-as-operator readiness), and industry-and-size calibrations across 10 regulatory regimes. The output is your Alignment Score (0–60) plus a prioritized fix list keyed to the actual tools you use.
The exact prompts we run, every cycle.
We run the same prompt set against each AI tool on every test cycle. Same wording, same model versions, same date stamps. When we change the prompts, we publish the version change. The current public prompt set is below.
Prompt set v1.0 publication in progress. The slots below are the categories we test. Final prompt wording is being locked in before public release so we can commit to it across cycles. Snapshots run today use the same internal protocol — what changes here is only the public-facing wording.
Purpose: Probes whether AI tools surface your business unprompted for a category-and-location buying query (the way a real buyer would ask).
Purpose: Asks the AI to summarize your business in plain language so we can score hallucinations, omissions, and confident-but-wrong claims.
Purpose: Asks the AI which third-party sources support its summary — reviews, licensing, registries, press — and logs the citation mix.
Purpose: For regulated industries: probes whether anything on your public surface would cause the AI to refuse to recommend you (HIPAA, IOLTA, SEC Marketing Rule, Fair Housing, etc.). Variant per industry.
Purpose: Forces the AI to commit: not “here are options,” but “who would you actually call?” Measures the difference between being listed and being recommended.
Vertical-specific prompt variants (for example, RIA-specific Compliance Red-Flag prompts that probe SEC Marketing Rule and FINRA 4511 surface signals) are documented on each /ai-secret-shopper-for-{industry}/ page.
When the same prompt gives two answers, we report both.
Each prompt is run more than once per cycle, across cleared sessions, to smooth out the single-run randomness AI models build in by design.
Every snapshot result names the specific model used (e.g., GPT-5, Claude Sonnet 4.6, Perplexity Sonar) and the date it was run. When a model version changes, we flag it.
Your snapshot result page shows the exact date and time of the test. That timestamp travels with the score — an AI snapshot from 60 days ago isn’t the same evidence as one from this week.
When two runs of the same prompt disagree, we flag both. We don’t silently pick the run that makes your site look better. Disagreement is often a more useful finding than agreement.
What this score doesn’t mean.
A diagnostic is only as honest as the things it refuses to claim. Here are ours.
AI tools change weekly. New models ship, old ones retire, defaults shift. Our rubric updates on a monthly cadence; between updates, the score reflects the world as it was on the test date.
A high CDI improves the probability an AI tool surfaces and recommends you when a buyer asks. It does not guarantee traffic, leads, or sales. Probability is what we measure; outcomes are what you operate.
Classic Google search still drives the majority of buying intent for most small businesses. CDI complements your SEO work — it doesn’t replace it. Anyone telling you AI search has “already won” is selling you something.
We publish ours so you can run them yourself and compare. Other testing frameworks are welcome — if anything, the category benefits when more people publish their methodology.
See what the AI Secret Shopper finds on your site.
60-second snapshot. No sign-up to view. Same protocol described on this page.
Run my AI Secret Shopper snapshot →