Most AI visibility audits test the wrong questions. How to run a buyer-first audit

The problem with free AI visibility audits#

Andy Crestodina of Orbit Media recently published We Audited 100 AI Visibility Audits. Here's What They Miss (October 21, 2026). He found a pattern that anyone who has used these tools will recognise. Most ask only for a website address. Many generate test questions by reading that website. Scores for the same site ranged from 28 to 100 depending on the tool.

His conclusion is hard to argue with: most of these tools check the plumbing of a website, then call it visibility. They start too far downstream, at the answer, and skip the beginning: the buyer.

We build an AI visibility tool, so this criticism applies to tools like ours too. Here is how we think an AI visibility audit should work, and how you can run one yourself, with or without software.

Why website-based questions miss the point#

Picture what many audit tools do:

1. Read your website.
2. Write questions based on what your website says.
3. Ask AI those questions.
4. Grade your website on the answers.

The flaw is circular. If the questions come from your own pages, the test mostly checks whether AI can repeat your page back. It cannot tell you whether your content matches what your buyers care about, because it never asked about your buyers.

It also produces a familiar symptom: lists of "competitors" you do not actually compete with, because the tool guessed your market from your words, not from your customers'. When AI does name a rival instead of you, the reasons are usually specific and fixable, as we explain in why AI recommends your competitor instead of you.

How real buyers talk to AI#

A search used to be three or four words. A conversation with ChatGPT is a story. For example:

We just hired a new sales leader who says our website is embarrassing. The CEO does not want a redesign, she wants better leads. Last time our branding agency did not understand analytics. Who should I talk to?

That buyer never typed "best B2B web design agency". They described a trigger (a new sales leader), a constraint (no redesign), a past failure (an agency that missed analytics) and a goal (leads). The AI builds a shortlist from all of it.

You cannot guess every version of that story. You do not need to. You need to know the few situations that start your buyers' search, and what they judge a provider on. Those are the inputs a good audit tests.

The buyer-first audit, in five steps#

Every step depends on the one before it.

1. Define two or three buyer types

For each one, write down:

• Who they are: role and type of company, such as "head of marketing at a 50-person SaaS company".
• What made them start looking: the trigger, in their words.
• How they choose: the criteria they judge providers on, such as proof of results, price model, speed or a specific skill.

2. Take the questions from your own data, not your website

The best source is what buyers already told you:

• Sales and discovery call notes or transcripts. Look for how they described the problem before they knew your name.
• Enquiry emails and contact form messages.
• Support tickets and chat logs for problems that lead to buying more.
• Lost-deal notes, which show who else was on the shortlist and why.

Copy their exact phrases. "Our website is embarrassing" tells you more than any keyword tool.

3. Write two kinds of test questions for each buyer type

• Story questions: two to four sentences in the buyer's voice, with their situation, a constraint or past bad experience, and a request for who to consider.
• Short questions: the quick version the same buyer might type, such as "best payroll software for a 40-person startup in India".

Never put your own company name in these questions. A question that names you will almost always mention you, which inflates the result.

4. Check that AI can read your website, separately

This is the plumbing, and it matters: can AI crawlers reach your pages, is the important content in the page itself rather than loaded later by scripts, and does structured data (schema) describe your company correctly? Our free Schema Check shows the second one in a minute. Score this on its own. A site can be perfectly readable and still never be recommended, and the other way round.

5. Ask, repeat, and report per buyer

Ask every question on ChatGPT, Gemini, Claude and Perplexity, more than once, because AI writes a new answer each time. Then report:

• How often you are named, per buyer type, as a share of answers.
• Who is named instead, and how often.
• Which sources AI cites when it answers.
• A range, not a single number, while the number of answers is small.

Keep every answer, word for word, with the date and the AI tool. If a result cannot be checked, it should not be trusted.

What a useful result looks like#

A buyer-first audit does not end in a score out of 100. It ends in a sentence you can act on, for example:

AI names us in about a third of answers when a marketing head describes a lead problem, but almost never when a founder asks about cost. Rival A is named first in both. Perplexity cites a review site we are not listed on.

Each part points to a different fix: content for one buyer type, pricing clarity for another, and a listing on the review site AI trusts.

How to run it without any tool#

You can do a first pass in an afternoon:

1. Pull ten recent call notes or enquiry emails and highlight the trigger and the criteria in each.
2. Group them into two or three buyer types.
3. Write two story questions and two short questions per buyer type.
4. Ask each question in ChatGPT, Gemini, Claude and Perplexity, ideally twice, in a fresh chat each time. Our step-by-step guide to checking ChatGPT shows how to do this without your history skewing the answers.
5. Record in a spreadsheet: buyer type, question, AI tool, named or not, who else was named, sources cited.

It is slow, but it is honest, and it will tell you more than any instant score.

How WebVisiblity does it#

We rebuilt our own product around this approach. In Live AI Monitoring, companies describe their buyer types or upload their sales notes, call transcripts or enquiry emails. We find the buyer types in those notes, write story and short questions in the buyers' own words, and label every question with where it came from. Each week the questions are asked across all four AI tools, and results are shown per buyer type, with every answer word for word and a button to check it yourself. Readiness (can AI read your site) is reported separately and never blended into the visibility number.

We are testing this with five B2B companies before launch. If your buyers research with AI, you can apply for a founding pilot spot.