What AI search actually needs from your website

Last Modified: November 22, 2026

Share:

Facebook
Twitter
LinkedIn
Email
Facebook
Twitter
LinkedIn
Email
AI assistants now answer customer questions directly. Here are the five things they need from your website in order to quote you, explained clearly.

For twenty years the job was to rank. A customer typed a question, got ten links, and your work was to be one of them.

Increasingly the customer gets an answer instead. They ask Google, or ChatGPT, or the assistant built into their phone, and something writes a reply that draws on a handful of sources. Sometimes you are one of those sources and they arrive on your site. Sometimes you are the source and they never need to.

Here is the encouraging part. AI systems are not mysterious about what they need, AI search optimisations have been made very clear. They assemble answers from evidence they can reach, read, trust and quote. That gives you a short, concrete list of things to supply, and four of the five are finite pieces of work that stay done once they are done.

Pew Research put some numbers on the shift. Across nearly 69,000 real Google searches, when an AI summary appeared at the top of the page 8% of people clicked through to a website. Without a summary, 15% did. So the behaviour has genuinely changed, and it is worth planning around rather than waiting out.

1. Access, decided on purpose

AI systems reach your website through crawlers, and different crawlers do different jobs. Knowing which is which turns this from a vague worry into a decision you can make.
OpenAI is the clearest example because it publishes exactly what each of its agents does. OAI-SearchBot is the one that surfaces websites in ChatGPT’s search features, and OpenAI’s documentation is direct about the consequence: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. GPTBot is separate and collects content for training foundation models. OAI-AdsBot only visits pages submitted as advertisements. And ChatGPT-User handles fetches a person has actively asked for, such as when someone tells ChatGPT to go and look at a particular page, which OpenAI notes robots.txt rules may not apply to in the same way.

That separation is genuinely useful, because it means you can welcome the crawler that brings you visibility while making a different choice about the one that trains models. They are independent settings and you are entitled to treat them differently.

Two places to look. Your robots.txt, read line by line rather than through a plugin’s summary of it, so you know which agents you have welcomed. And the layer above your site, because your CDN or host may hold a view of its own. Cloudflare announced on 1 July 2025 that it was changing its default to block AI crawlers unless they compensate creators, which is a defensible position on their part and a decision worth confirming is the one you want for your business.

Worth doing: confirm your robots.txt says what you intend, check your CDN or host for bot rules applied above the site, then look at your server logs to see which AI agents are actually arriving. Permissions tell you what is allowed. Logs tell you what is happening.

2. Something they are allowed to quote

Google’s guidance for generative AI features contains one requirement that decides a great deal: a page must be indexed and eligible to be shown in Google Search with a snippet.
Eligibility is the operative idea. Snippet controls such as nosnippet, max-snippet and the data-nosnippet attribute govern how much of a page can be displayed, and they apply to AI features as well as to ordinary results. Google uses the same set of controls for both, so a generous snippet setting is what gives an AI answer something of yours to work with.

This is a good thing to know rather than a trap, because it means there is no separate AI permission to go and find. Get your snippet settings right and you are eligible everywhere at once.

Worth doing: check the robots meta tags and X-Robots-Tag headers across your templates rather than just the home page, since blog and service templates often carry different settings. Pay particular attention to any data-nosnippet wrapped around headings, summaries, service descriptions or pricing, because that is exactly the material an AI answer would otherwise draw on.

3. Content that is readable on arrival

Google renders JavaScript and points site owners at JavaScript SEO best practice rather than steering them away from modern frameworks. Its guidance is also clear that generative AI features work from publicly accessible, crawlable content.

The practical point is that not every AI crawler is as capable as Googlebot. If your site assembles its content in the browser, some systems will arrive at a page and find the structure without the substance. Putting your commercially important content into the HTML that arrives first is the fix, and it improves things for screen readers and slower connections at the same time.

Google also recommends semantic HTML, for the same reason: headings that are genuinely headings and lists that are genuinely lists give any reader, human or machine, a reliable map of what matters.

Worth doing: load your key pages with JavaScript disabled and read what comes back. Your headings, service descriptions, locations and contact details should all be there. Where they are not, server side rendering or shifting that content into the initial HTML puts it back in reach.

4. One consistent account of who you are

An AI answer is assembled from many sources, and your website is only one of them.
Google’s guidance points specifically at Google Business Profile for local business information and Merchant Center feeds for products. For most Australian businesses the Business Profile is the one that earns attention, and it rewards being treated as a real page of your website rather than something set up once and forgotten: accurate categories, current service areas, a description that reflects the services you actually offer now.

Beyond that, consistency is the principle. When your website, your Business Profile and the industry directories you appear in all describe your business the same way, you are giving these systems one clear account to work from. That coherence is what lets an assistant answer confidently about you.

Structured data is worth a mention here. Google is explicit that it is not required for generative AI search, so nobody should sell it to you on that basis. It remains genuinely valuable for rich results in ordinary search, which is a good reason to have it, just a different one.

Worth doing: bring your Business Profile fully up to date, then find every significant third party listing of your business and make the core facts match across all of them.

5. Something only you could have published

This is the one Google is most emphatic about. Its guidance states that creating content people find unique, compelling and useful will likely influence your presence in generative AI search more than any of its other suggestions.

Which is quietly good news, because it means the thing that works is the thing you are best placed to do. You know what your customers ask before they buy. You know which assumptions in your industry are wrong. You know what a project actually costs and how long it actually takes. Nobody else can write that, and an AI system assembling an answer has nothing to draw on but the sources that said something specific.

Google also makes clear what is not needed, which is worth knowing simply so you can spend your effort well. You do not need to write in a special style for AI. You do not need to break your content into small pieces. Plain, well organised, genuinely useful writing is the format.

Worth doing: pick the five questions you answer most often in sales conversations and write the definitive version of each, with your actual numbers and examples in them.

Where llms.txt fits

You will hear about llms.txt, so it is worth being clear about what it currently does.

It is a proposed convention: a simple markdown file that offers a guided map of your site’s important content. It is cheap to produce and it does no harm. Today its demonstrated value is concentrated in documentation, where it genuinely helps developers point coding assistants at the right resources.

For the major AI search systems, take up is still early. Ahrefs analysed server logs across 137,000 domains in May 2026 and found that 97% of llms.txt files received no requests at all that month, with a good share of the requests that did occur coming from SEO auditing tools. Google’s own guidance says the file is not required for its generative AI features.

So the sensible position: have one if you publish documentation, where it earns its place now. Otherwise treat it as cheap optionality rather than a priority, and put the effort into the five above. We are watching adoption closely and will tell you the moment that changes.

What to expect when conducting AI search optimisations

Items one through four are finite. They can be checked, fixed and verified, and they stay fixed. Item five is ongoing, which is simply the honest shape of it: the technical work makes you reachable and quotable, and then the content has to be worth quoting.

On timing, our standard is the same as for all search work. Expect three to six months before movement shows, and twelve or more before it consolidates. We would rather set that expectation now than manage a disappointment later.

Measurement lives in Search Console. Treat anything claiming to show you internal Google metrics on AI visibility with caution, since Google has said no such access exists.

If you would like a hand with any of it, whether that is running the technical checks properly or keeping the content going month after month, we would be glad to help. Just give us a call!

Sources:

Soto Group are a full service creative agency delivering impactful brands, websites and marketing strategies.

7 + 4 =