A customer types "best mortgage broker in Peterborough" into ChatGPT. The model searches the web, reads several websites, and returns an answer with sources. Your competitor's site appears in the response. Yours does not.
You rank on page one of Google. You have good reviews. You have been in business for fifteen years. ChatGPT did not mention you because it could not read your website.
How AI search works
AI search engines like ChatGPT, Perplexity, Claude, and Gemini send crawlers across the web. Those crawlers read your pages and store the content. When a user asks a question, the AI pulls from that stored content to build an answer.
Google does something similar but returns a list of links. AI search returns a written response with your business woven into the answer or left out of it.
The difference is selection. Google shows ten results and lets the user pick. AI search picks for the user. If your site is not in the pool of content the AI can read, you are excluded from the answer.
What AI crawlers read
AI crawlers look for structured, machine-readable content. Three things determine whether your site gets included.
Structured data (JSON-LD). A block of code on each page that tells crawlers what your business does, where you are, what services you offer, and how to contact you. Google uses it for rich results. AI search engines use it to understand your business at a glance.
llms.txt files. A plain text file at the root of your domain that describes your site in a format optimised for language models. Think of it as a robots.txt for AI. It tells AI crawlers what your site is about and which pages matter.
Crawler permissions. Your robots.txt file controls which bots can access your site. If you block AI crawlers, they skip you. If you allow them, they index your content alongside Google.
Most websites have none of this
Run a test. View the source code of your website and search for "application/ld+json". If you find nothing, your site has no structured data. AI crawlers read your pages as raw text and guess what your business does.
Check your robots.txt file. If it does not mention GPTBot, ClaudeBot, PerplexityBot, or Bytespider, you have not configured access for AI crawlers. Some hosting platforms block them by default.
Look for an llms.txt file at yourdomain.com/llms.txt. If it returns a 404, AI models have no summary of your site to reference.
Building for AI search
We add structured data to every page we build. LocalBusiness schema on the homepage. Service schema on each service page. BreadcrumbList for navigation. FAQPage schema where relevant. The data is embedded as JSON-LD in the page head and validated against Google's structured data testing tool.
We create llms.txt files that describe the business, list the services, and link to the most important pages. The file is written in plain language that AI models parse without ambiguity.
We configure robots.txt to allow access for Google, Bing, and AI crawlers including GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), and Bytespider (TikTok). Access is granted. The crawlers do the rest.
The window is open
AI search adoption is growing. The businesses that appear in AI search results now are the ones building for it now. The ones that wait will spend the next two years wondering why their competitor shows up in ChatGPT and they do not.
Google took twenty years to become the default. AI search is moving faster. Your website needs to be readable by both.