Robots.txt, Structured Data, and AI Crawler Access
Technical accessibility is the foundation: if a page cannot be fetched, rendered, or associated with the right entity, its content is harder to evaluate.
Published 26 October 2026 · Practical guide, not a promise of rankings or inclusion.
Review access before adding markup
Check that important pages return a successful response, are linked from the site, use the intended canonical URL, and are not accidentally blocked by robots rules or noindex metadata. Test the public URL, not only a local development copy.
Use structured data that matches the page
Add only schemas that describe visible content: organisation details for the organisation, Article data for an article, and FAQPage only when the questions and answers are visible and unique. Structured data clarifies a page; it does not create evidence that is not there.
Keep a change log
Record the URL, change, date, test result, and any uncertainty. This makes future visibility checks reproducible and stops a team from attributing a change to an outcome it did not measure.