Making Fathom Data agent-ready: What we changed and why

The previous article explored why agent readiness matters. This one focuses on how we applied those principles to our own website: what we changed, why we changed it, and where we decided not to invest yet.

Our goal was not to implement every emerging AI standard. A lot of immediate value can be realised by improving the foundations that help AI systems find, interpret, and accurately represent your website. Some of these practices build on established areas such as technical SEO and accessibility, while others are newer approaches whose value and adoption are still becoming clearer. The challenge is knowing which changes are practical investments today and which are better treated as possible future opportunities.

Improving the agent-readiness of a website falls into three tiers: discovery, understanding, and action. The first two are practical today and overlap heavily with strong technical SEO, accessibility, and content practices. The third is still emerging, but the foundations built in the first two tiers will determine how easily websites can adapt as more capable agent-based experiences become mainstream.

Tier 1: Discovery

The first step is making sure AI systems can find your content. This is the lowest-effort, highest-confidence tier. It does not require changing the content or design of your website, only ensuring that the signals and infrastructure around it are working correctly.

That is becoming more important as AI-specific controls move into the infrastructure of the web. For example, Cloudflare has announced that, from 15 September 2026, new domains will block AI agents by default on pages that display ads unless site owners choose otherwise explicitly. A website can be perfectly structured for AI and still be effectively invisible to many agents if they cannot access or discover it.

Our own website is built on Astro and hosted on GitLab Pages, which means none of these signals are configured automatically. We had to make our content discoverable ourselves by exposing the right machine-readable files and access controls.

Help AI find your content

AI systems first need to know what content exists and be able to access it. An up-to-date XML sitemap remains one of the clearest ways to advertise the pages on your site, while sensible internal linking helps both people and machines navigate between them. It’s also worth checking that your hosting, CDN, or security configuration isn’t unintentionally preventing compliant AI crawlers from reaching your content.

Our sitemap is generated automatically using Astro’s official sitemap integration, configured directly in astro.config.mjs and run as part of the build pipeline. Rather than a single flat file, Astro builds a sitemap index that points to one or more sitemap files, splitting URLs automatically as the site grows, so AI crawlers always have a complete map to work from. This runs at build time, so that map stays current with no manual updates needed.

Help AI understand your site

An llms.txt file provides AI systems with a concise overview of a website and links to its most important content. It complements traditional discovery mechanisms by giving AI systems a machine-readable summary to the site. The standard is still evolving, and not every AI system uses it. Even so, support is growing, developer tooling is improving, and it is simple enough to include as part of a well-maintained website.

Rather than maintaining an llms.txt file manually, we built our own Astro integration that generates it automatically during every build, keeping it current with no manual upkeep. Alongside the standard index file, it also produces an llms-full.txt file containing the full content of each page. Each page’s HTML is parsed and stripped of navigation, headers, footers, scripts, and other layout elements, then converted into clean Markdown, so AI systems get a clean content layer to process rather than having to extract it from the underlying site structure.

Tell AI how your content can be used

Compliant AI crawlers look to standards such as robots.txt and newer directives such as Content-Signal to understand a publisher’s preferences. These mechanisms do not enforce behaviour, but they communicate how you would like your content to be accessed and used.

We updated our robots.txt to include Content-Signal directives:

Content-Signal: search=yes, ai-input=yes, ai-train=no

This expresses our preference to allow search indexing and AI systems to use our content as input, while opting out of its use for model training.

Tier 2: Understanding

Once an AI system has reached your site, the next challenge is helping it interpret what it finds. This tier overlaps heavily with technical SEO, accessibility, and good content design. The same practices that make a website easier for machines to understand usually make it better for people too.

Use semantic HTML and structured metadata

Semantic HTML gives structure and meaning to a page. Elements such as <main>, <nav>, <article>, and <button> tell systems what different parts of a page represent, rather than leaving them to infer meaning from generic containers and styling alone.

Structured data provides another layer of context. Using Schema.org markup, websites can explicitly describe entities such as organisations, services, products, articles, and events. We added Organization structured data across our site as JSON-LD, including information about our services, areas of expertise, location, and related profiles. This can be verified using Google’s Rich Results Test.

One useful example is the knowsAbout property, which allows us to explicitly describe the areas where Fathom Data has expertise, such as data engineering, machine learning, AI, and cloud infrastructure. Meanwhile, properties such as sameAs link our entity to verified profiles like LinkedIn, helping AI systems disambiguate “Fathom Data” from unrelated entities with similar names, rather than inferring identity from context alone. These properties are intended primarily for machine interpretation rather than human visitors, illustrating how some website elements are increasingly being created specifically to help AI systems understand a business.

Structured data should be viewed as infrastructure rather than a shortcut to visibility. It helps provide context about your business, but it does not replace clear content or guarantee that AI systems will cite your website.

Make content accessible to both humans and machines

A website that is difficult for people to use is unlikely to be easy for AI systems to interpret. Accessibility, performance, and clarity all contribute to a stronger foundation.

We improved our Lighthouse performance scores, reviewed colour contrast to ensure content remains accessible, and removed design elements that added complexity without improving the experience, such as unnecessary animations and interactions.

We also implemented metadata across the site, including canonical URLs, robots directives, Open Graph metadata, and social sharing information. While much of this originated from traditional SEO and social sharing requirements, it creates clearer signals about each page’s purpose and identity.

Make important content explicit

People are good at reading between the lines, AI systems are less so. If your business specialises in a particular industry, serves a specific region, or offers a distinctive capability, say so clearly.

We applied this principle throughout our own website. For example, we changed a page heading from “About” to “About Fathom Data”. A human visitor understands what an “About” page contains from context, but an AI system benefits from explicit signals about the page’s purpose.

We also reviewed our content to make it less generic, adding details specific to our experience, expertise, and approach. We strengthened our unique voice and are working on adding case studies that provide deeper examples of the problems we solve, the approaches we take, and the outcomes we deliver.

Provide content in formats agents can use

As AI systems become more ubiquitous, some websites may benefit from exposing information in formats designed for direct machine consumption. This is different from an llms-full.txt file, which provides a static, AI-friendly representation of a website’s existing content.

Alternative content formats and content negotiation allow a system to request the same information in a different representation, such as Markdown or structured data, rather than a traditional HTML page. This becomes more valuable for websites where agents need to retrieve and act on information dynamically, such as software documentation, developer platforms, or API-driven services.

For a typical company website, this is less urgent. Our current focus has been on making our content clear, structured, and accessible through approaches such as semantic HTML and llms-full.txt. Our current hosting setup also does not support the required middleware for this type of implementation. If we move to infrastructure that provides more flexibility, this is something we will revisit.

Tier 3: Action

The final tier is about moving beyond understanding to interaction. Instead of simply reading and summarising a website, AI systems are beginning to carry out tasks on a user’s behalf, such as checking availability, retrieving live information, searching a catalogue, or submitting a form.

Supporting these workflows requires websites to expose structured capabilities, not just structured content. A growing ecosystem of standards, including Model Context Protocol (MCP), WebMCP, and API discovery specifications, is emerging to make this possible. These allow an AI system to discover not only what a website says, but what it can do.

We have deliberately not implemented this layer on our own website. Our site is primarily designed to communicate who we are, what we do, and how organisations can work with us. While AI systems can already answer questions, summarise our expertise, and guide users to relevant information using the content we provide, there are currently no workflows that benefit from exposing agent-facing actions. Organisations offering APIs, software platforms, or transactional services may find these capabilities relevant much sooner.

For most organisations, however, this is not where the work should begin. The standards are still evolving, best practices are still emerging, and many businesses will gain more value by investing in the first two tiers. Those foundations, making your website easy for AI systems to discover and understand, will remain valuable regardless of which protocols ultimately become established.

Getting started

Agent readiness is not about rebuilding your website or implementing every emerging standard. It is about making sure the foundations are in place: that AI systems can find your content, understand your business, and accurately represent what you offer.

For some organisations, that may mean improving technical foundations such as metadata, structured data, and accessibility. For others, the biggest opportunity may be clarifying content, exposing expertise more explicitly, or preparing systems for future agent-driven interactions.

If agent readiness is becoming a priority for your organisation, we can help assess your current position, identify the changes that will have the greatest impact, and build the foundations needed to remain visible as the way people discover and evaluate businesses continues to evolve.

We are happy to help you identify practical next steps to get your website agent ready. Get in touch today to start the conversation.