llms.txt, agent-card.json and robots.txt explained
Three small files that tell AI how to treat your website
Quick answer: robots.txt says which crawlers may visit your site. llms.txt is a short Markdown summary of your business for AI systems. agent-card.json describes what your business can do for AI agents using A2A. They do different jobs and you may want all three.
robots.txt: who may visit
A plain text file at /robots.txt. It tells crawlers which pages they may fetch. It is the oldest of the three and search engines respect it.
AI companies use different crawler names. Examples are GPTBot (OpenAI training), OAI-SearchBot and ChatGPT-User (ChatGPT search and browsing), ClaudeBot (Anthropic), PerplexityBot and Google-Extended (whether Google may use your content for its AI models). You can allow or block each one. Blocking them can make you invisible in AI answers, so decide on purpose.
llms.txt: a summary for AI
A Markdown file at /llms.txt with a short description of your business and links to your key pages. A longer llms-full.txt can hold more detail. The idea was proposed in 2024.
Honest limit: it is a proposal, not an official standard, and the big search engines have not confirmed they rely on it. It costs little to publish and helps some AI tools, so we include it, but it is no substitute for good pages. Ours is at /llms.txt.
agent-card.json: what you can do for agents
Part of the A2A protocol. A JSON file at /.well-known/agent-card.json that lists your agent’s name, skills, endpoint and how to authenticate. Only useful if you actually run an A2A endpoint behind it. Ours is at /.well-known/agent-card.json.
Which should you publish?
- Every site: a sensible robots.txt and a sitemap.
- Most business sites: add llms.txt.
- Businesses taking agent requests: add an Agent Card and an A2A endpoint.
Work through the full readiness checklist for the other steps.
Frequently asked questions
Does llms.txt improve Google rankings?
There is no confirmed evidence that it does. Treat it as a low-cost help for AI tools, not a ranking trick.
Can robots.txt stop AI using my content?
It asks well-behaved crawlers to stay away. It cannot force anyone to comply.
Where do these files go?
In the root of your website, for example yoursite.co.uk/llms.txt. The Agent Card goes in the /.well-known/ folder.
Can you set these up for me?
Yes. It is part of our website upgrades and agent readiness work.
Registered in England & Wales, company no. 16478408 · ICO ZC079121