llms.txt, services.md, and ai-agents.json: The New SEO Files for AI Search
For years, technical SEO teams focused on robots.txt and sitemap.xml. Those files still matter, but they are no longer enough for AI search. AI assistants and autonomous agents need a clearer map: what does the site do, what does it sell, which pages matter, and what should a buyer do next?
That is why AIKit now treats `llms.txt`, `services.md`, and `ai-agents.json` as first-class SEO assets.
The new file stack
A modern AI-search-ready site should expose:
- `robots.txt` for crawler rules
- `sitemap.xml` for canonical URLs
- `llms.txt` for compact AI context
- `llms-full.txt` for deeper article context
- `services.md` for offers, pricing, and timelines
- `ai-agents.json` for structured parsers
- `agentic-engine.md` for a plain-language optimization manifest
Each file serves a different job. Together, they make the site easier for AI systems to understand without forcing them to scrape every page.
What llms.txt should contain
`llms.txt` should be short enough for quick reading but specific enough to be useful. It should include:
- site identity
- positioning
- primary services
- key URLs
- blog or documentation index
- pricing summary
- contact path
The mistake is treating llms.txt like a marketing page. It should be direct and factual. AI agents do not need hype. They need context.
What services.md should contain
`services.md` is for buying intent. It should answer:
- What services are available?
- What is the service ID?
- What is the price range?
- What is the timeline?
- What are the deliverables?
- Who is it best for?
- Where should a human or agent request a quote?
This is especially useful for consultants, agencies, and productized services. If a user asks an AI assistant, `find someone who can set up llms.txt and AI SEO for my Cloudflare site`, the assistant needs structured offer data to recommend a vendor accurately.
Why ai-agents.json exists
Markdown is good for humans and language models. JSON is good for automation. `ai-agents.json` gives parsers a structured object with:
- site name
- description
- discovery URLs
- service list
- topic clusters
- recommended agent flow
- live content counts
- contact fields
That lets programmatic agents ingest the site without guessing.
How sitemap.xml still fits
The sitemap remains the canonical URL inventory. AIKit generates it dynamically from Cloudflare D1 so every published post can appear in the crawl map. The sitemap should include not only blog posts, but also key machine-readable endpoints like `llms.txt`, `services.md`, and `agentic-engine.md`.
How robots.txt changes
Robots.txt should still allow normal crawling, but AI search creates a new need: explicitly show that AI discovery paths are public. AIKit’s robots rules point crawlers toward:
- `/llms.txt`
- `/llms-full.txt`
- `/services.md`
- `/agentic-engine.md`
- `/ai-agents.json`
- `/blog/*.md`
This makes the intent clear.
The AIKit implementation pattern
AIKit uses Astro routes on Cloudflare Workers. The dynamic routes query D1 for published posts, then return plain text or JSON with public cache headers. The static service catalog is generated during build so it exists as a durable asset.
This combination gives speed, freshness, and low operational cost.
The practical takeaway
Classic SEO made your site crawlable. Agentic SEO makes your site understandable and actionable. If your site sells a product or service, it should not make AI agents infer pricing, positioning, or next steps from vague homepage copy.
Make the machine-readable path explicit:
1. robots.txt points to sitemap and AI files
2. sitemap lists canonical URLs
3. llms.txt gives compact context
4. llms-full.txt gives depth
5. services.md describes offers
6. ai-agents.json structures the same information for automation
7. agentic-engine.md explains how agents should use the site
That is the new technical SEO baseline for AI search.