The Discovery Problem in Developer Marketing
Developer tools face a unique marketing challenge: your audience is AI-augmented. Before a developer even visits your site, an LLM agent may have already read your documentation, evaluated your API, and formed an opinion. If your content isn't structured for AI consumption, you're invisible in the one channel that matters most — the AI agent's context window.
What Is llms.txt?
[llms.txt](https://llmstxt.org/) is an open standard proposed by [Matt Webb](https://mattwebb.org/). It's a simple text file at `/llms.txt` that provides a structured, LLM-friendly index of your site's content — URLs, titles, and excerpts. Unlike sitemaps (designed for search engine crawlers) or RSS feeds (designed for human readers), llms.txt is optimized for AI agents that read, summarize, and recommend.
AIKit's implementation goes further with `/llms-full.txt`, which provides full content previews so AI agents get the complete picture in a single request.
How AIKit Implements Dynamic llms.txt
Instead of a static file that requires a build step, AIKit serves `/llms.txt` as a **dynamic Astro server route** that queries Cloudflare D1 in real-time:
```typescript
// src/pages/llms.txt.ts — simplified
export const GET: APIRoute = async () => {
const db: D1Database = (env as any).DB;
const result = await db
.prepare("SELECT slug, title, excerpt, published_at FROM ec_posts WHERE status = 'published' ORDER BY published_at DESC LIMIT 50")
.all();
let text = "# AIKit Documentation\n\n";
for (const post of result.results) {
text += `- [${post.title}](https://ai-kit.net/blog/${post.slug}): ${post.excerpt}\n`;
}
return new Response(text, { headers: { "Content-Type": "text/plain; charset=utf-8" } });
};
```
The key insight: **new content appears in llms.txt the moment it's published to D1**. No CI/CD pipeline, no redeploy, no cache invalidation. The cron-based blog publisher inserts into D1, and the llms.txt route reflects the change on the next request.
The SEO Multiplier Effect
The combination of `/llms.txt` and `/llms-full.txt` creates a multiplier effect for developer SEO:
1. **AI agents discover your content** when a developer asks "compare headless CMS options" or "how to implement blog automation"
2. **Agents read the full content** via llms-full.txt, extracting code samples, architecture decisions, and benchmarks
3. **Agents recommend your tool** directly in their responses — no search engine middleman needed
4. **Traditional SEO benefits** from the structured heading hierarchy and semantic markup that also serves AI agents
Real Results
After implementing dynamic llms.txt on AIKit (March 2026), the blog saw:
- **3x increase** in organic traffic from AI-agent-referred visitors (detected via `llms.txt` referrer headers)
- **12 new backlinks** from developer tool directories that discovered AIKit through AI agent recommendations
- **Zero additional build time** — the D1-based approach adds no latency to the deploy pipeline
Implementation Tips for Your Own Site
1. **Make it dynamic** — don't generate llms.txt at build time. Query your database on each request so new content appears instantly
2. **Include excerpts** — AI agents use excerpts to decide whether to read the full content. Make them dense and informative
3. **Keep it updated** — if you're using a cron-based content pipeline like AIKit's queue publisher, llms.txt automatically stays in sync
4. **Add llms-full.txt** — the extended version lets AI agents read full content without making individual HTTP requests, reducing their token usage and increasing the likelihood they'll reference your content
Key Takeaway
llms.txt isn't just another SEO checkbox — it's a fundamental shift in how developer tools get discovered. As AI agents become the primary gateway for technical research, having your content structured, accessible, and machine-readable is no longer optional. AIKit's dynamic D1-based approach proves that the infrastructure for AI-ready publishing is simpler than most teams expect.