The Architecture Behind Automated Content Publishing
When we set out to build AIKit's blog automation pipeline, we weren't just solving "how to write blog posts." We were solving a distributed systems problem: how do you coordinate cron-triggered content generation, database inserts, multi-channel distribution, and rate-limited API calls — all running on a serverless budget?
The answer is a multi-tenant pipeline that uses Cloudflare Workers as the orchestration layer, D1 as the content store, and a carefully managed cron schedule that prevents collision between concurrent runs.
Why Multi-Tenant Matters
Most CMS blog automation tools are single-tenant: one user, one schedule, one queue. But a developer-first platform like EmDash needs to support:
- **Multiple content streams** — product blog, engineering blog, partner content
- **Concurrent generation runs** — LLM-powered creation happening in parallel
- **Independent queues** — each stream with its own cadence and category tags
- **Conflict-free publishing** — no two runs should produce the same slug
AIKit's plugin sandbox runs each content stream as an isolated V8 Isolate on Cloudflare Workers. Each isolate gets its own KV namespace for queue state and independent access to the shared D1 database.
The Schedule Coordination Problem
The tricky part is cron schedule collision. When you have multiple cron jobs running at 6 AM, 10 AM, and 2:30 PM every day, and each one generates 1-2 posts, you need to ensure:
1. **No duplicate slugs** — Two concurrent runs could pick the same topic from a shared calendar
2. **No D1 write conflicts** — UNIQUE(slug, locale) constraint means INSERTs fail on collision
3. **No queue file races** — Multiple processes writing to the same queue directory
The solution is a two-phase locking pattern at the application level:
```
Phase 1 (Pre-check): Query D1 for last published slug → Derive next number
Phase 2 (Publish): Insert → Verify → Archive (idempotent on success)
```
If Phase 2 hits a UNIQUE constraint error, the pipeline doesn't crash — it logs, archives the duplicate, and moves to the next file.
D1 as a Content Delivery Layer
One of the most powerful design decisions was using D1 not just as a database but as a content delivery layer. Here's why:
```sql
-- Single query pulls everything needed for a blog post
SELECT p.title, p.content, p.excerpt, p.published_at, p.author_id,
r.data as revision_data
FROM ec_posts p
JOIN revisions r ON r.id = p.live_revision_id
WHERE p.slug = ? AND p.status = 'published' AND p.locale = 'en'
```
This query runs in 2-5ms from Cloudflare Workers. The response gets cached at the edge. Unlike a traditional CMS that needs Redis or Memcached, D1's read performance at 100 published posts is still sub-5ms.
The Math Behind 100 Posts
At post #100, here's what the pipeline has achieved:
| Metric | Value |
|--------|-------|
| Total posts published | 100 |
| Average words per post | 1,200 |
| Total content generated | ~120,000 words |
| Average time to publish | ~45 seconds (generate + insert + verify) |
| Pipeline failures | 0 (all slots recovered via archive-on-collision) |
| Queue files archived | 100 |
| D1 storage used | ~4.2 MB (with revisions) |
Key Engineering Decisions
Why Not Use a Message Queue?
We evaluated using Cloudflare Queues for the pipeline. The overhead wasn't justified at 100 posts. Queues would add latency, complexity, and cost for a pipeline that's already fast enough. The filesystem-based queue (JSON files in a directory) works perfectly for this scale.
Why Not Use D1 Transactions?
D1 doesn't support `BEGIN TRANSACTION`/`COMMIT` via the SQL API. The three-step insert (post → revision → update) is not atomic. If the pipeline crashes between step 2 and 3, you get an orphan revision. The fix: a periodic cleanup worker queries for posts with NULL `live_revision_id` and removes them.
Why Not Use HTTP Webhooks?
All content flows through direct D1 inserts — no HTTP middleware, no webhook latency. The `queue-publisher.py` script runs `wrangler d1 execute` directly, bypassing the entire HTTP layer. This makes the pipeline resilient to site outages and deployment windows.
What's Next at Post 100
The pipeline is about to evolve. At post 100, we're adding:
1. **Automated social distribution** — Post excerpts to X/Twitter and Telegram on publish
2. **Content scoring feedback loop** — Track page views per post and feed back into the LLM prompt
3. **Multi-language support** — D1's locale field was designed for this from day one
4. **Scheduled republishing** — Auto-refresh posts that have dropped in SEO rankings
The infrastructure is already there. Now it's about connecting the channels.
Conclusion
Building a multi-tenant blog pipeline on serverless infrastructure isn't just possible — it's elegant. D1's sub-5ms reads, Workers' isolated V8 sandboxes, and a filesystem-based queue give us production-grade content automation for effectively zero infrastructure cost. If you're building a content pipeline, start with the simplest model (files + database) and only add complexity when the metrics prove you need it. At post 100, we still don't.