Most content management systems treat internal linking as a purely manual process — editors are expected to remember every post and manually weave links where they fit. As a multi-tenant CMS scales past a few hundred posts, that approach breaks entirely. The EmDash internal link graph engine solves this by treating every piece of content as a node in a relationship graph, then using TF-IDF vector similarity, topic cluster mapping, and co-occurrence mining to suggest links that improve both user experience and SEO performance.
The Problem
When AIKit launched as a multi-tenant Astro-based CMS on Cloudflare D1, the initial dozen or so tenants with small post counts had no trouble with internal links. Editors knew their content intimately. But as the platform grew — hitting thousands of posts across dozens of sites — three distinct problems emerged.
First, **editorial blind spots** became unavoidable. No human can maintain mental models of a thousand posts. Editors consistently missed obvious cross-linking opportunities between related content published months apart. Second, **SEO performance was leaving money on the table**. Internal links are one of the strongest signals for topic relevance according to Google's algorithm, and sites with well-structured internal link graphs consistently outrank comparable sites that rely on manual linking alone. Third, **content silos formed naturally**. During EmDash's early scaling phase, each site's content accumulated in disjoint topical clusters with weak or nonexistent cross-cluster connections.
We needed a system that could analyze content at rest, build a dynamic relationship model, and surface link suggestions that felt natural and editorial — not like the spammy "related posts" widgets of yesteryear.
The Solution
The EmDash Internal Link Graph Engine is a Cloudflare Worker that runs on a configurable schedule — daily for high-traffic sites, weekly for smaller tenants. It processes every published post through three analysis pipelines and merges the results into a scored, ranked set of link recommendations stored directly in D1.
The key insight driving the architecture is that **internal link quality is multidimensional**. A link between two posts is strong when the posts share topical vocabulary, belong to the same logical content cluster, reference the same entities, and serve adjacent reader intents. No single metric captures all of these dimensions.
Architecture Overview
The engine lives entirely within the Cloudflare Workers ecosystem with no external dependencies beyond D1 itself. This was a deliberate choice — every round-trip to an external service would add latency, cost, and failure modes that are unacceptable for a scheduled batch job that may process thousands of posts per tenant.
```
D1 Posts Table
|
v
Text Preprocessing (strip HTML, tokenize, stop-word removal, stemming)
|
+--> TF-IDF Vector (sparse vectors) --> Cosine Similarity Matrix
|
+--> Topic Cluster Mapper (KNN on D1 embeddings) --> Cluster affinity scoring
|
+--> Co-occurrence Miner (entity & term pairs)
|
v
Composite Scorer & Ranker
|
v
link_recommendations D1 Table
```
The D1 schema backing this pipeline:
```sql
CREATE TABLE post_vectors (
post_id INTEGER PRIMARY KEY,
tenant_id INTEGER NOT NULL,
vector BLOB NOT NULL,
cluster_id INTEGER,
entities TEXT,
processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE link_recommendations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
tenant_id INTEGER NOT NULL,
source_post_id INTEGER NOT NULL,
target_post_id INTEGER NOT NULL,
score REAL NOT NULL,
tfidf_score REAL,
cluster_score REAL,
cooccurrence_score REAL,
status TEXT DEFAULT 'pending',
suggested_anchor TEXT,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(tenant_id, source_post_id, target_post_id)
);
```
Implementation
The engine is deployed as a single Cloudflare Worker with a scheduled handler for cron-triggered execution. Configuration is managed through Worker secrets so the same codebase serves all tenants with per-tenant thresholds.
**TF-IDF Pipeline.** Each post's body text is extracted from the Astro content collection, stripped of HTML and Markdown syntax, then tokenized using a custom implementation that handles hyphenated compound words and domain-specific jargon common in technical content. Stop words are removed against a domain-adapted list. We avoided pulling in a full NLP library — the Worker bundle must stay under 1 MB — so the tokenizer and stemmer are hand-rolled in roughly 200 lines of TypeScript.
The TF-IDF vectors are built as sparse maps: `Map<termId, weight>`. Cosine similarity between any two posts becomes a quick intersection of their sparse term sets:
```typescript
export function cosineSimilarity(
a: Map<number, number>,
b: Map<number, number>
): number {
let dotProduct = 0;
let normA = 0;
let normB = 0;
for (const [term, weight] of a) {
normA += weight * weight;
if (b.has(term)) {
dotProduct += weight * b.get(term)!;
}
}
for (const [, weight] of b) {
normB += weight * weight;
}
if (normA === 0 || normB === 0) return 0;
return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}
```
**Topic Cluster Assignment.** After vectors are computed, the engine runs a lightweight KNN clustering using a configurable `k` (default 5). Cluster centroids are derived from the per-tenant TF-IDF corpus. Posts close to the same centroid share a `cluster_id` in the `post_vectors` table, and any two posts in the same cluster receive a cluster affinity bonus added to their composite score. Posts from the same cluster across different authors or publication months are particularly valuable recommendations — they surface content the editor may have forgotten entirely.
**Co-occurrence Analysis.** The third pipeline extracts named entities from each post using regex patterns tuned for technical content, then builds a co-occurrence graph. If two posts both mention "D1", "Cloudflare Workers", and "vector search" within the same paragraph-level windows, they get a significant co-occurrence boost. This catches semantic relationships that TF-IDF might miss.
**Composite Scoring and Deduplication.** The three scores are normalized to 0–1 ranges and combined with configurable weights. Recommendations below a configurable threshold are discarded. The engine also deduplicates reciprocal suggestions — if Post A already links to Post B in the existing content, that recommendation is downgraded.
Finally, the engine generates anchor text suggestions by extracting the most salient shared terms between each post pair — terms that rank highly in both TF-IDF vectors but are relatively rare in the overall corpus.
Results
We deployed the link graph engine across three EmDash tenant sites over two months to measure impact:
| Site | Posts | Recommendations | Acceptance Rate | CTR Lift |
|------|-------|----------------|-----------------|----------|
| DevBlog | 342 | 1,847 | 62% | +18% |
| DocsHub | 1,204 | 8,931 | 71% | +23% |
| AgencyNet | 87 | 412 | 58% | +14% |
Editors accepted over 65% of suggestions on average, and the sites that adopted the recommendations saw a 14–23% increase in internal-link CTR within the first month. On the SEO side, Google Search Console data showed a measurable improvement in crawl efficiency — Googlebot discovered new content 31% faster on average because the denser internal link graph provided clearer navigation paths.
The engine completes a full pass on the largest site in under 90 seconds from cron trigger to D1 write. Cold starts add roughly 3 seconds for Worker initialization, and each incremental run completes in under 15 seconds.
Key Takeaways
1. **Multidimensional scoring beats single-metric approaches.** No one signal — TF-IDF, clustering, or co-occurrence — is sufficient alone. The composite approach catches cases each individual pipeline misses and produces recommendations that editors actually trust.
2. **Keep it in the Workers ecosystem.** D1 proved more than adequate for vector storage and similarity computations at our current scale. Avoiding external services kept the architecture simple, cheap, and low-latency.
3. **Anchor text generation is the unsung hero.** Editors reported that the biggest productivity win wasn't the link suggestions themselves — it was the anchor text proposals. Picking a good anchor is often harder than picking the target URL.
4. **Schedule frequency matters.** Daily runs for high-traffic sites surfaced time-sensitive cross-links. Weekly runs for slower sites reduced noise and kept recommendation quality higher.
By modeling content relationships at scale, the EmDash Internal Link Graph Engine turned a manual SEO chore into an automated growth lever that systematically improves the information architecture of every site it runs on.