CCFish runs A/B tests on its App Store product page to validate features before shipping — turning App Store Optimization into an extension of the development pipeline.
The Problem
Mobile game developers face a disconnect: product decisions (new features, UI changes, game mechanics) are made by engineers, but App Store listing copy and screenshots are owned by marketing. When these teams don't communicate, product launches miss their organic growth potential. CCFish discovered that App Store experiments — product page A/B tests — could bridge this gap, but only if integrated directly into the development workflow.
The Solution: App Store Experiments as a Development Tool
CCFish treats each App Store experiment as an extension of its feature rollout pipeline:
1. **Feature ships to staging** — the team enables the feature for internal testers
2. **Experiment launches on App Store** — screenshots and copy highlighting the feature go live immediately
3. **Data flows back** — impression-to-install conversion rates for the experiment variant feed into the product analytics dashboard
4. **Feature goes GA** — if the experiment shows positive conversion lift, the feature ships wide
```json
// Example: CCFish App Store Experiment config (Cloudflare Workers)
{
"experiment": {
"name": "multiplayer_lobby_redesign",
"treatment": "screenshot_v2",
"control_impressions": 15420,
"treatment_impressions": 15780,
"conversion_lift": "+12.3%",
"decision": "ship"
}
}
```
Architecture Overview
The pipeline runs entirely on Cloudflare Workers and D1:
| Component | Tech | Purpose |
|-----------|------|---------|
| Experiment API | Cloudflare Worker | Stores experiment config + results |
| Database | D1 | Persistent storage for experiment history |
| Analytics Pipeline | Cron Worker | Fetches App Store Connect API daily |
| Dashboard | EmDash CMS | Displays experiment results to stakeholders |
Step 1: Set Up App Store Connect API Access
Apple's App Store Connect API provides product page optimization endpoints. CCFish uses a Cloudflare Worker scheduled to run daily:
```bash
Wrangler cron schedule: daily at 6 AM
curl -X GET https://api.appstoreconnect.apple.com/v1/appStoreVersionExperiments \
-H "Authorization: Bearer $(generate_jwt_token)"
```
Step 2: Build the Worker Endpoint
```typescript
// excerpt from experiment-tracker Worker
export default {
async fetch(request: Request, env: Env): Promise<Response> {
const url = new URL(request.url);
const { results } = await env.DB.prepare(
"SELECT * FROM app_store_experiments ORDER BY created_at DESC LIMIT 10"
).all();
return Response.json(results);
}
}
```
Step 3: Connect Experiments to Product Analytics
The experiment data feeds into CCFish's product dashboard. Each experiment gets tagged with:
- **Feature ID** — links to the internal feature tracker
- **Status** — running, completed, shipped, or reverted
- **Conversion rate** — treatment vs control
- **Statistical significance** — calculated via chi-squared test
Results
In the first 90 days of running App Store experiments linked to feature development:
- **30+ experiments run** across 4 product iterations
- **18 feature ships** validated by positive conversion lift
- **Average +8.7% conversion lift** on shipped features
- **2 features reverted** before spending engineering resources on full launch
Key Takeaways
App Store experiments are not just a marketing tool. When integrated into the development pipeline, they become a data-driven product validation mechanism that:
- Prevents shipping features users don't care about
- Gives engineers direct visibility into organic growth metrics
- Creates a feedback loop between product iterations and App Store conversion