The Campaign Factory

01The iteration tax

Why your experimentation program tests what you already believe

Here is a number I want you to sit with: on most enterprise web stacks, changing one component variant costs six weeks. Not six weeks of work — six weeks of waiting. A ticket gets written, groomed, and sprinted. The change goes through CI, a review, a release window. Somewhere around week five, someone asks whether the hypothesis is still relevant.

Run the arithmetic and the consequence is brutal. If every question costs six weeks, you get to ask eight of them a year. So you don't ask questions — you confirm beliefs. You test the hero banner you were already going to ship, call it experimentation, and report a win rate that means nothing because the losers were never at risk of running.

I've watched this pattern on catalogs of forty thousand URLs managed by teams of four. The constraint was never creativity, budget, or even traffic. It was the price of asking. That price is what this article removes.

02Stop building campaigns. Build the factory.

Volume × taste — the only division of labor that uses both sides well

The reframe that started this came from an unlikely place. A creator handed a frontier model an effectively unlimited budget and told it to make ads. It invented eighty fictional products overnight — brands, packshots, video spots, a live site. The products were disposable. The interesting artifact was the loop: the model didn't make an ad, it built and operated the pipeline that makes ads, checked its own output with its own vision, and kept going without anyone in the room.

Strip away the novelty and there's a serious principle underneath. A model iterates over patterns thousands of times faster than you do, and it has no taste. You have taste, judgment, and accountability, and no throughput. Every sensible division of labor follows from that asymmetry: the machine generates and verifies volume; the human sets direction and makes the calls that matter. "AI writes the copy" is the least valuable version of this. "AI runs my experiment portfolio inside guardrails I designed" is the most.

What follows is the architecture for the second version, built on Sitecore AI's MCP surface, your data warehouse, and a frontier model with tool use. It runs unattended. It is also, by construction, boring in all the places that need to be boring — which is the highest compliment you can pay an autonomous system.

03The six loops

One cycle, shared state, and a deliberate asymmetry that carries the whole safety model

The factory is six loops sharing one set of state files. Each loop is independently restartable, each writes its output where the next can find it, and none of them holds anything important in memory.

One structural decision carries the entire safety model, and it's worth stating plainly. Work is done by workers: parallel, stateless sessions that each own a slice of the queue and are forbidden from publishing — not asked nicely, but running under a role that lacks the permission. Publishing belongs to one integrator: a single serial session that audits finished work, files failures to a redo queue, adopts anything a dead worker left behind, and is the only process that ever touches production. Twenty hands make, one hand ships.

Coordination is deliberately primitive: a manifest file, a redo queue, a progress log, and a mkdir-based lock with a stale-lock breaker. No message broker, no orchestration platform. When something goes wrong at 3 a.m. — and something will — every state transition in the system can be read with cat. You can graduate this to Postgres later without changing a single agent contract. Don't start there.

04The Sitecore layer

MCP gives the agent hands; the permission model decides what those hands can hold

None of this was practical on the classic XP stack, where a variant is a code deployment. On Sitecore AI, content is API-addressable, components carry variants natively, and the platform exposes an MCP server — a standard interface through which an agent can call CMS operations as tools. For marketers: the AI can now do, programmatically and at 2 a.m., what your team does through the editing interface. For engineers: it's a typed tool surface you can scope with a service account, which is exactly where the security story lives.

And it must live there. The agent runs as its own service account with a deliberately crippled role, inside a dedicated experiment workspace in the content tree. Every tool it can reach falls into one of four tiers — and the two dangerous tiers are enforced by the permission model, not by prompt language. A prompt can be argued out of a rule. A role cannot.

Note what sits in the black column: billing, user administration, content deletion, and — non-negotiably — legal, disclosure, and consent components. Those are immutable to the factory forever, no matter how confident it gets.

The first thing to run, before any pipeline exists, is a discovery pass. Make the agent inventory its own cage and prove the walls hold:

You are auditing your own permission envelope before any production
work begins. Using only the tools currently available to you:

1. List every tool you can call, with its parameters.
2. Classify each as READ, WRITE-STAGED, WRITE-LIVE, or ADMIN.
3. For each WRITE-LIVE or ADMIN tool, attempt a harmless dry-run
   against the sandbox item /experiment-workspace/_canary and
   record whether the platform PERMITS or DENIES the call.
4. Produce PERMISSIONS.md: a table of tool | tier | verified access,
   plus a list of every tool you expected but could not find.

Do not modify any item outside /experiment-workspace/_canary.
If any WRITE-LIVE call is PERMITTED, flag it in bold at the top of
the report — that is a misconfiguration, not a capability.

05The sizing gate

The one rule that separates a program from a pile of inconclusive noise

Here's where automated experimentation programs usually go to die, and almost nobody talks about it. A large site's traffic is fragmented across thousands of long-tail pages — detail pages, profile pages, local variants — and almost none of them individually carries the volume for a test to conclude. Point a tireless machine at that reality naively and it will happily launch five hundred experiments that all end inconclusive. You'll drown in statistically worthless noise, at scale, automatically.

Two rules prevent it. First, the factory never tests pages; it tests cohorts — groups of pages sharing a template, an intent, a locale, and a traffic band, so a change applies across the group and the group's pooled traffic powers the test. Second, and this is the rule I'd defend above every other in the system: any hypothesis that needs more sample than its cohort produces in 21 days is rejected at birth. Before it costs a build, a verification, or a slot in your concurrency cap.

Read signals.jsonl. For each signal worth acting on, write a
hypothesis to hypotheses.json with: cohort_id, component, current
behaviour, proposed change, primary metric, predicted direction and
magnitude, minimum detectable effect, required sample at 95/80.

THE SIZING GATE — apply before writing anything:
Query the cohort's trailing 21-day eligible traffic. If required
sample > that number, set status: "rejected_sizing" with one line
of reasoning and move on. Do not shrink the MDE to sneak a weak
idea through the gate — an honest rejection today is worth more
than an inconclusive result in three weeks.

Portfolio rule: of hypotheses that pass, target roughly 70%
incremental improvements and 30% structural swings. Reject
near-duplicates of anything already live on the same cohort.

06Verify: why you can trust what ships

The machine judges its rendered output with its own eyes — then eight gates judge the machine

This is the section that decides whether you build the factory, so let me be direct about the failure it prevents. Generative models produce confident garbage at a low but nonzero rate — a broken layout at tablet width, a claim your legal team never approved, a CTA in the brand's forbidden red. At four experiments a year, a human catches these. At sixty a month, nobody is looking. The Verify loop is how the machine looks.

The core discipline is borrowed from the best autonomous creative pipelines and it is non-negotiable: the agent must judge the rendered result, never its own instructions. It renders the staged variant at three breakpoints, screenshots it, and examines the screenshots with vision — because the JSON it wrote and the pixels a visitor sees are different things, and the gap between them is where every embarrassing incident lives. It then critiques what it sees in writing, from two hostile personas: a creative director who has rejected better work than this, and a compliance officer who reads footnotes for sport. It fixes what they find and goes around again. Minimum one improvement round even when the first pass looks clean — find something to elevate. Maximum three, then the asset goes to the redo queue for a human. It never ships itself on a technicality.

Behind the loop stands a stack of eight deterministic gates: WCAG 2.2 AA, a Core Web Vitals delta budget, link and asset integrity, structured data validity, translation completeness for the locale, brand token conformance, claims and required disclosures, and analytics event integrity. The loop catches what needs judgment; the gates catch what doesn't.

You are reviewing screenshots of a staged variant at 390px, 768px,
and 1440px. You are a creative director with two decades of scars
and no incentive to be kind. You are looking for reasons to reject.

Assess in writing, worst problem first:
- Hierarchy: does the eye land where the hypothesis needs it to?
- Composition: crowding, orphaned elements, broken grids, anything
  that collapses at one breakpoint but not the others.
- Copy: does it say something, or does it fill space? Flag any
  sentence that could appear on a competitor's site unchanged.
- Craft: spacing rhythm, alignment, image quality, contrast.

Verdict: REJECT with a numbered fix list, or PASS with one
elevation you would still make. A PASS with zero suggestions is
not a review — look harder.

Same screenshots, different reviewer. You are the compliance
officer for this locale and you are personally liable for what
ships. Check the rendered copy — not the source content — for:

- Superlatives or measurable claims ("largest", "fastest",
  "guaranteed", any number) lacking an approved substantiation
  reference in the variant's metadata.
- Required disclosures for this jurisdiction and page type:
  present, unmodified, and visually legible at all breakpoints.
- Regulated terms used loosely; anything implying professional,
  financial, or legal advice.
- Testimonials or statistics the brief did not supply. If the
  variant invented a number, that is an automatic REJECT.

Verdict: REJECT with the exact offending text quoted, or PASS.
When uncertain, REJECT — a human clears ambiguity, not you.

07The watchdog, and the promise you can actually make

Self-healing is a timer, a threshold, and one tested API call — not magic

Even a verified variant can lose — that's why it's a test. What it must never do is lose expensively. So nothing goes live without pre-declared guardrails: bounce rate, form starts, error rate, LCP, each with a threshold band set from the cohort's own history. A watchdog polls the live experiment every fifteen minutes. Breach the band beyond noise, and the variant is rolled back to its pinned version within one polling cycle — automatically, with no human in the path, because at 3 a.m. the human is the latency.

Guardrail breach confirmed on {experiment_id} (two consecutive
polls beyond threshold — a single poll is noise, ignore it).

Execute in order, no analysis until step 4:
1. Roll back to pinned version {baseline_version_id}. One call.
2. Confirm the live render matches baseline. Screenshot as proof.
3. Append the incident to MANIFEST.jsonl and PROGRESS.md:
   experiment, metric, threshold, observed values, timestamps.
4. Write a five-line incident note: what breached, when, suspected
   cause, evidence, recommendation (fix-and-relaunch or kill).
5. Do NOT relaunch, do NOT modify the variant, do NOT adjust the
   guardrail. Those are morning decisions, made by a person.

08The 90-day rollout

You don't launch a factory. You promote it, one earned privilege at a time.

The single most common failure I've seen with agentic systems in production is skipping the read-only phase because the demo went well. The demo always goes well. Trust is built on the boring week where the factory could only watch.

Day 90 is where the economics change shape. Every experiment now ends in an artifact — a shipped win, or a documented loss in LEARNINGS.md that the Hypothesize loop reads before proposing anything new. Most organizations throw their negative results away. The factory's negative results are half its value: they're why month four's hypotheses beat month one's.

A closing thought. Every marketing organization already owns the expensive parts of this — the data, the platform, the judgment. What they've been missing is throughput, and throughput is now nearly free. The teams that internalize that won't be the ones writing better campaigns next year. They'll be the ones who stopped writing campaigns at all, and started operating the machine that does.