Product AI agent · Experimentation
Experiment & Analytics AI Agent
Assign no user to a variant that varies a price, a consent or a credit term without recording what it varied, and hold each readout for the named analytics owner who calls it.
- Typical delivery
- 4–6 weeks
- Deployment
- Your stack
- Analytics owner
- Outcome-scoped
- After launch
- Agent Care
What this agent does
Runs the experiment, never calls it
In
- 01
A variant is assigned, and what that variant varies is recorded beside the assignment.
- 02
A metric is chosen before launch, and the day it was chosen travels with the readout.
Reason
- 03
A price variant is flagged, since New York has required a stated disclosure since November 2025.
- 04
A consent variant is measured on path length both ways, not on the opt-in rate alone.
- 05
A credit term is held back, since Regulation B treats a varied term as an adverse action.
Decide
- 06
An assignment log is kept, because the FTC names A/B testing in its retention orders.
- 07
A readout is written to the standard the EDPB invites as evidence of compliance.
Out
- 08
A test cannot be run without varying something a rule governs, and that is raised as the finding.
- 09
Execute write actions only inside the approval boundaries agreed during implementation.
Product statement
Design, assignment, instrumentation and readout belong to the agent. Calling a test belongs to a named analytics owner, who takes the variant into the product and owns it there.
Example workflow
One experiment, design to call
- 1Test request receivedA hypothesis, a surface, a target metric, an audience definition or a rollout request
- 2Exposure scopedThe surface it changes, what each variant varies, who may be assigned and the markets they sit in
- 3Design and assignment draftedThe variants, the assignment rule, the primary metric and confidence
- 4Controls appliedOutcome checks, exposure checks, guardrail-metric checks and design confidence
No human action required
Stages 1 to 4 run unaided, and no variant reaches a user at any of them — the agent is designing, and the owner lane opens at the launch gate.
Goes to the named owner to call.
Adds a legal and privacy read first.
Owner review
The test is held with its design, what each variant varies and the surfaces it would touch.
Launch · Amend design · Send to legal review
Called — by the named owner
- 6Assignment and warehouse records updatedOnly where write access and release policy allow it
- 7Outcome evaluatedExposure accuracy, guardrail breaches, owner amendments and what review found
Amendments
Each amendment an owner makes is scored in the evaluation.
What should not run autonomously
Human approval stays in control
Outside the boundary — human approval required
8 items
- Calling a test and shipping the winning variant.
- Deciding that a price may vary between users.
- Approving a consent flow for release.
- Varying a credit term or a safety behaviour.
Automation boundary
Agent acts unaided
- ✓Record what each variant varies and the rule it touches.
- ✓Raise a flag when a test varies a price, a consent or a credit term.
- ✓Hold the design for the named owner who calls it.
- ✓Report the peeks, metric changes and early stops in the readout.
No variant reaches a single user except by a named owner, inside the agreed boundaries.
- Judging whether a result may be stated as a claim.
- Declaring an experiment outside a regime.
- Setting the metric the business optimises for.
- Changes to assignment, rollout or guardrail rules.
Example output
One experiment, annotated
Our campaign analytics agent measures what a campaign did; this one deliberately gives some users a different product to find out, and below is one test exactly as the agent leaves it.
Experiment design · single test
Illustrative example- Test
- Checkout copy, two variants
- What it varies
- Wording only, no price moves
- Primary metric
- Completed checkouts
- Exposure of record
- Design filed, 6 August 2026
- Confidence
- Held unlaunched
- Held for
- The named analytics owner
As received
Built from the design and the assignment log; a random draw evaluates nobody, and a targeted one does.
What the record holds
- Assignment log
- Exposure snapshot
- Guardrail metrics
Why no launch here
Calling a test and shipping the winner is a judgement an owner makes.
What the score decides
Where a variant touches price or consent the design gets a legal read before the owner.
Action
LaunchAmend designSend to legal reviewValue
Where AI adds value
The same four claims, placed at the point in the workflow where each one applies.
Where the value lands
Value 01 – 04
Every experiment
From the surface it changes
Where the result is used
The agent does not decide that a variant is lawful, only what it varied and whom it reached. When the winner has already shipped, the correction is filed against the release that carried it.
The winning variant is evidence
Ask which outcome a test varies. Copy, layout and ranking bind almost nobody: Robinson-Patman reaches commodities, not services, and Article 22 misses a cohort draw. Vary a price or a consent and the test is the evidence.
What was checked, and not found
No rule was found requiring a test on your own users to be powered, pre-registered, honestly analysed or reported at all, and none governs peeking or calling a winner early. The Common Rule follows federal funding, and Annex III does not reach product experimentation.
Build an evidence trail
The experiment, the metric it optimised and the person who called it stay together.
Integrations
Typical integrations
Five system groups connect to the same agent. Which of them are in scope is decided in discovery.
Experimentation platforms
- Optimizely
- LaunchDarkly
- Statsig
- Eppo
- Split
- GrowthBook
Product analytics
- Amplitude
- Mixpanel
- Event streams and exposure records
Warehouse and modelling
- Snowflake
- BigQuery
- Databricks
- Modelled metrics and cohort tables
Agent
Experiment design and readout
- Reads the surfaces
- Designs the test
- Holds for the owner
Consent and delivery surfaces
- Consent platforms
- feature flags
- Banner variants and rollout state
Observability & evaluation
- OpenTelemetry
- Langfuse
- Supported monitoring/evaluation sources
Integration availability depends on the client's existing systems and API access.
Agent controls
Six valves between the design and the user
Six valves on one pipe, the last the tightest. What flows through is set out in the map below.
L6 · Outermost — last line of defence
Inward → L1 · closest to the model
- L1 – L2
- Test whether a design may launch
- L3
- Leaves the call to a named owner
- L4 – L5
- Keep the experiment and the metric behind it
- L6
- Holds the call open when signals degrade
How Nestack evaluates it
Evaluate the whole experiment — not only the variant that wins.
Coverage runs the whole depth of the workflow, and every layer is cut by slice.
Surface — the variant a user is given
- E1Final-output evaluationDid the readout record the metric the test was designed on?
- E2Step-level evaluationDid the agent read the right surface, the right cohort and the live guardrails?
- E3Tool evaluationDid it read and write the correct experiment and the correct cohort?
- E4Confidence calibrationDo low-confidence designs actually attract more owner amendments?
- E5Slice evaluationHow does performance change across specific test surfaces?
- E6Business outcomeHow many tests needed an amendment before the owner called them?
Floor — the outcome a rule already governs
Failure modes
Where each failure originates in the agent
Seven failure modes, each caught at the stage it first shows itself.
Agent lifecycle
Direction of processing →
01 · Retrieval
1 mode
- QJ-03
Stale cohort read
The cohort read is not the one now assigned.
Stage gathers
The surfaces, the cohorts, the metrics and the flags
02 · Reasoning
2 modes
- QJ-04
Winner called, not shown
A variant is promoted with no readout behind it.
- QJ-06
Capped metric optimised
The test wins on the thing a rule limits.
Stage proposes
The variants, the metric and the exposure
03 · Tool / write
2 modes
- QJ-02
Thin design passed forward
A test moves on without the legal read.
- QJ-05
Exposure bound to wrong cohort
The variant is served to another cohort.
Stage writes
Only where write access and approval policy allow it
04 · Output
1 mode
- QJ-01
Called, exposure unrecorded
The record shows the call but not who saw the variant.
Stage returns
The variant a user is given and a regulator reads
05 · Change / Version
1 mode
- QJ-07
Silent metric drift
A primary metric changes while the stored readout keeps the old one.
Stage tracks
Model, prompt, design rules and metric versions
- Sev-1 · a variant shipped on no readout
- Sev-2 · a capped metric reaches the call
- Sev-3 · signals degrade, launch held back
Affected slices
Pricing surfaces absorb the amendments
A surface-level exposure figure can read clean while pricing and packaging surfaces carry most of the owner amendments. Nestack reports the amendment rate by test surface, not only in total.
Slice performance — reported separately, not only in aggregate
Illustrative example| Slice | Failure rate | Lift | Lift vs. threshold | Status |
|---|---|---|---|---|
| Pricing and packaging surfaces | 8.9% | 3.7× | Review | |
| Consent and permission flows | 6.3% | 2.6× | Review | |
| Onboarding and activation flows | 3.9% | 1.6× | Watch | |
| Routine copy and layout tests | 2.0% | 0.8× | Normal |
Bar: amendment-rate lift vs. routine-copy-test baseline · scale 0–4.0× · tick marks the 2.0× review threshold
2 of 4 slices over threshold
Evidence-linked improvement
What a winning variant costs
A cycle ends when the consent flow optimised for opt-in rate is a standing case. That suite is what the next test launched is measured against.
Improvement cycle · five stages
Switchback — the path turns at Improve and returns at Learn
- 01
Detect
Amendment rate rises on pricing and packaging surfaces.
- 02
Diagnose
The consent banner that won, and won by making refusal one click longer than acceptance, is worked backwards until one cause is left standing.
- 03
Improve
Each change ships numbered, and the experiments behind it are attached.
- 04
Verify
A single red experiment case is enough to hold the rollout.
- 05
Learn
The case is retained, and the design rules are amended in the same commit.
Learn → Detect
The return edge. The next test is measured against a suite one case longer.
Typical build scope
Twelve workstreams across six weeks
The build scope read against the delivery timeline. Week structure follows the six-week plan — discovery, sources, experiment design, evaluation, integration, then production validation and handover.
Workstream
Week 1Week 2Week 3Week 4Week 5Week 6
- 01Test-surface discovery and automation-boundary work.Week 1
- 02Platform, analytics and warehouse sources.Week 1, Week 2
- 03Variant-to-outcome and assignment-integrity mapping.Week 1, Week 2
- 04Experiment source intake.Week 2
- 05Surface, cohort and metric binding.Week 2, Week 3
- 06Exposure scoring and review routing.Week 3
- 07Owner launch workflow.Week 3, Week 4
- 08Platform and warehouse integration.Week 5
- 09Design and readout cases.Week 4, Week 5
- 10Guardrails and rollout controls.Week 4
- 11Assignment-trail instrumentation.Week 5, Week 6
- 12Deployment, documentation and Agent Care handover.Week 6
12 workstreams · 6 weeks · bar shows the weeks a workstream is active — several run in parallel
Final scope and sequence confirmed in discovery
Engagement tiers
What each tier includes
Rows are the capabilities named in each tier's scope. Higher tiers include everything below them.
| Capability✓ in scope · — not at this tier | PilotOne surface, one test | ProductionProduction experiment workflow | AdvancedMultiple surfaces / markets |
|---|---|---|---|
| Introduced at Pilot | |||
| Experiment design to your rules | ✓ | ✓ | ✓ |
| Named owner launch | ✓ | ✓ | ✓ |
| Experiment-inventory baseline | ✓ | ✓ | ✓ |
| Introduced at Production | |||
| Reporting by test surface | — | ✓ | ✓ |
| Legal review workflow in your systems | — | ✓ | ✓ |
| Approved write-back | — | ✓ | ✓ |
| Assignment-and-warehouse integration | — | ✓ | ✓ |
| Introduced at Advanced | |||
| Multi-surface exposure design | — | — | ✓ |
| Cross-surface experiment packs | — | — | ✓ |
| Large experiment estates | — | — | ✓ |
| Multi-surface exposure controls | — | — | ✓ |
| Build price | From $5,000 | From $8,000 | Custom quote |
- Final build price
- Confirmed after discovery based on integrations, workflow complexity, experiment volume, approval controls and deployment requirements.
- Separate from build
- Build pricing is separate from recurring Agent Care, which covers managed monitoring, evaluations, incidents and verified improvements after launch.
What we need from you
What you bring, and what we build with it
Each input maps to a piece of build scope and a week in the delivery timeline.
You bring
We build with it
01Your live surfaces and what a test may vary on each
→Surface capture and exposure mapping
Week 102Representative past tests and their readouts
→Source binding, design logic and the exposure baseline
Week 203Your test calendar and the owners named against it
→Outcome mapping, surface binding and the automation boundary
Week 104Access to relevant APIs, feeds or exports
→Platform, analytics and warehouse assessment, then integration setup
Week 205Variants you would not want produced
→Readout cases and failure-mode testing
Week 406What no variant may vary
→Exposure scoring, review routing, guardrails and release controls
Week 307A named owner who calls the experiment
→Release to the named owner, before pilot and production checks
Weeks 5–6
Nothing else is required
Deployment, documentation and Agent Care handover are ours.
Delivery timeline
Four phases across six weeks
Band widths follow what each phase costs, which is why the fifth week runs two of them together.
Phase
W1
W2
W3
W4
W5
W6
- DiscoveryW1
- BuildW2 – W3
- EvaluateW4 – W5
- Pilot & LaunchW5 – W6
Week focus
W1Exposure discovery, outcome mapping and the automation boundary
W2Source integration and the experiment-inventory baseline
W3Experiment design, assignment logic and release controls
W4Evaluation suite, readout cases and failure-mode testing
W5Platform integration, pilot tests and targeted corrections
W6One experiment cycle run under the analytics owner, then Agent Care handover
- Reading the band
- Each bar spans only the weeks its own work is named in, and week five carries two.
- At the end of W6
- Once the assignment record validates, Agent Care picks the agent up.
- Duration
- Six-week plan shown · typical delivery 4–6 weeks depending on scope confirmed in discovery.
Next step · Product AI agent
Build an experiment and analytics agent around the outcome your last test quietly varied.
Show us one test you ran last quarter and the metric it was called on. Not the randomisation. The outcome it varied. If that outcome is a price, a consent or a credit term, the winning variant is already the evidence. A test nobody may run comes back as a finding.
- Typical delivery
- 4–6 weeks
- Deployment
- Your live surfaces
- Analytics owner
- Outcome-scoped
- After launch
- Agent Care available
Nestack Agents · Experiment and analytics
AGT-PRD-06 · Agent Care available after launch