A/B testing
Run statistically sound campaign A/B tests with tri-state significance, lift, p-values, and one-click winner promotion.
A/B testing in Pact tells you not just which variant is ahead, but whether you can trust the result. It's pure statistics — no AI is involved.
Where to find it
Open a campaign's A/B test view at /marketing/campaigns/[id]/ab-test. A per-variant table breaks down every arm of the test.
Read the table
Each row represents one variant and shows:
- A leader badge on the arm that's currently ahead.
- A tri-state significance status: reached significance, in-flight, or underpowered.
- Lift, color-coded by sign so gains and losses are obvious at a glance.
- A p-value.
- A Winner badge — but only when the test is significant and that arm is the winner.
Set your confidence
A confidence selector defaults to 95% and ranges from 80% to 99.9%. Raise it to demand stronger evidence before a winner is declared; lower it to call results sooner.
Call the winner
When a test is decided, Call the winner promotes the winning variant.
How significance is calculated
Pact uses a Yates-corrected chi-square test. A variant is declared the winner only when:
- the p-value is below (1 − confidence), and
- the lift is above 5%.
Small tests stay inconclusive
Below 100 trials, significance is never declared — no matter how large the apparent lift looks. This prevents calling a winner on noise.
Variants in sequences too
Sequence send steps support their own A/B variants, so you can test subject lines and copy inside automated outreach as well.
It's statistics, not a model
Every figure here comes from a deterministic statistical test. No AI is in the loop.