Headline A/B Significance
Calculate conversion lift and a pooled two-proportion z-score for two headlines.
Description
A headline A/B comparison uses conversion proportions and a pooled standard-error model.
A headline A/B comparison uses conversion proportions and a pooled standard-error model. Its z score measures second-minus-first rate difference under that model.1
Calculate conversion lift and a pooled two-proportion z-score for two headlines.
When to use Headline A/B Significance
Use this tool to calculate conversion lift and a pooled two-proportion z-score for two headlines.
Prepare the text, recorded observations, aligned figures or editorial criteria requested. Confirm what each label, count, timestamp or supplied flag represents before interpreting the result.
How to use Headline A/B Significance and understand its results
- Prepare the text, recorded observations, aligned figures or editorial criteria requested. Confirm what each label, count, timestamp or supplied flag represents before interpreting the result.
- Enter firstViews, firstConversions, secondViews, secondConversions, absoluteZThreshold using the field table and calculation rules, then run the tool.
- Read firstConversionPercent, secondConversionPercent, liftPercent, zScore, meetsThreshold. The explanation below defines the units, signs, ordering and any flags or placeholder values.
- If the tool reports an input issue, correct the values and run it again. Apply the stated assumptions when interpreting a successfully returned result.
| Input field | Meaning | Value type; see rules below | Requirement |
|---|---|---|---|
| firstViews | First headline views | integer; minimum: 1 | Required |
| firstConversions | First headline conversions | integer; minimum: 0 | Required |
| secondViews | Second headline views | integer; minimum: 1 | Required |
| secondConversions | Second headline conversions | integer; minimum: 0 | Required |
| absoluteZThreshold | Absolute z-score threshold | number; minimum: 0 | Required |
Enter positive integer views, integer conversions between zero and views, and nonnegative z threshold. Return rates, signed relative lift, z and inclusive absolute-z gate. Zero SE gives z=0; both zero rates give lift=0, but zero first rate with positive second rate is an input error.
SE=√(pooledP(1−pooledP)(1/n1+1/n2)); z=(p2−p1)/SE Units, indexing, boundary conditions and exceptional cases are defined in the calculation rules above.
| Output field | Meaning | Value type |
|---|---|---|
| firstConversionPercent | First conversion rate (%) | number; minimum: 0; maximum: 100 |
| secondConversionPercent | Second conversion rate (%) | number; minimum: 0; maximum: 100 |
| liftPercent | Relative lift (%) | number |
| zScore | Two-proportion z-score | number |
| meetsThreshold | Meets z-score threshold | boolean |
{
"input": {
"firstViews": 1000,
"firstConversions": 100,
"secondViews": 1000,
"secondConversions": 130,
"absoluteZThreshold": 1.96
},
"output": {
"firstConversionPercent": 10,
"secondConversionPercent": 13,
"liftPercent": 30.000000000000004,
"zScore": 2.102740605622114,
"meetsThreshold": true
}
}Limitations and assumptions
No p value, randomization check, multiple-testing correction or causal conclusion is supplied.
Numeric inputs must be finite. Follow the calculation-specific rules for overflow, underflow and exceptional cases. Missing required fields, extra fields, wrong value types and violations of the calculation rules are input errors. A successfully returned false flag is an evaluated outcome; its meaning is explained above.
Alternative or Complementary methods
Use Read Completion Rate: Calculate read completion rate from the supplied measurements. For a complementary task, use Open-Rate Inflation: Adjust reported email opens for estimated machine-generated opens.
References
-
Headline A/B Significance — tool.io
Similar or alternative tools
- Read Completion Rate
Calculate read completion rate from the supplied measurements.
- Open-Rate Inflation
Adjust reported email opens for estimated machine-generated opens.