Chapter 7: Test Duration and Statistical Rigor

Determining appropriate test duration and ensuring statistical validity are critical components of successful MMT implementation. This guide covers duration guidelines by campaign type, power analysis requirements, and validation frameworks for reliable results.

Recommended Test Durations by Campaign Type

Lower Funnel Tactics: 4-8 Weeks

Examples: Google Search, Shopping, Meta retargeting

Rationale: Quick conversion cycles with immediate measurable impact
Considerations: Shorter tests are acceptable due to direct response nature

Characteristics of Lower Funnel Tactics:

• High intent audiences with short consideration periods
• Direct correlation between exposure and conversion
• Immediate measurable impact within days of launch
• Less influenced by external factors and seasonality

Minimum Duration Factors:

• At least 2 full conversion cycles
• Sufficient volume for statistical significance
• Account for weekly seasonality patterns
• Allow for campaign optimization period

Higher Funnel Tactics: 8-12 Weeks

Examples: YouTube, Connected TV, Display advertising, Meta prospecting

Rationale: Longer consideration periods and delayed conversion impact
Considerations: Allow time for awareness to translate into measurable outcomes

Extended Duration Requirements:

• Awareness Building: Time for brand awareness to develop
• Consideration Period: Longer path from exposure to conversion
• Frequency Building: Multiple exposures needed for effectiveness
• Attribution Complexity: Delayed and assisted conversions more common

Top-of-Funnel/Awareness Campaigns: 8-12+ Weeks

Examples: Brand awareness campaigns, influencer partnerships, sponsorships

Rationale: Extended impact periods with significant carryover effects
Considerations: May require additional “cooldown” measurement periods after test conclusion

Unique Characteristics:

• Carryover Effects: Impact continues beyond campaign end
• Delayed Attribution: Conversions may occur weeks after exposure
• Brand Building: Long-term effects on brand perception and recall
• Complex Attribution: Multiple touchpoints influence final conversion

Factors Influencing Duration

Sales Cycle Length

B2B Considerations: Longer sales cycles require extended test periods to capture full impact
High-Consideration Purchases: Big-ticket items need time for research and decision-making
Subscription Models: Focus on trial-to-paid conversion timing

Attribution Windows

Platform Attribution Windows: Tests must be longer than platform’s maximum attribution window
Cross-Channel Attribution: Account for multi-touch attribution complexity
Offline Attribution: Include time for online-to-offline conversion tracking

Seasonality Considerations

Avoid Peak Seasonal Periods: Testing during extreme seasonality can skew results
Account for Business Cycles: Consider monthly, quarterly, or annual business patterns
External Events: Avoid testing during major industry events, holidays, or known disruptions

Pre-Test Power Analysis

Understanding Statistical Power

Pre-test power analysis determines the minimum detectable effect size, ensuring the test is designed to yield statistically significant results.

Key Components:

• Effect Size: Minimum meaningful difference you want to detect
• Statistical Power: Probability of detecting a true effect (typically 80% minimum)
• Significance Level: Probability of false positive (typically 5%)
• Sample Size: Number of observations needed for reliable detection

Power Analysis Implementation

# Example power analysis using R

library(pwr)

 

# Calculate required sample size for desired effect detection

power_analysis <- pwr.t.test(

  d = 0.2,           # Effect size (Cohen’s d)

  sig.level = 0.05,  # Significance level

  power = 0.8,       # Statistical power

  type = “two.sample”

)

 

print(paste(“Required sample size per group:”, ceiling(power_analysis$n)))

Minimum Detectable Effect (MDE)

Calculation Factors:

• Historical performance variance
• Test market size and control group composition
• Desired confidence level and statistical power
• Business relevance threshold for meaningful impact

Practical Considerations:

• Smaller detectable effects require larger sample sizes or longer durations
• Business significance may differ from statistical significance
• Cost of extended testing vs. precision of measurement trade-offs

Ensuring Test Validity

Cooldown Periods

We can read the results 3-12 weeks after the test concludes. During this extra time period, spending has returned to normal. Additional reads during this time can improve results and increase confidence intervals.

Why Cooldown Periods Matter:

• Carryover Effects: Marketing impact may continue after campaign ends
• Attribution Delays: Some conversions occur weeks after initial exposure
• Market Stabilization: Time for markets to return to baseline performance
• Data Validation: Additional time improves statistical confidence

Cooldown Duration by Campaign Type:

• Lower Funnel: 2-4 weeks cooldown sufficient
• Higher Funnel: 4-8 weeks cooldown recommended
• Top-of-Funnel: 8-12 weeks cooldown critical for full impact measurement

Reset Periods

After a market is used in a test, we enforce a reset period of 90 days during which it cannot be included in subsequent tests as a test or control market.

Reset Period Requirements:

• Market Recovery: Time for market to return to natural baseline
• Carryover Elimination: Ensure previous test effects don’t influence new tests
• Consumer Behavior Normalization: Allow for audience behavior to stabilize
• Data Quality: Prevent contamination between sequential tests

Quality Assurance Metrics

Statistical Confidence Validation

Key Performance Indicators for Test Validity:

Market Correlation Metrics:

• Main KPI Correlation: >0.5 minimum, >0.7 preferred between test and control markets
• Correlation Stability: Consistent correlation across multiple time windows
• Log-Correlation: Often examined to account for size differences between markets

Synthetic Control Quality Scores:

• Causal Impact (CI) Score: <0.7 indicates reliable predictive capability
• Pre-Test RMSPE: Lower values indicate better synthetic control fit
• Weight Distribution: Balanced allocation across multiple control markets preferred

Data Quality Indicators

Completeness and Consistency:

• Completeness: <5% missing data points across measurement period
• Consistency: Stable reporting methodologies across test and control markets
• Outlier Detection: Identification and treatment of anomalous data points that could skew results

Volume and Coverage Metrics:

• Test Market Coverage: Typically 10-15% of total business volume
• Minimum Volume Thresholds: At least 10 conversions per day or $500 daily revenue per test market
• Control Pool Size: Minimum 10-15 potential control markets for robust synthetic control creation

When to Pause Tests Early

Performance-Based Termination

Holdout Tests: If revenue drops significantly more than expected based on channel’s estimated incrementality

Growth Tests: If no measurable lift appears after sufficient time for the channel’s typical conversion window

Risk Thresholds: When potential business impact exceeds acceptable risk levels

External Disruption Factors

Major Promotional Events: Unplanned promotional activity affecting test validity
Supply Chain Issues: Product availability problems impacting conversion ability
Competitive Actions: Major competitive campaigns or market disruptions
Platform Changes: Significant platform algorithm or policy changes

Early Termination Decision Framework

1. Performance Threshold Assessment: Compare current results to expected ranges
2. Business Risk Evaluation: Calculate potential revenue impact of continuing
3. Statistical Significance Review: Determine if early results are directionally reliable
4. External Factor Analysis: Identify any factors compromising test validity
5. Stakeholder Alignment: Ensure business alignment on termination decision

Statistical Significance Interpretation

Confidence Level Standards

95% Confidence Level: Our standard threshold for making investment decisions
80-94% Confidence: Suggests positive trends but may require additional validation
Below 80% Confidence: Results likely due to random variation rather than marketing impact

Progressive Statistical Monitoring

Weekly Significance Checks: Monitor p-values and confidence intervals throughout test period
Trend Analysis: Track directional consistency even before reaching full significance
Early Signal Detection: Identify strong positive or negative trends for potential early action

This rigorous approach to test duration and statistical validation ensures MMT results provide reliable, actionable insights for marketing optimization and scaling decisions.

Next Steps: Learn how MMT integrates with Media Mix Modeling

Next Chapter