Chapter 1: Understanding MMT Test Types and When to Use Them

When planning a Matched Market Testing strategy, choosing the right test type is crucial for getting actionable insights that align with your business objectives. There are two primary approaches, each designed to answer different strategic questions about your marketing performance.

Holdout Tests: Measuring Current Effectiveness

Holdout tests measure the incremental impact of existing media spend by temporarily reducing or pausing campaigns in test markets while maintaining normal spend in control markets.

These tests answer the question: “What is the current effectiveness of our marketing investment?”

Key Characteristics

• Purpose: Validate existing channel performance and identify potential waste
• Risk Profile: Low to moderate risk, limited by test market size
• Best For: Established channels with steady spend, validating current performance
• Example: Pausing Google Search ads in Denver while maintaining normal spend in similar control markets

When to Use Holdout Tests

Holdout tests are ideal when you need to validate whether your current marketing investments are truly driving incremental value. This is particularly valuable for:

• Established channels where you suspect diminishing returns
• Budget reallocation decisions
• Quarterly or annual performance reviews
• Justifying continued investment in specific tactics

Risk Calculation for Holdout Tests

Maximum potential revenue impact = (Test Market Size %) × (Channel Incrementality %) × (Total Revenue)

Example: Testing in markets representing 10% of revenue, with a channel driving 40% incremental sales, maximum risk = 10% × 40% = 4% of total revenue.

Growth Tests: Exploring Scaling Opportunities

Growth tests evaluate the potential for scaling by increasing media spend in test markets beyond current levels.

These tests answer: “Can we efficiently invest more in this channel or tactic?”

Key Characteristics

• Purpose: Identify scaling opportunities and test new channels
• Risk Profile: Higher risk due to upfront investment without guaranteed returns
• Best For: Net-new channels, underinvested tactics, or validating expansion opportunities
• Example: Increasing TikTok ad spend by 200% in Phoenix while keeping control markets unchanged

When to Use Growth Tests

Growth tests help you identify where additional investment can drive profitable returns:

• Testing new advertising channels or platforms
• Validating whether successful tactics can be scaled
• Exploring seasonal or promotional lift opportunities
• Supporting budget increase requests with data

Growth Test Risk Management

• Incremental Budget Approach: Start with smaller increases to validate before larger investments
• Performance Monitoring: Daily tracking allows for quick optimization or termination
• ROI Thresholds: Establish minimum acceptable returns before launching growth tests

Choosing the Right Test Type

The decision between holdout and growth tests depends on your current situation and business objectives:

Choose Holdout Tests When:

• You need to validate existing performance
• Budget constraints require proof of current channel effectiveness
• You suspect potential waste in current spend
• Risk tolerance is low

Choose Growth Tests When:

• You have additional budget available for testing
• Early indicators suggest scaling opportunities
• You’re exploring new channels or tactics
• You can absorb potential losses from unsuccessful scaling

Test Timing Considerations

Both test types require careful timing to avoid external factors that could skew results:

• Avoid major promotional periods or seasonal peaks
• Consider your industry’s natural business cycles
• Account for any competitive activity or market changes
• Plan around internal operational changes

Understanding these fundamental test types provides the foundation for designing MMT experiments that deliver actionable insights for your specific marketing challenges and growth opportunities.

Next Steps: Ready to determine your minimum investment requirements? Learn about MMT Budget Planning and Spend Thresholds.

Next Chapter