May 2, 2013 11:13:00 AM

Determining Statistical Significance in Marketing Campaigns

Posted by Matthew Campion

Find me on:

Questions | Statistical Significance

What is important about statistical significance in marketing?

In search marketing, we rely on captured data to guide us in our decision making.  Based on key performance indicators (KPI’s) such as click through rate (CTR), conversion volume, conversion rate, cost per order or acquisition (CPO/CPA), and return on ad spend or investment (ROAS/ROI),  we gauge the overall success of a paid search program against that program’s goals.  Usually in e-commerce that is driving sales, revenue, and return.

Knowing how much data we need to capture before making changes that can impact our KPI’s is known as statistical significance.  It is how we answer the question, “at what point is this data significant enough to be a solid indicator of performance with a less than 5% chance of being explained purely by randomness?”  This is a huge challenge, and while there are a lot of tools, articles, and information out there about split or A/B testing with a control against a test group, very little information has been published around statistical significance of a brand new campaign or program for which no baseline and no control group exists.

What’s the risk of not understanding statistical significance when making changes to a new campaign?

The risk of making changes too soon is you risk changing or pausing something that may perform and just hasn’t had enough exposure yet.   When there are just a few impressions or clicks, it is hard to know if your ad is not appealing or if it just has not had enough exposure to potential customers to view and click the ad. 
Pausing terms based on spend alone does not take into consideration that a new campaign will lack history. In order to build a good quality score and establish a stronger CTR, it is recommended to start with a higher bid.  Looking at a relatively few number of clicks that came in at a higher cost than you will be paying down the line after some optimization again.

So how do I know when to make changes to a newly launched campaign?

Most marketers and search analysts agree that 100 is the minimum number of clicks needed to gain enough statistical significance in marketing campaigns before making changes, and that 50 -75 is considered low to aggressive. If you can afford it, more is better and some marketers prefer 200 or more clicks before making changes.

What you need to do is understand what makes sense by product line based on profit margin and how much you are willing to invest.

A product with a higher profit margin may be worth the larger investment in PPC to test performance because the higher profit margin gives you better headroom.  A product with a slim margin, however, will likely be one you need to be more aggressive about in determining if it is worth waiting for 100 clicks before you make adjustments to that product’s corresponding PPC terms and ads

Remember that CPC is also a factor.  With a lower CPC – say around $0.25 per click - it is a lot easier to wait until 500 clicks because that translates to just $125 in media spend, whereas a $2.50 per click term would need to spend $250 before reaching a mere 100 click baseline – a vast difference in cost.  Multiply that by the number of ad groups (assuming you are grouping together terms at the product level and looking for clicks at the ad group level) and you have a pretty significant accumulation of testing funds. 

To determine your estimated CPC, use history from one or more non-branded campaigns (or ad groups depending on how granular you want to get) that are expected to perform similarly to your new campaign and ad groups.  Calculate the average CPC by the number of clicks you are using to set your baseline to determine your total investment cost per ad group or new campaign.

What does this mean to my overall performance numbers?

When you have determined what you can afford to test and the initial investment needed to launch the new campaign/ad groups, make the assumption that worst case scenario is 0 orders.  Also keep in mind that the initial investment is your baseline for you to make ANY changes.  Chances are, you are going to want to make a few tweaks before calling it a win or loss. Therefore, you may want to double (or if you can afford it, even triple) that investment number for each round of changes you plan on making, assuming you need 100 clicks from each major change to understand if the changes have a positive impact on performance.  Remember, that if you lower CPCs along the way, the total investment cost may come down, but it is always better to estimate high.

So taking the total planned investment, add that to your current average daily budget and calculate estimated performance without any additional sales.  That will give you an idea as to how the new campaigns will impact overall performance.  Again, you may need to make adjustments here based on volume and opportunity.  If you need to instill budget caps, or your campaigns have lower volume, the total investment for the new campaigns may be spread out over more than one month, higher volume campaigns will accrue volume more quickly and allow you to gain learnings faster. 

This should give you a good idea of where to start and help you and your clients budget for new campaign launches. It also will prevent anyone from wanting to make knee-jerk reactions at the cumulative spend level when they realize no one ad group has spent over $25 so far, nor have they gotten more than 10 clicks, despite the total of the new campaign reaching several hundred dollars within a few days of launch.  Long term, understanding statistical significance in marketing campaigns will give you the best chance for success, and the understanding of performance necessary to know what terms to cut and what to keep for the highest ROAS! 

Unlock the ROI of Your Marketing with Analytics

           comments powered by Disqus

Search Blog

Subscribe to Email Updates

Latest Posts

Most Popular Posts




"Teknicks was able to create, launch, and manage a fast paced social marketing link building campaign that helped to significantly improve Google rankings for a wide array of topic areas."

Director, Search Marketing


< View all>


Teknicks ClientLogos3 resized 179


as seen in resized 179