Web Analytics: Forecasting the Impact of Change

BYJason Burby
June 20, 2006
Over the past year, I‘ve spoken andwritten aboutmonetizing and prioritizing opportunities identified through analyticsto ensure your efforts are focused on the right opportunities.
Partof prioritizing opportunities is forecasting how much you can increaseor lower a metric. I‘m often asked, "How can I determine potential liftso I can prioritize opportunities based on the greatest impact to mybusiness?"
Before you can forecast the lift, however, you must assign a value to the desired site behavior. For example:
The value of a lead for a lead-generation site
The value of an additional sale on a commerce site (you maywant to look at this from a profit standpoint, not just a revenuestandpoint)
The value of deflecting calls to the call center
The value of the use of locating a store
The value of newsletter signup or other registration
The value of a page view, primarily for sites that primarily generate ad revenue
Theseare just a sampling of desired behaviors you may want to define. Stepback and think about what behaviors visitors can perform on your sitethat positively affect your business. Yes, this is a very selfish wayto look at a visit, and we always want to consider what‘s importantbased onvisitor needs, but starting this way allows you to understand yourbusiness drivers‘ value.
Onceyou‘ve identified the desired behaviors and assigned a value to each,you can forecast the changes‘ impact. Unfortunately, there isn‘t aspecific equation to determine how much you can realistically lift aspecific key performance indicator (KPI).
First,learn everything you can about visitor behaviors around the desiredbehavior you are trying to affect. We break this research up into threecategories:
Behavioral.Through Web analytics, we can understand where people are going, wherethey came from, where they drop out, and what might help them toconvert or stop conversion.
Attitudinal. Through surveys, attitudinal studies, orusability groups, we can begin to get more of an understanding of thebehaviors we see.
Competitive. Using competitive data from the likes ofcomScore, Hitwise, and Nielsen//NetRatings, we can see how ourconversion or flow compares to others in the space.
Whenyou look at this data, it‘s helpful to look at visitor segments. Youmay find what works for some visitor segments won‘t work for others.You don‘t want to change something that‘s already working really wellfor one group.
Withmore detail in the three categories of understanding, you should beable to better understand the problem behind the desired behaviors youare trying to improve. Start off with conservative changes.
Let‘ssay your site focuses on generating leads and currently has avisit-to-lead conversion rate of 3 percent. You do some research(behavioral, attitudinal, and competitive) and find a few potentialissues. You wouldn‘t want to forecast the impact for 5 percent.Instead, you select a rangebased on what you see in the research, then tune that over time as yourun tests. You may put a range together, such as 3.25 percent to 3.75percent in 0.1 percent increments, to understand the value. Again, youcan tune this over time, but from a prioritization standpoint you wantto beconservative and look at a range you think you can realistically hitthrough one or two tests.
Aword of warning: Don‘t increase visit-to-lead conversion in the aboveexample at the cost of reducing lead quality. You can surely increaseconversions by giving away an iPod to every fifth person who registers.But you will most likely be driving unqualified leads, greatly reducingleadvalue. You must consider and measure the outcome of such things.
Asyou do more of these, you‘ll get better at determining the potentiallift based on the contributing factors you see during your research.You will be able to tighten the forecast range in terms of change.
Thekey is to start prioritizing opportunities based on monetized valuesand quickly move into testing different ideas through either simple A/Btests or more advanced multivariate tests using tools like Offermatica.Forecast the potential lift you think you can realize from thedifferentopportunities. You‘ll nail some and miss others, either forecasting toohigh or too low. That‘s OK. Small changes can often lead to a bigresult in terms of monetized value. You forecast change to helpprioritize your opportunities based on the greatest impact, as well asto help get yourorganization to realize the potential in opportunities that are beingidentified and to drive them to act on those opportunities.
Shoot me an e-mail and let me know how it goes.