Key Performance Indicator (KPI) frameworks

Original publish date: 28th January 2014.

Key Performance Indicator (KPI) frameworks are very important. As far as Web Analytics is concerned, they have been referred to as the “difference between winners and losers” by none other than Avinash Kaushik, the Digital Marketing Evangelist for Google – one of the most influential people in digital marketing.

What KPI frameworks are, and why exactly we need them

In a recent article by DM news (http://www.dmnews.com/infographic-marketers-struggle-to-make-the-data-driven-grade/article/322172/), it was revealed that the average grade marketers received when being asked to evaluate their relationship with data was a worrying “B-”. This serious lack of confidence is in some ways unsurprising. Marketers are overwhelmed by data – there are increasing quantities of information from an ever-widening variety of sources which we have the theoretical opportunity to interrogate, but the practical impossibility of being able to examine.

This has serious ramifications – paraphrasing Avinash: the principle cause of failure for most digital marketing campaigns is the inability to assess this data and to determine its significance. Creating a KPI framework, also referred to as a Digital Marketing and Measurement Model, addresses this by organising desired business outcomes such as “increased brand awareness” with things that we can measure such as “branded traffic”. This is done via various processes which vary based on the nature of the particular type of KPI framework used, but all have the same intended outcome: to expose opportunity by helping us prioritise what metrics we should pay the most attention to. Besides helping to direct optimisation, these frameworks have the additional benefits of helping to align thinking across an organisation, and creating consistency with reporting which makes it much easier to quickly gain a holistic view of all digital marketing activity. Hopefully, you are now intrigued by how this tool can help you improve your “data grade”. In the next section we will help you get started by going through the step-by-step process for generating a full KPI framework.

The process of building KPI frameworks

In the previous section I outlined what KPI frameworks are, and why exactly we need them. To summarise, KPI Frameworks are an antidote to what Avinash Kaushik calls “the root cause of failure” for many digital marketing campaigns, namely “the lack of structured thinking about what the real purpose of [a] campaign is and a lack of an objective set of measures with which to identify success or failure.”.

In this section, I outline a structured process for generating a Top Down KPI framework (one of the several varieties available), borrowing heavily from Avinash’s article on the subject (http://www.kaushik.net/avinash/digital-marketing-and-measurement-model/), and input from the great guys at Lynchpin (http://www.lynchpin.com/). The different stages involved in generating such a framework are as follows:

Identification of business objectives.
Identification of goals for each objective.
Identification of Key Performance Indicators (KPIs) for each goal.
Identification of targets for each KPI.
Identification of important segments to help interpret progress towards targets.
Identification of Critical Success Factors (CSFs).
Achieving final approval.

The example used to illustrate each of these stages is for a website-based freemium game which monetises through subscriptions. This is motivated by Ambergreen’s work with various gaming clients.

Identification of business objectives

What are business objectives?

Business objectives are the answer to the question, “Why does this website/marketing campaign exist?”. Often, the answer to this question is surprisingly difficult to discern, varying significantly from stakeholder to stakeholder.

Why is setting business objectives important?

Formally specifying business objectives is important to make sure there is true buy-in from decision makers into the purpose of the website/marketing campaign, and that the overall focus is correct. It provides a foundation for the rest of the framework which makes it easier to quickly interpret why particular goals/KPIs exist. It is also an early check on unrealistic expectations so long as the objectives are “DUMB”, or:

Doable

The objectives should be within the realm of possibility.

Understandable

The objectives should require little explanation, even to those with a cursory understanding of the website/marketing campaign.

Manageable

The objectives should be achievable given the constraints on resource.

Beneficial

The completion of the objectives should genuinely help the organization that is intended to profit from the website/marketing campaign.

How do you go about determining business objectives?

We recommend going through the following steps when determining business objectives:

Explore the context of the website/marketing campaign

It is important to understand the challenges being faced by the organisation in charge of the website/marketing campaign in order to make sure that the most promising opportunities are considered.

Identify key stakeholders

It is important to make sure that the people who are most involved in the website/campaign are formally identified so that the right people are asked the crucial questions as to what the website/campaign is there to do.

Solicit business objectives from key stakeholders

Once they have been identified, the next stage is to hold discussions with key stakeholders in order to extract what they think the business objectives should be. This is not always easy, as it is often difficult to get hold of senior decision makers, but is very important for discovering true success criteria.

Interrogate the business objectives

It is often useful for whoever is leading the KPI framework generation to identify additional opportunities based on the previous steps. This helps ensure that marketing gets the full credit for what it is responsible for.

Gather business objectives and get sign off from key stakeholders.

Once the business objectives have been collected together into a formal document, getting sign off from the key stakeholders is useful to make sure that subsequent work is not misspent on objectives that lack buy-in.

While going through this process, it is helpful to ignore anything to do with how these objectives are to be achieved. For example, if you immediately get bogged down too soon in the details of planning a PPC campaign targeting generic search terms, it is very easy to get distracted from the fundamental questions regarding why the activity should be done in the first place.

What are some examples of business objectives?

Sample business objectives could be:

Grow brand.
Generate paying players.
Increase Customer Lifetime Value (CLV)

Identification of goals for each objective

What are goals?

Goals are “specific strategies that are necessary to accomplish business objectives” (http://www.kaushik.net/avinash/web-analytics-101-definitions-goals-metrics-kpis-dimensions-targets/#goals). These are the next level breakdown from the top-level business objectives. It is not possible to implement business objectives directly – they are too general. For example, “Generating more awareness” might translate to something like “Reinforce online/offline advertising”.

Why is setting goals important?

Formally identifying goals helps make sure you can see all your options for achieving business objectives. Often people will focus on reducing spend for example, rather than improving conversion rates. Having goals as part of your KPI framework acts as a constant reminder of the different approaches you can take, and thus makes it more likely that resource will be allocated efficiently.

How do you go about determining goals?

The process for generating goals is very similar to that for business objectives, involving discussions with key stakeholders as well as independent research by the analyst generating the KPI framework.

It is important that you focus here on “micro goals” (“the relationship building activities that lead up to a macro conversion” – https://analyticsacademy.withgoogle.com/assets/pdf/DigitalAnalyticsFundamentals-Lesson4.4SettingupgoalsandecommerceText.pdf) as well as “macro goals” (goals that lead directly to your primary business objectives). For example, from a gaming perspective, instead of only thinking about getting players to give money (a macro goal), think in terms of getting players to intermediary stages on the way to giving money, for example, achieving a certain level of social engagement with other players (an example of a micro conversion). this is useful for identifying all opportunities for improvement.

Sometimes, the value of these micro-conversions can be multiples of the macro-conversion For an excellent article on how to determine the value of these micro-conversions, including how to put a value on Youtube views, see http://www.kaushik.net/avinash/web-analytics-tips-identify-website-goal-values/.

What are some examples of goals?

For the Grow brand objective, sample goals might include

Macro – Increase the number of people who come to the website
Macro – Increase the extent to which people come back to the site
Micro – Increase owned channel (e.g. Youtube) reach

Identification of Key Performance Indicators (KPIs) for each goal

What are KPIs?

KPIs are the measures used to determine performance toward goals. It is important to determine the most relevant KPIs for your goals to make sure that a change in those metrics is truly significant in terms of the goal with which it is associated.

Why is setting KPIs important?

Setting KPIs is important because they enable measurement towards goal completion in an objective fashion.

How do you go about determining KPIs?

In order to determine what KPIs to use for a particular goal, answer the following question: “what metrics would give a complete, and specific representation of progress towards my goal?”. This often leads to a series of important questions. For example, are you really interested in pageviews? If someone went to your site to find a particular piece of information, multiple pageviews could be a bad sign, as it required visitors to go through irrelevant content before they found the information they needed. Depending on the scope of your project, you may not be able to use the most appropriate KPIs due to tracking limitations.

What are some examples of KPIs?

For the “Increase the number of people who come to the website” goal, a sample KPI could be:

Increase the number of visitors

Identification of targets for each KPI

What are targets?

To borrow from Avinash: “targets are numerical values you’ve pre-determined as indicators of success or failure”.

Why is setting targets important?

Setting targets helps enforce discipline with the way marketing plans are carried out. We all have far too little time to do everything we want to do in marketing – setting targets makes us carry out cost/benefit analyses on a regular basis, helping us to ensure our time is well spent. For example, if we have to choose between spending time trying to get more visitors from organic search, or improving conversion rates, the knowledge that there is very little incremental traffic available would certainly influence that decision – as would be revealed by the process of generating targets. It also lets everyone know instantly whether marketing is on track or not, which makes it easier for people to buy into this data-driven approach.

How do you determine targets?

There are several sources of information to help set targets.

These include:

Historical performance.
Estimates based on research tools (e.g. using the Adwords keyword planner tool to forecast PPC traffic).
Consulting other parties within the organisation (e.g. Finance, people who have run similar websites/campaigns previously).

If there is insufficient data available to create reliable number-based targets, it is best practice to go through the process of trying to set targets regardless until the first batch of actual data comes in to focus marketing effort.

What are some examples of targets?

For the “Increase the number of visitors” KPI, the target could be

10% increase in visitors per month within 6 months.

Identification of important segments to help interpret progress towards the set targets

What are segments?

A segment is a group united by some set of shared characteristics.

Why is setting them important?

Quoting from Avinash (again): ”all data in aggregate is crap”. Fluctuations in business metrics are caused by groups of certain people doing things. It is important to break down overall numbers to help identify precisely where opportunity lies.

How do you determine which segments to use?

Again, consulting key stakeholders will provide valuable insight into which segments to choose for each goal. To help stimulate this conversation, try asking the following questions:

Who are we trying to attract?
Should we have different outcomes based on the different types of people coming to the website?
What are the key traffic drivers?
Do we expect traffic behaviour to be different by device?

What are some examples of segments?

For the number of visitors KPI, segments could include:

New/returning visitors
Traffic source
Specific referrers
Landing page

Identification of Critical Success Factors (CSFs)

What are CSFs?

A critical success factor is an element of a marketing campaign which is necessary (but not sufficient) to complete its goals.

Why is setting them important?

Without understanding what is necessary to complete marketing goals, these crucial elements may not be dealt with properly, which in turn can critically affect the ability of marketing teams to execute on goals.

How do you determine what your CSFs are?

To identify CSFs, try brainstorming around the following key types of CSFs, as presented by Rockart and Bullen:

Industry
Competitive strategy and industry position.
Environmental factors
Temporal factors
Managerial position (if considered from an individual’s point of view).

See http://rapidbi.com/how-to-write-a-critical-success-factor-csf/ for more information.

What are some examples of CSFs?

An example of a CSF which applies to the “10% increase in visitors per month goal” could be:

Implementation of click tracking (e.g. Bit.ly) to assess traffic sources.

Achieving final approval

Once the full plan has been created, make sure the key stakeholders provide the final sign off on the document. Producing a KPI framework is not an easy thing to do, and will certainly take substantial time to produce, but it is time that you would be hard pressed to find a better use for; a KPI framework acts as the beating heart of well managed digital campaigns. In terms of estimating how much time to put aside for this process, this is generally proportional to the number of conflicting views/business models in the organisation and how many iterations are required to bring out the common ground. Commonly this in the range of 1-3 days. Also, as a general rule, if you can’t fit the KPI framework on a single side of A4 then it has become too complex to manage.

It is also worth noting that CSFs can change over time whereas ideally the rest of the framework is reasonably static. We like to connect CSFs to the KPI framework to emphasize that in order for targets to be met, certain things need to happen, but due to the volatility of CSFs this requires regular review.

In the next section we will look at real life examples and sample models for making the most of analytics and KPIs.

Sample models and further discussion

In this section I provide an overview of the alternatives to top-down models, discuss some issues that people commonly encounter when setting up these frameworks and provide pointers to further reading to learn more about this subject.

Consequently this document is split into the following sections:

Alternative models
Common pitfalls
Further reading

Alternative models

While KPI frameworks contain much of the same information, different approaches can be taken based on personal preference. The three alternatives to “Top-Down” are as follows:

Lifecycle-based models

Lifecycle-based frameworks are structured around the different stages of a visitor’s relationship with websites. Consequently goals are bucketed by these different stages, rather than being derived from explicit business objectives as is the case with Top-Down models. There are multiple varieties in this category, including Xavier Blanc’s REAN model, and Dave Chaffey’s RACE model. These are often named according to the way the different visitor interactions are modelled.

For example, in the case of RACE, the stages are:

The definitions provided below are largely taken from http://www.smartinsights.com/digital-marketing-strategy/race-a-practical-framework-to-improve-your-digital-marketing/.

Reach

“Refers to building awareness of a brand … in order to build traffic by driving visits to different web presences”.

Act

“Act is short for Interact. It’s about persuading site visitors or prospects to take the next step, the next Action on their journey when they initially reach your site or social network presence”.

Convert

This is where users reach the “conversion point” e.g. a sale for an e-commerce company.

Engage

This stage is about “building customer relationships over time through multiple interactions using different paid, owned and earned media touchpoints such as your site, social presence, email and direct interactions to boost customer lifetime value”.

Here is an example RACE framework.

While this does not have the segments and targets we see in the Top-Down model, these can (and should) be included in an accompanying document. Dave Chaffey manages this in a formal way with the “SOSTAC®” planning model (http://www.smartinsights.com/digital-marketing-strategy/sostac-model/).

Persona-based models

Persona based frameworks are as the name suggests: based on personas, or “fictional characters created to represent different user types within a targeted demographic, attitude, and/or behaviour set that might use a site, brand or product in a similar way” (http://en.wikipedia.org/wiki/Marketing_persona). Persona-based planning is commonly used in offline marketing and can provide an easy-starting point to digital measurement. The process for creating a persona-based framework is as follows (credit to Lynchpin):

Step 1: Define key site personas (this can be done via research/workshops) by determining why people might be on the site.

Step 2: Define how each persona may be identified online (i.e. what does the behavioural segment look like).

Step 3 : Define what a successful visit looks like for each persona based on the visitor’s objectives. This involves setting KPIs for each persona.

Step 4 : Use data to verify those personas are accurate. This can be done either by directly asking your visitors e.g. via onsite surveys or email, or by determining whether the given personas show behaviour onsite which is in keeping with the way they have been categorised.

Composite models

Composite models combine different frameworks for the goal of ensuring completeness. E.g. Top-down & persona-based or RACE for each persona.

Common Pitfalls

Successfully implementing KPI frameworks can be tricky. In this section I explore some of the most common issues and offer some solutions.

Too many unknowns and data trust issues

If a company is relatively new to structured approaches to digital measurement, there may be tracking limitations which prevent analysts from being able to see enough of the user journey to assign credible values to each of the conversion types. This makes it difficult to justify decisions which are based on numbers rather than “expert-opinion”.

Related to this issue is a lack of “data trust” – where tracking has been installed, but there are doubts about the accuracy of the data, either because of technical issues with the analytics program/implementation, or because of the assumptions which have been built into it. Attribution is a common area where such doubts arise. This again has the potential to turn people away from data-based decisions entirely.

The answer in both situations is the same – provide a roadmap that addresses the issues that cause the problem or doubts and facilitates a robust framework to be established. Using an interim model, with caveats that cover these issues, is sensible where possible. It should be noted that Analytics is data is never 100% accurate – it is intended to provide insights into trends and allow for approximate comparison. Do not become bogged down in calculations if data is marginally out. For further information on this subject, see Avinash Kaushik’s post “Data Quality Sucks, Let’s Just Get Over it” (http://www.kaushik.net/avinash/data-quality-sucks-lets-just-get-over-it/).

Using KPIs that are too high/low for level of reporting

Certain metrics are useless at an aggregate level. One prime example is bounce rate . If a visitor bounces after having read the latest blog post, it would be hard to say for sure whether that it was a “bad” bounce. However, if a visitor came on transactional keywords, landed on a product page and then bounced before adding the item to their shopping cart, it is clear that is not the ideal outcome. Here context is critical to interpretation.

Similarly, trying to apply high-level metrics to low-level activity does not work. For example, attempting to determine the ROI of an individual tweet is a waste of resource as it offers little insight. This is because there are so many influencing factors that ROI measurement becomes futile. A far better approach is to analyse ROI at campaign level as this substantially lessens the issues around attribution.

To summarise; it is important to consider the scope of measures and how meaningful they are when including in reports.

Neglecting to gain buy-in

It is absolutely critical that key stakeholders involved in a project understand the importance of the KPI framework, and are provided with the opportunity of feeding into it. If an individual independently creates a framework without the appropriate feedback it can often result in resistance from the people it affects – even if the framework accurately captures all of the necessary objectives/goals/KPIs. This is because people want to be able to contribute or influence the areas in which they operate, particularly something as important as how results are recorded and performance is measured. Not facilitating this can result in a negative emotional response.

Not knowing what you can/can’t measure

There are limits to what you can/can’t measure, especially with standard analytics installations. For example, it is not possible to measure time on site for visitors who bounce in Google Analytics (given a standard, out-of the box implementation). This is because of the way this metric is calculated using time stamps every time a visitor visits a new page. As with bouncing visitors there is no second pageview: time on site cannot be calculated. Consequently, it is useful to have a technical analyst on hand to be able to report on whether the desired KPIs are immediately measurable and if not, whether it would be possible to customise tracking.

Mixing strategic KPIs and tactical metrics

It is important not to mix strategic KPIs (e.g. ROI) with tactical metrics (e.g. Google PageRank). Tactical metrics are only of use to the extent that they contribute to strategic KPIs. For example PageRank is only useful to the extent it helps garner organic search traffic. Consequently tactical metrics do not belong on the KPI framework.

Douglas Holmes