Not all data is equal. Do you know how to measure the true business impact your digital transformation efforts and investments?

Meet Billy Smith. Billy’s 6 month ERP implementation just went live this month and a deep sigh of relief was in the air. 7,159 people just attended a training session in the past week and the project even came in on budget. Even the initial in-app CSAT survey scores are coming back positive. Billy reaches out for a celebratory glass of Moët when a catastrophic realization bubbles up amidst the celebration. Billy has no idea whether the project actually worked. It turns out, Billy is the latest victim of vanity metrics.

What are vanity metrics?

Vanity metrics exist in plain sight – buried in reports, dashboards and even OKRs. Look closely, and you’ll be able to identify the tell-tale signs, the gorgeous looking metric highlighted in bold, may not be that good looking after all.

Vanity metrics aren’t actionable 

The heart of data-driven teams is the understanding that if something can’t be measured, it can’t be affected. The days of running blind off of gut feelings should be as far behind us as faxing sales contracts and cashing checks. 

Whether you’re building WalkMe solutions, or sending an action-oriented email, we can all take a page of marketing optimization wisdom by thinking about a call to action (or CTA.) v   Pop quiz – what metric would you use to optimize and report on? Well, it’s surely not the click – it’s the action. Measure that. If you’re measuring something else. It’s probably a vanity metric.

Vanity metrics look impressive, but might not be repeatable.

Is that dashboard that came pre-loaded in your ERP providing data around how many people logged into your system every month? Good – that information is super critical for understanding whether or not it’s being found and accessed, but it may not tell you whether or not you’ve had any impact on fulfillment times. Perhaps it’s a number that looks great, but if it’s not tied to the work your team is doing and iterating, it might not be the best metric to look at.

If you’re leading a Digital Adoption effort, you’ll want to dig into more than just the reporting you’re getting out of the box with your software, or that stale BI dashboard your team might still be using from 2020, and instead think about what you need to know about your in-platform user behavior to measure success.

The problem with how we measure digital transformation projects.

So what does this have to do with your digital transformation project? Well, if you’re working on one, go take a look at your latest status update. I’ll wait. It’s just us friends here… go ahead and give this article 5 stars if you’re tracking go-live date, SDLC milestones, or maybe training and enablement dates. 

Bravo! Well, here’s the thing, all of those data points are absolutely critical in measuring the project, but ask yourself if they’re providing any visibility into your actual transformation. If you’re scratching your head, let’s talk about how value was determined back when your mammoth project was a wee slide in your executive team’s planning off-site.

What organizations really want to do

This one is easy… drive growth and maximize revenue. That’s clear, but how it’s done usually falls into a few common categories.

• Reduce operating costs (improve margins)
• Improve employee satisfaction and engagement (increase productivity)
• Accelerate rate of distribution  (maximize earnings)
• Accelerate deal cycles (boost revenue)

That’s why projects that get funded at organizations large and small are often tied to the types of efficiencies that impact the company’s bottom line. The ability to move faster, more efficiently, and with greater satisfaction is the cornerstone of every digital transformation effort. So why do so many transformation projects end up measuring go-live dates and delivery costs?

It’s how we forecast risk and costs

The realization of intended benefits can’t be had if there’s no platform to dole out said benefits. As a result, it’s all hands on deck to get new software in place. This pushes a ton of focus on the service delivery teams and project management teams to ensure new software tools and platforms are ready on time. Attention shifts from benefits to risk, and teams concentrate on what they know best.

• Expected return versus project costs
• Platform and integration costs
• Training costs
• CSAT

What should we measure instead?

If we think about our friend Billy Smith, part of the reason they’re up at night is because they’re caught up in a risk mindset instead of a transformation mindset. So let’s help Billy sleep better on their next project so they can focus on real impact metrics instead of vanity metrics.

Go Live Date → Value Realization Period

System go-live is easy to measure. Did we push to production? Congrats! But that doesn’t inform whether the investments paid off. Identify the motivations behind the system in the first place. 

Is your goal to expedite contract signing in order to accelerate deal cycles? Measure that, and start to calculate how long it takes to see the value of the system you mapped out when you first purchased it. Measuring Value Realization Period, or time-to-value, will help ensure the Moët is twice as nice once you open it.

CSAT → Process Abandonment Rate

CSAT is an incredible metric used by almost every organization to measure whether the people that use your system are happy. The funny thing about CSAT is that while it looks like a quantitative measurement because it’s a number, CSAT is actually entirely subjective and therefore easy to measure, but difficult to replicate. More so, it’s less correlated to positive outcomes than ever before.

So instead, measure what your users are actually doing. Are they starting a fulfillment order and stopping midway? Are your sales people generating a proposal but getting confused along the way? This metric gives you a near real-time insight into how successful that shiny new platform is when it comes to driving outcomes so that you can provide help along the way when you need it – not when support tickets start piling up. 

Training Participation Rate → Process Initiation Rate

Oftentimes, companies may rely on time-consuming training, webinars, or the more advanced, in-app guidance. Teams will often measure how many people joined, engaged or RSVPd. This is a great way to measure the reach of your training, but it misses the mark when it comes to measuring whether people know what they need to do.

Instead, measure how your employees or customers are actually starting the process they’ve been trained on. Perhaps it’s step one of benefits enrollment, or using a macro in the contact center. Being able to quantify and measure process initiation rate is your signal into what’s working and what’s not.

Are you ready to look past that shiny vanity metric?

As you look ahead to your next transformation project, make sure to examine the data you’re using to validate your investments. Are you measuring outcomes that you have an effect on? Are you connecting those to repeatable and measurable business impact? If you are, you might just have what it takes to harvest the real promise of digital transformation in your organization. Good luck!

Whether your organization aspires to be data driven (or to start, data informed) , one of the hardest aspects of developing an analytics strategy is to define lagging and leading indicators. Quick definition (we’ll dive deeper later):Lagging indicators define success, while leading indicators predictably measure progress along the way.

According to Harvard Business Review, 49% of executives indicated that analytics are critical to measuring the success of their digital initiatives. But what exactly needs to be measured isn’t always apparent. 

So how should you measure the success of your digital initiatives? Let’s dive into the metrics high performing teams use to ensure they’re on the path to success.

Leading vs lagging indicators to measure your success

So, what are most organizations measuring to actually define and measure success? Let’s take a look at some of the top metrics used by digital transformation executives.

• 36% customer review and feedback 
• 35% company revenue
• 32% speed of new technology implementation

(source: CIO Outlook 2021)

These indicators, often tied to revenue or margin, have an important role in calculating the outcome of projects in a spreadsheet, but they often miss the mark when trying to identify real impact across the organization and how it affects your employees.

By the time the usual lagging reporting comes into play, it’s too late to have an effect on the outcome and maximize the invested time, energy and resources into a project. It’s a real problem for transformation and technology professionals. But it’s a problem we can help solve.

That’s why digital transformation has to start with metrics, planning, and data along the way. Let’s introduce you to the yin and yang of data points, leading and lagging indicators. 

Why both? Well, let’s say you and your friend Sue decided that this was the year you were going to run a marathon. No simple feat, and one which requires a tremendous amount of training. So, you and Sue both commit to training 4 times a week for the year leading to the big race. 

It’s the day of the race, you’ve both done it, trained 208 times, and now, it’s time to run 26.2 miles like the crazy humans that you are. How are you going to do? If you feel like you need more information, you’re thinking about it the right way.

Measuring for outcomes with lagging indicators

Lagging indicators, while slow to the race, are critically important. Lagging indicators provide a current view of the business by analyzing backward looking data. They also are usually tied to top-tier initiatives that connect to the expected value of your project. By defining the right lagging indicators, you’ll be able to gather alignment cross-functionally to set clear and reportable goals.  It also makes it easy to ensure you’re speaking the same language with other teams.

Examples:

• Annual revenue
• NPS
• Customer churn rate
• Number of trainings completed

If we think about you and Sue training for the race, it’s great to see that the goals were met and completed on time. It helps track progress, and also ensures there’s an alignment between multiple parties (or teams) on what needs to be done.

Measuring for progress with leading indicators

Leading indicators should be frequently updated in order to predict a future state. By using data representative of the current timeframe, they are indicative of how outcomes may be affected in the future. By identifying the right leading indicators,  you can measure progress towards your goals, as it’s happening, allowing nimble and outcome-driven teams to predict success, identify challenges along the way, and make changes in the moment to ensure broader long-term success.

Examples:

• Daily active users
• Cart abandonment rate
• Daily support tickets 
• Miles per training & time per miles

Here’s the missing piece of the data story you need to predict outcomes. In this case, focusing on the predicted outcomes based on today’s metrics helps us understand what tomorrow will look like. If you’re training 208 times a year, but you aren’t measuring output, change (week over week), and completion metrics, you’re missing out on key data points for predictability. These are the leading indicators that allow you to ensure you’ll reach your goals, or inform whether or not you’re going to miss and need to make changes along the way.

Predict better outcomes: Use digital adoption metrics as leading indicators

When it comes to digital analytics, tools have been developed to measure advertising yield, eCommerce progress, and product adoption. Digital adoption metrics help teams take the best of these various evolutions of metrics and find the ones that are uniquely suited to digital adoption. 

When it comes to being able to accurately predict outcomes, think about measuring Reach, Engagement and Activation.

Reach: How are people interacting with your systems?

• Daily active users per platform
• Week-over-week change in users per platform
• Monthly log-ins per team

At the most basic level, teams that are rolling out new systems need to ensure that those tools are being accessed and used in the first place. Perhaps you’re relying on a communication plan to promote the new tool using email, Slack, or Workstation. When doing so, you can measure the effectiveness of that communication by defining your reach metric. 

Where to Start: Start by measuring daily active users (DAU) especially for a new platform roll out. Measure change over time to understand how and when platform adoption starts to succeed.

Engagement: How are people interacting with critical processes?

• Feature interaction rate 
• Week-over-week change in interaction rate
• Single session completion rate
• Process abandonment rate

If your system is more complex, perhaps a CRM, or an expense management system, you’ll want to make sure you’re measuring the actual process, and not just system access. Perhaps you’ve rolled out a new expense management process that people will need to learn. By measuring process initiation, you can ensure people are able to find the initiation point, navigate to the right aspect of the system, and get to where they need to go. If you’re not measuring engagement, you won’t know if there’s an opportunity for improvement. 

Where to Start: Start by measuring feature interaction rate to baseline how often people are finding their way to the feature. From there, you can think about % of population as you drive broader awareness.

Activation: How are people completing tasks?

• Completion rate by team
• Time to complete
• Decrease in time to completion

At its most critical, activation is your metric to measure whether or not people have learned a new process, or completed a critical milestone. The concept of activation, borrowed here from Dave McClure’s growth-hacker manifesto, is the key metric that apps like AirBnB, Uber, and Facebook use to measure long-term platform adoption. Uber defined it as a user’s first ride. AirBnB a first booking.  If you’re rolling out a new process, or system, take a minute to think about what ‘success’ looks like from the user’s point of view.

At its most basic, it’s a measure of completion, establishing a metric to understand in real time, whether or not your users are able to complete a critical task. Whether it’s updating a forecast, or completing open enrollment, getting the process completed, is the very nature of why your system is put there in the first place. Measure it.

Where to Start: Start by measuring process completion of the most high-impact process in a system. Not sure if it’s high impact? Find a process that affects a lot of people or has a high financial impact (ex: related to compliance or revenue).

Lead with purpose, lead with data

Remember – Digital adoption metrics provide a glimpse into what’s happening under the hood of your platforms as they’re being rolled out. Beyond the typical measures used to measure ‘done’, digital adoption metrics help you better quantify actual adoption of your systems, processes, and tools that you’ve invested so heavily in already. 

Historically, this level of analytics was only available to marketers or product teams which is why technology teams were forced to rely on disconnected metrics of success. That’s no longer the case with modern digital adoption tools. Now, from the moment you go live with a system, tools are available for you to answer the questions that matter most  to your organization – did it work?

Measuring “did it work” isn’t just interesting for senior executives, but also helps connect the dots to the teams and people that are invested in delivering the successful project.  It also helps teams identify their purpose and teams that have purpose perform better.

So the next time you’re rolling out a new tool or system, don’t lag on defining your outcomes. Lead with purpose, lead with data, and lead with leading indicators to measure success.

How to get started with your next WalkMe Project:

• Data-driven decision making datasheet

• Connect with other people on our LinkedIn group: Digital Leaders in Operational Excellence