Test, measure and learn how it’s working

We've heard from many trust leaders who believe that their trust isn't good enough at benefits realisation: once a project is delivered everyone moves on to the next big thing and rarely measures whether the benefits expected in the business case were delivered. This isn't unique to digital transformation, but it is particularly important given the significant investment that trusts are making in the digital space. Measurement is fundamental for any digital transformation: without it, there's a risk of investing in something that doesn’t deliver the intended outcomes.

 

Benefits realisation isn't just a reporting exercise, it's why we invest in digital

Realising benefits from digital investments should be the primary objective of everyone involved in digital transformation. If we're not aiming to deliver positive outcomes for patients, staff and organisational goals, then why bother to invest? But sometimes the aim can get lost: teams end up focusing on delivering the plan (outputs), rather than delivering the benefits (outcomes).


In response, some organisations have created dedicated 'benefits management' functions, or rely on academics and consultancies to evaluate whether digital investments have delivered the expected outcomes. However, external evaluation such as this should complement – rather than replace – a benefits-focused approach within the delivery team, much like external audits provide assurance that a trust has a robust approach to financial management. Trusts should ensure that delivery teams have a benefits-focused mindset, clear goals and the skills and tools needed to measure progress against them without waiting for an external review. Embedding data analysis skills in a digital delivery team can act as a force-multiplier: enabling teams to rapidly understand how they can make the biggest impact.

It's about creating measures that matter. It is the responsibility of the whole board to understand how data is used to improve services and know the most important questions to ask of their analysts. At DCHS, conversations at board-level and across the organisation focus on identifying and measuring the things that matter most to our teams so we can better understand what good looks like for each service.

Jim Austin    Chief Information and Transformation Officer, Derbyshire Community Health Services NHS Foundation Trust

Understand what you're trying to change and how you’re going to measure it

There's a few reasons why trusts find it hard to measure the impact of digital transformation:

• Attribution problems: it's often difficult to isolate the impact of a particular initiative from everything else. Other changes, external factors, staff changes and seasonality can have a big impact on key metrics.
• Poor data: trusts may not have the right data infrastructure to report effectively on impacts. Some data may be unstructured, data may not be linked in the right way and some data may not exist at all.
• Vague benefits: if a benefit is not expressed in a clear and measurable way, it will be difficult to determine whether the benefit has been realised or not.
• Lack of skills / capacity: delivery teams may lack the analytical skills required to evaluate benefits.

 

Regardless of these common challenges, there are some tried-and-tested ways to measure the impact of digital investments:

 

• Experimental methods: rolling out changes to a subset of users and observing the differences between this group and a control group is a great way to isolate and accurately measure the impact of a particular change.
• Proxy measures: if it's not possible to track a particular metric, there may be other metrics that could be used as a proxy.
• Sampling: if universal, regular reporting is not possible, then running periodic samples can provide a good alternative evidence source.

 

Don't wait until the end of a project to measure the benefits

The way programmes, business cases and contracts are constructed can lead to benefits only being measured when a new digital service is fully implemented. There are some advantages to taking the long view: digital transformation often takes time to bed in, as staff adapt to new ways of working and the technology improves. However, the end of a project is far too late to find out that you've made a poor investment. It's better to deliver and measure incrementally, using early indicators to track whether you are on track or need to change course. You may also find benefits you didn't expect, that open up new opportunities.

Key questions for boards:

• Are delivery teams focused on delivering the plan, or delivering benefits?
• How will you know if your digital investments are working?
• Do you have weekly or monthly reporting in place for key metrics?

 

Case Study

Measuring value at Cheshire and Wirral Partnership NHS Foundation Trust (CWP)

The role of a finance director is ultimately to enable change. Our job is to make [the case for digital transformation] simple and accessible to all. It has to be communicated in a language that people can understand and demonstrated in a way that people can see and feel.”

Tim Welch    Deputy Chief Executive and Director of Business and Value, Cheshire and Wirral Partnership NHS Foundation Trust

Context

For a number of years, the trust has been making substantial regular revenue investments into digital devices and system development. During the height of the pandemic, the trust signed-off the development of a new EPR and began gradually developing the system with clinical champions and ultimately deploying the first phase in October 2021. Across all these digital endeavours the board has taken a novel approach to measuring the value of its investment.

Approach

As part of a broader approach to embed quality improvement within the organisational culture, the trust's leadership began avoiding references to cost improvement plans and instead talked about efficiency. The board took this one step further and focused on the currency that would make sense to those on the frontline delivering service improvement, such as ward managers and clinicians. They determined this would be about value.

Consequently, the finance department was rebranded to business and value, and the trust began appraising its digital investments based on value. For most digital decisions this largely revolves around time saved. The board categorises time savings into three categories:

1. Additional time that allows staff to do more or better things (e.g. time for additional home visits and time to care).
2. Efficiency savings that can add up to a budgetary release.
3. Additional time to allow teams the space to think and perpetuate a virtuous cycle of improvement.

 

Not only does this method work for larger scale digital investments but it also helps the board rationalise regular investment into device refreshes and other day-to-day upkeep.

This approach isn't perfect in terms of benefits realisation. For example, the trust recognises that there is often a degree of running existing and new processes in parallel when rolling out changes, however, they try to resource this where possible.

Freeing up time requires CWP's leaders to:

• Make change as easy as possible for staff – for example by establishing multiple training sites.
• Acknowledge that staff are only able to properly familiarise themselves as users outside of the classroom. So, make training short and sharp and invest in floorwalkers to smooth the transition.
• Understand that in order to realise value you need to build confidence across the workforce. Use everything you have to reach a "critical mass of confidence".

 

By using time savings as a measure of success, the board sends a clear signal to the wider organisation that improving quality of care and making life easier for staff to do their job well are top board priorities. This in turn has led to improved clinician engagement and buy-in.