Evaluating co-production and measuring outcomes by Sam Callanan

Two main reasons are put forward for using co-production to improve existing services and design new ones:

  1. Co-production gives people a genuine opportunity to influence the design and delivery of the public services that they use and pay for (indirectly through taxes).
  2. Co-production should deliver better services and decisions because those affected by the services, with real life experience of them, are involved in designing them.

However, to date the evidence that co-production does deliver better services is limited. SCIE’s recent review of the difference that co-production makes in social care (SCIE, 2022) found evidence of benefits for participants in co-production projects, but the available evidence for the impact of co-production on services was weaker.

For people who access services that have taken part in co-production there was evidence of increases in self-confidence, self-esteem and sense of empowerment, better health and wellbeing, increased engagement and trust, and higher levels of satisfaction with and awareness of services (SCIE, 2022, p. 2). For those responsible for designing and delivering services there was evidence of improved job satisfaction, motivation and practice, and increased trust, engagement and dialogue with people who draw on care and support, and with unpaid carers (SCIE, 2022, p. 2).

But the review found that there was less evidence on the difference that co-production makes to services and to the organisations delivering those services. The review notes that in social care in particular more needs to be done to evaluate the outcomes of co-production on services (SCIE, 2022, p. 10). To date there has been limited work on how services have changed following co-production and on linking any changes to the co-production.

To address this future co-production projects need to evaluate both the co-production itself and the impact co-production has on services.

It is important to evaluate whether the service has improved because if we can show that co-production leads to better services that supports the argument for more co-production.

It is also important that the co-production process itself is evaluated because our working hypothesis is that good co-production leads to better services. If the co-production process itself hasn’t worked well, we would be less likely to expect it to lead to improved services.

Evaluating the service being co-produced.

When co-production is used on an existing service we want to find out whether the service has improved since it has been co-produced.

If we are co-producing a new service we want to try and determine whether the service is better (i.e. more fit for purpose) than it would have been if it had not been co-produced.

For existing services, ways of judging service quality may already be in place and these could be used to measure service improvement. Such as:

  • Changes in satisfaction with the service, service user feedback.
  • Feedback from staff delivering the service.
  • Changes in the number or type of complaints.

Exactly what to measure and how to do that will depend on the nature of the project and service.

For some projects there may be objective measures that could be undertaken. For example, in environmental services this might be things such as improvements to pollution levels, reduction in carbon use, or a reduction in landfill.

It is harder to determine whether a new co-produced service is better than it would have been if it had not been co-produced. There may not be an existing service to compare the new service against. However, it is still possible to assess whether the new service meets its goals, and how successful it is through measures such as user satisfaction.

It is worth noting that changes may not be apparent until some time after a new or redesigned service has started, so evidence that services have improved may need to be collected after the co-production part of the project has finished.

Evaluating the co-production

To determine whether a co-production process/project has worked well it may be helpful to consider how well it has followed the principles underlying co-production. Various principles for co-production have been proposed, but most approaches highlight the importance of equality, diversity, contribution, and inclusion.

SCIE set out four core principles:

Equality:

Co-production starts from the idea that no one group or person is more important than any other group or person. In co-production everyone is equal and everyone has assets to bring to the process. Assets refers to skills, abilities, time and other qualities that people have.

Diversity:

Diversity and inclusion are important values in co-production. This can be challenging but it is important that co-production projects are pro-active about diversity and work to reflect and include the range of people who will be using the service.

Accessibility:

Accessibility is about ensuring that everyone has the opportunity to take part in an activity fully in the way that suits them best. Co-production needs to be accessible to allow everyone to contribute on an equal basis.

Reciprocity:

‘Reciprocity’ is a key concept in co-production. It has been defined as ensuring that people receive something back for putting something in. It is a way of showing people that they are valued and needed. It is related to ‘mutuality’ and all parties involved having responsibilities and expectations.

There are many ways in which we can assess whether a co-production project has followed these principles. Methods can be combined to ensure all perspectives are covered.

An evaluator could participate in or observe the project and record what the project did, what worked and what didn’t work. Project documents (terms of reference, meeting notes, etc.) could be reviewed to see what steps were taken to ensure equality, diversity, inclusion and so on. 

This could be complemented by interviews with participants during and after the co-production to capture what worked well, what less well, and why.

Co-production should be flexible and adaptable, with participants able to shape the work. Co-production projects often take a learning approach, with participants actively reflecting on how the project is running and whether it is meeting these principles as the project runs. This both generates evidence to do with how the project is working and allows the project to identify where things are not working and adapt.

As well as trying to build an understanding of whether or not the project actually realised these principles it is important to assess how participants experienced the project. Did they feel that they were treated equally, did they feel heard and valued, were they able to access and contribute to the project in ways that worked for them, did they feel it was a worthwhile use of their time?

Again, this information could be gathered via interviews with participants, surveys, or other feedback mechanisms.

Further evidence of the success (or otherwise) of the co-production and its value to participants can be gathered by evaluating other outcomes that taking part in co-production projects can have for participants. Such as:

  • Whether or not it improved people’s self-confidence or self-esteem.
  • Whether or not people felt more empowered.
  • Whether or not it improved trust or engagement between those using the services and those responsible for designing and delivering them.
  • Whether or not participants came to understand each other better. I.e., did people responsible for delivering services understand more about what people want or need from the service, and how best to deliver it for them? Did people using or drawing on services understand more about the constraints on services, and what is involved in delivering them?

By evaluating both the co-production itself and the difference that co-production makes to services we can build a stronger evidence base around co-production.

It is important that co-production projects are evaluated so that those running them know whether the time and resources invested in the co-production have led to positive outcomes. But it is also important that co-production projects share their findings to grow the evidence base around co-production.

If the evidence demonstrates that co-production has positive benefits for participants and leads to better services then that builds a strong case for more co-production of public services, which would then mean more people experience the benefits of taking part in co-production and then lead to better services for all.

Sam Callanan
Senior Consultant, Strategy unit

If you are interested in finding more about how to evaluate your co-production work please get in touch with co-production works via the enquiry form .

References:

SCIE. (2022). Developing our understanding of the difference co-production makes in social care.