Good Evaluation = learning how to do things better

Dr Kate Hamilton has written a blog about the evaluation process used within Actif Woods Wales…

Like all good projects we take the evaluation of Actif Woods Wales seriously – not just because we want to be diligent about monitoring project delivery, but also because one of our priorities is to deepen the evidence base about woodlands being good for health and wellbeing.

Our Central Challenge

One of the central challenges in doing this is measuring health and wellbeing in ways that are credible, robust and meaningful. As mentioned [in blog 1] we have very diverse participants with different needs, capacities and health conditions, attending different types of activities in many different locations, and participating as and when they choose to do so. This is pretty much the direct opposite of an ideal set-up for systematic research. But such are the realities of many real-life interventions – and as learning from real-life interventions is currently a bit of a gap in the evidence base, we just work with it and try to learn from it.

We’ve ended up taking a bit of a mixed-methods approach to evaluation, trying to strike a pragmatic balance between our own ambitions, different stakeholders’ expectations, and what can actually be implemented in the course of project delivery.

We have experimented with incorporating validated health measures into our questionnaires*. This came about following advice from key stakeholders in the pilot phase of Active Woods Wales: they suggested that while our achievements were all very nice, if we wanted this kind of work to be convincing – particularly to our health sector peers – we needed to use proper recognised health measures to assess our impact. So, after a bit of research and some careful testing, we came up with a selection of questions that includes established tools such as the short version of the Warwick Edinburgh Mental Wellbeing Scale (SWEMWBS).

The hope was that this would be a straightforward way of making our evaluation data more robust. To be honest we are still in the process of analysing our findings from SWEMWBS, so I can’t tell you whether or not it’s worked – as in, will it show that we improve participants’ wellbeing according to this measure. We are very fortunate that through one of our budding research partnerships we have a pair of Psychology students at Bangor University working on this for their final-year project as I write.

IMG_1240.JPG

Actif Woods Wales 

What we have learnt to date

In the meantime, though, I can share some reflections on what we have learned from the experience.

  1. Firstly, gathering questionnaire data systematically across the project has been a real challenge – and the inclusion of validated measures in itself does nothing to change this. What I find most interesting is the clash of cultures between the project ethos – humane, inclusive, sociable, holistic, non-medicalised, open– and the procedures involved in completing questionnaires. Feedback from participants and leaders tells us that questionnaires are an unwelcome intrusion into the human dynamics of Actif Woods sessions, meaning that everyone concerned wants to get them out of the way as quickly as possible. This has obvious detrimental impacts on the quality and reliability of the data that they produce.
  1. Secondly, using validated tools doesn’t in itself make questionnaire responses valid. On a simple level, you need a reasonable degree of confidence that participants have understood the questions and their purpose so that their answers are relevant – which is not always the case when you are administering questions to lots of people at once, in a non-therapeutic context, and where participants sometimes misunderstand their purpose as a kind of ‘customer satisfaction’ type exercise. Without the time and human resources to explain the questions and their purpose each time to each person we know that some gobble-de-gook sneaks into our data for these reasons.

Realistically, we just aren’t able to meet the strict criteria and protocols by which validated questions have been validated. This doesn’t make our findings useless – we can still interpret them in other ways – but it does mean they are no more robust than findings based on any other questions.

  1. Thirdly – and most importantly – SWEMWBS (and other standardised, quantified measures) tell us very little about how and why things are changing for participants and, particularly, why it matters to them. This is not a criticism of the tools themselves – they are not really designed to pick up these things: in fact, they are specifically designed to bracket these things out so that interventions can be assessed in terms of their impact on wellbeing net of all these confounding effects. But it does make us question their value to a project like ours.

Wellbeing is a complex and subjective state, and we are just one of many influences on our participants. So, any ups and downs in their ‘scores’ cannot simply be attributed to Actif Woods Wales in the first place. We also know from their responses to narrative questions that participants experience significant benefits which aren’t picked up by the standardised measures – things like being able to maintain a level of health despite a degenerative illness, or cope with episodic conditions better. These may not show up net ‘improvement’ but they are hugely important to the people concerned.

On top of this, we know that the benefits participants experience are
multi-dimensional and often unexpected. For instance, a stand-out finding from our monitoring so far is the importance to participants of social gains – meeting new people, making friends, developing social skills, and learning new things – even when these are not their primary reasons for joining. This is not just ‘soft stuff’ which emerges around ‘real’, ‘hard’ health improvement – social connectedness and learning are hugely important in themselves as contributors to good health, predictors of mortality, and as strategies for lifelong wellbeing.

fun in the rain.jpg

Actif Woods Wales

So, what and who are we evaluating for?

This all seems fairly obvious – we all know that our wellbeing is more than just a number. But it leaves me wondering why, if this is so self-evident, projects like ours continue to feel pressure to use these methods. Why are they still assumed to be more convincing than other measures, and why do we get praise for using them even when we’re pointing out that they don’t work?

It seems to me this is not about evidence but all about traction: there’s a kind of symbolic and political value to being seen to be using the accepted tools, even though we can’t use them properly.

Pragmatically it might be wise just to accept this – but my concern is that trying to use ‘proper’ methods ends up sapping resources (time, effort, money, attention) that could arguably be being better used – to produce a lot more insight from a lot more stakeholders, using methods that give participants some voice in saying what matters to them, with a lot less anxiety all round and a lot more contribution to overall project impact and learning.

If evaluation is not squarely about maximising impact and learning how to do things better then I don’t believe it can’t be seen as a good use of anyone’s resources.

So, where does this leave us as a project? For now, we continue to work with our mixed methods approach, enabling us to at least partially meet a wide range of expectations. But increasingly where we’d like to see ourselves is as part of a movement towards evolving better ways of evaluating wellbeing, ones which find rigour in validating what matters to the people concerned rather than just what fits into our existing measures. It feels like the current climate is ripe for such a shift – and through engagement with the Good Practice Exchange and others we are glad to be finding our allies in this quest.

 

 

One thought on “Good Evaluation = learning how to do things better

  1. Pingback: The future of parks and their positive impact on wellbeing | Good Practice Exchange at The Wales Audit Office

Leave a Reply / Gadael ymateb

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s