Who should be responsible for policy evaluation?

The Guardian has published a note by Nick Axford on when a charity might wish to carry out a randomised controlled trial. Axford works for a charity which promotes the use of evidence in designing services for children and families.  This rather neatly wraps up a number of my key interests: policy evaluation, longitudinal data collection and the role of the voluntary sector.

Randomised controlled trials (RCTs) are usually associated with medical, rather than policy, interventions.  Actually, they are usually associated with modern pharmaceuticals.  We generally kid ourselves that our medicine is evidence-based.  A lot of it is ‘stuff that we’ve done before that seemed to work so we did it again’.  And so it is with social interventions.  Hence Axford’s suggestion that maybe we should try to gather some actual evidence for some of our pet policies before putting them into full scale production.

It’s not a bad idea – in fact it’s a good one (although I should add that the evidence wouldn’t have to come from an actual RCT – there are other options).  Who should take responsibility for making sure it happens?  We (well, the National Institute for Clinical Excellence) demand that pharmaceutical companies come up with some evidence before the NHS will buy their products.  Pharmaceutical companies, unlike charities with a new idea for helping children living in poverty, stand to make a lot of money from their products, so it makes sense to ask them to come up with the evidence.  Should we really ask a charity to do the same?  I can see why a charity might wish to promote its ideas this way, but can we expect it?

In fact I think that if the state wants to ‘buy’ the charity’s new idea, it should do the research itself.  Charities don’t have the same kind of financial interest vested in their ‘products’ as pharmaceutical companies: they simply don’t have the resouces to do this kind of research.  Following people over time – which is what you’d need to do to study the effect of a particular policy or intervention – is expensive.

As an aside, I think Axford’s article is missing a paragraph about regression to the mean.  He writes: “One of our evaluations showed that children whose parents attended a parenting programme were better behaved at the end of the programme than at the beginning. Great – the programme worked! Except that the same happened to similar children whose parents didn’t attend the group. The programme made no difference.”  The key here is “similar children”.  If you take a group of children with poor behaviour (the kind to whose parents one might recommend a parenting programme) you might expect to see some improvement in their behaviour over time, whether you intervene or not.  Why?  Because most children are neither consistently bad nor consistently good.  A period of bad behaviour will be followed (eventually!) by a period of better behaviour, no matter what you do.

This entry was posted in Academic research, Journalism and tagged , , , , , . Bookmark the permalink.