The tax payer buys some data. Now who owns it? A private research company.

It turns out that when the Government commissions research (including data collection) from a private research company, it is not standard practice to require, as part of the contract, that the data is later placed in the UK Data Archive.  Who knew?  I’m told that the Government expects it, but cannot require it under the terms of the contract.

I feel a letter coming on…

Of course, there is a healthy serving of self-interest here.  I would like to get my hands on the data from the Citizenship Education Longitudinal Study, which started in 2001 and concluded in 2011 with Citizens in Transition, a follow-up survey which contacted the cohort at age 20.  The follow-up section was paid for by ESRC (rather than the Department of Education) and they, of course, require that the data be deposited. Not that it’s much use without the previous waves…

EDIT: The follow-up data has been deposited. The original survey is still private.

Posted in Uncategorized | Comments Off on The tax payer buys some data. Now who owns it? A private research company.

Who should be responsible for policy evaluation?

The Guardian has published a note by Nick Axford on when a charity might wish to carry out a randomised controlled trial. Axford works for a charity which promotes the use of evidence in designing services for children and families.  This rather neatly wraps up a number of my key interests: policy evaluation, longitudinal data collection and the role of the voluntary sector.

Randomised controlled trials (RCTs) are usually associated with medical, rather than policy, interventions.  Actually, they are usually associated with modern pharmaceuticals.  We generally kid ourselves that our medicine is evidence-based.  A lot of it is ‘stuff that we’ve done before that seemed to work so we did it again’.  And so it is with social interventions.  Hence Axford’s suggestion that maybe we should try to gather some actual evidence for some of our pet policies before putting them into full scale production.

It’s not a bad idea – in fact it’s a good one (although I should add that the evidence wouldn’t have to come from an actual RCT – there are other options).  Who should take responsibility for making sure it happens?  We (well, the National Institute for Clinical Excellence) demand that pharmaceutical companies come up with some evidence before the NHS will buy their products.  Pharmaceutical companies, unlike charities with a new idea for helping children living in poverty, stand to make a lot of money from their products, so it makes sense to ask them to come up with the evidence.  Should we really ask a charity to do the same?  I can see why a charity might wish to promote its ideas this way, but can we expect it?

In fact I think that if the state wants to ‘buy’ the charity’s new idea, it should do the research itself.  Charities don’t have the same kind of financial interest vested in their ‘products’ as pharmaceutical companies: they simply don’t have the resouces to do this kind of research.  Following people over time – which is what you’d need to do to study the effect of a particular policy or intervention – is expensive.

As an aside, I think Axford’s article is missing a paragraph about regression to the mean.  He writes: “One of our evaluations showed that children whose parents attended a parenting programme were better behaved at the end of the programme than at the beginning. Great – the programme worked! Except that the same happened to similar children whose parents didn’t attend the group. The programme made no difference.”  The key here is “similar children”.  If you take a group of children with poor behaviour (the kind to whose parents one might recommend a parenting programme) you might expect to see some improvement in their behaviour over time, whether you intervene or not.  Why?  Because most children are neither consistently bad nor consistently good.  A period of bad behaviour will be followed (eventually!) by a period of better behaviour, no matter what you do.

Posted in Academic research, Journalism | Tagged , , , , , | Comments Off on Who should be responsible for policy evaluation?

First, select your cohort

The first of the national cohort studies, the one begun one week in March 1946 has been the subject of a series of Radio 4 interviews and a newspaper article this week.  A cohort study follows a group of people (the cohort) over a period of time.  There are five big UK cohort studies begun in 1946, 1958, 1970, 2000 and 2012.  The three earlier studies were designed as medical projects, looking at maternal health and then at child development.  They have, however, made fascinating social history, and the questions posed have been so broad in topic that they are of interest to researchers in many fields. 

I am using the 1958 cohort, known as the National Child Development Study (NCDS), to look at volunteering in adolescence and young adulthood, and how those life experiences might influence adult behaviour.  As these cohort studies were designed with medicine in mind there are only a limited number of ‘social’ questions.  This study also has a rather large gap – no surveys took place in the 1980s.  I think there is a rather nice piece of meta-analysis to be done here: a paper on how long-running studies evolve over time, responding to changes in funding, fashions in research, and shifts in culture.  For example, no educationalist today would offer teachers the opportunity to tick a box to describe a pupil as “like a suspicious animal” – an option which was given to the teachers of the 1958 cohort when the children were eleven years old.

Unlike the 2000 and 2012 cohorts, who were selected from babies born throughout the year, the older cohort studies followed babies born in a particular week.  The 1958 cohort were all born in the week 3rd-9th March.  This selection method rules out a number of interesting avenues of research.  We can learn nothing about the health of people born in different seasons; nothing about the educational achievements of people born at the start or end of the school year; and we can’t control for these effects in other analyses either.  Survey methodology has changed in the last 40-odd years.  As part of my imaginary meta-analysis of cohort studies, I’d be interested to know whether there have been any advantages to the ‘one week in the year’ approach.  For example the ‘one week’ studies send birthday cards to all their participants, reminding them about the study and how important it is, urging them to stay in touch.  It’s part of the ‘story’ of the cohort.

Posted in PhD | Tagged | Comments Off on First, select your cohort

“…part of what I call the Big Society…”

The Prime Minister has gone and brought it up again.  I confess that I’m a little slow off the mark, and this is actually from before Christmas (thank you to George Disney for pointing it out to me) but I think it’s worth noting nevertheless.

At the last PMQs before the Christmas break, in response to a question from Ed Miliband about food banks and against a backdrop of Labour jeers, David Cameron said: “First of all, let me echo what the right hon. Gentleman said about volunteers and people who work hard in our communities, part of what I call the Big Society, to help those in need. It is a good time of year to thank our volunteers and what they do, but I do share the right hon. Gentleman’s concern about people who are struggling to pay the bills and to deal with their budgets.”

I think I can see where this all started to go wrong.  Volunteering is ‘good’ – so it’s part of the Big Society.  Volunteering in food banks is also ‘good’ – hence, also part of the Big Society.  Food banks themselves, on the other hand, are less clearly on the side of the angels.  Helping poor people is good, but a first world economy with citizens who cannot feed themselves is a bit embarrassing.

I’ve written about food banks before.  Here in the UK, they are mostly run by churches (although some, like Winchester’s Basics Bank, are also supported by local councils and secular charitable groups like the Round Table).  There’s a lot to recommend them, in a “help of the helpless” way, but it’s not clear to me that this is the direction we want to take.

It’s interesting that Cameron chose to use a “Big Society” reference in response to Miliband’s question.  Maybe he didn’t think it through?  Maybe he isn’t worried about food banks? It’s hard to tell.

EDIT: The PM has now been goaded into promising to visit a food bank.  The Guardian quotes a No. 10 source as saying: “Benefit levels are set at a level where people can afford to eat. If people have short-term shortages, where they feel they need a bit of extra food, then of course food banks are the right place for that. But benefits are not set at such a low level that people can’t eat.”

Posted in Government, PhD | Tagged | 1 Comment

Time use surveys and volunteering

I’ve been reading up on the available (UK) time use data on volunteering and found a short technical paper produced by Kimberley Fisher from the Centre for Time Use Research.  The paper’s interesting in its own right, but here’s what made me smile…  Many time use studies allow participants to record a main and a secondary activity: for example, I could be preparing food for my family and simultaneously cooking for a cake stall for church.  The paper itself is a lovely example of main and secondary activities.  Its main purpose is to provide time use data on volunteering to the Cabinet Office.  The secondary activity is to persuade the Cabinet Office that funding new time use studies is a good idea.  In a section of “methodological observations” Fisher notes: “Time diaries offer particular value for money. Diaries have higher administration costs than questionnaire surveys, but daily activity schedules inform a wide range of policy areas, including transport, physical activity, energy and resource use, total economic activity (paid and unpaid), work-life balance, parenting, eating and drinking behaviours, and quality of life. One time-use survey can address more areas than comparable funding on a series of questionnaire surveys.”  Fisher does concede that relatively rare activities like volunteering are harder to capture in diaries than in standard surveys (for “harder” read “more expensive”) but not until later in the paper!

The other thing that stood out to me was that this is yet another data source which suggests that rates and levels of volunteering have been pretty constant over time (since the mid 1970s in this case).  Can anyone do anything to change that?  This brings me to another paper that caught my eye today…  Beyerlein and Sikkink (2008), quoted in Fisher (2012) look at volunteering to help the victims of the 9/11 attacks in the US.  Those most likely to volunteer lived close to the World Trade Centre, knew a victim, experienced sorrow and were previous volunteers.  So, from the base of volunteers, being personally touched by the troubles of others made people more likely to volunteer.  But was it just displacing existing voluntary work…?  To answer that one, we’d need to go back in time and talk someone into funding a longitudinal study.

Posted in Methods, PhD | Tagged , , | Comments Off on Time use surveys and volunteering

Having trouble funding your longitudinal data collection…?

My friend Francis Brearley was running into just such a problem and has decided to try crowd-sourcing i.e. asking many people to fund a little of the project.  (If you’re in the US, you’ll be familiar with the technique through NPR fundraising weeks.)  He’s an ecologist sitting on 15 years of data on recovering rain forests in Borneo.  He can’t bear the thought of giving up on such a long running data set, but has trouble raising the funds for further waves…  So he’s asking for help.  It’s really interesting to me from a methods (both of data collection and funding) point of view – well worth a look for other academics.

Posted in Academic research, Methods | Tagged , | Comments Off on Having trouble funding your longitudinal data collection…?

Infant feeding again. And peer reviewed journals again. Resulting in an overlong post. Again.

According to the Guardian (and the Daily Mail, Channel 4 News etc etc) we’ve learned that babies fed on demand ‘do better at school’.  This makes me sound smug but I learned this a year ago when the first author, Maria Iacovou, presented her paper at Southampton.  The original paper was published in the European Journal of Public Health this month.

As an aside, I believe that Iacovou hoped this paper would be published in the BMJ and that this explains the long gap between the time I heard about the research and the time the world heard about it.  The BMJ refuses to print anything which has been printed elsewhere, even in working paper or conference paper form.  And that explains why I didn’t blog about it straight after Iacovou’s seminar: I was keen to avoid queering her pitch.  (An aside within an aside – is it still OK to say that?  It doesn’t feel like a Tory peer in the woodpile thing, but it’s slightly closer to Gove’s reference to “welshing” on a deal.)

It’s another illustration of how journals change the way academics work.  In my field (social statistics, or possibly political science, depending on how you look at it) it’s common to publish a little at a time: first an internal working paper, then a conference paper, then finally a paper for peer review.  It allows us to bounce our ideas off a wider community and test the strength of our assertions before we submit our work for shredding peer review.

As a still further aside (would that be as far as Outer Mongolia, I wonder?) I very nearly just blogged without asking – which could have caused trouble.  Academics facing these journal policies should take care when they present their results – academic blogging isn’t as rare as it used to be and making a presentation could easily be considered ‘publication’.

Back to the point…

Iacovou and Sevilla used ALSPAC data to look at whether scheduled versus demand feeding of infants had any effect on education outcomes for children or mental health outcomes for mothers.  The study involved 14,000 mothers in the Bristol area who signed up in the early nineties during a pregnancy.  This paper used a sample of some 10,500 children and looked at whether they were fed on demand or to a schedule when they were four weeks old.  The outcomes were SATs scores and the answers to questions on maternal well being which were administered twice: when the children were 8 weeks and 3 years old. Mothers who schedule fed were happier and well rested (but suffered no less clinical depression…) than those who demand fed.  Babies who were demand fed did better in school than babies who were schedule fed.  You can check out the effect sizes for yourself, but they seemed interesting enough to me: the school results were worth 3 or 4 ranking places in a class of 30.  Not huge, but not tiny either.

The interesting thing here is to compare babies who were fed to a schedule, babies who were fed on demand and babies whose mothers wished they could feed to a schedule but failed.  The mothers who attempted to feed to a schedule but failed had babies who looked like the demand fed babies.  It suggests that some of the unobserved heterogeneity between mothers has been accounted for in the research design.

Let’s unpack that a little…  Compared to schedule feeders demand feeders were older, owner-occupiers, breastfeeders, were breastfed themselves and read to their children almost daily. Those who desired to schedule feed but couldn’t make it happen looked like their schedule feeding sisters – not like the demand feecders.  And yet, the differences in SATs results and maternal well-being seem to divide along the scheduled vs. demand feeding line, rather than along any of the other, better known fault lines.  Wouldn’t we expect the children of demand feeders to do better in school because their mothers are smarter and richer than the schedule feeders, rather than because of the way they were fed as babies?  If that were the case, the children of failed schedule feeders would be expected to look like the children of successful schedule feeders.  Instead, they look like the children of demand feeders.

The study controlled for sex of baby, age of mother, birth order, gestational age, weight, whether the baby spent time in special care, smoking, mode of delivery, and method of feeding (breast vs bottle) amongst other things and STILL found a (statistically significant) difference.  So…  Is the study any good?  Are its findings a good reflection of the world as it is?

It’s a little hard to say.  For starters, this is the first paper to address the question of scheduled vs demand feeding (as opposed to breast vs bottle feeding, which has been addressed left, right and centre).  With no corroboration, the authors are rightly cautious about their findings, but the story of the failed schedule feeders does make fascinating reading.  Still, it’s worth asking a few questions.  Is the ALSPAC data good?  Are the measures good?  Are we absolutely sure that this rather sweet little natural experiment (in combination with standard statistical controls) has accounted for any confounding?  In fact the answer to all those questions is ‘no’.

First question…  The ALSPAC data is not representative of the UK and it has large levels of non-random non-response.  It’s a longitudinal study – that is, the same people are asked questions at different points in time – so drop-out is a real problem.  In this case, it seems likely that demand feeders would be over-represented (since they are the kinds of people who are more likely to answer survey questions) which would alter the effect size (making demand feeding look more important than it really is).

Second question…  Asking mothers a question when their babies are four weeks old is unlikely to produce reliable results (I type this as a mother of a six week old – believe me on this one!).  Also, what mothers do when their babies are four weeks old and what they do when those babies are six weeks, eight weeks, or twelve weeks 0ld may not be that closely related.  The key thing here is that we don’t know whether the babies whose mothers failed to schedule feed at four weeks later (and maybe only days or weeks later) successfully introduced a schedule.  If failed schedule feeders go on to be successful schedule feeders, we may be looking at a spurious correlation between demand feeding and academic results.  Essentially, we need more than one measure of schedule/demand feeding.

Third question…  The authors account for as much of the difference between women as they can by using a long list of controls, and by the cunning use of the category ‘failed schedule feeders’.  However, there are still many, many differences between women and most especially between babies which are not accounted for by this study.  Some babies are easier to feed or easier to settle than others.  These relatively placid babies might be easier to put on a feeding schedule but also more (how shall I put this…?) stupid.  The relationship would therefore be between low IQ and ease of scheduling, not between high IQ and demand feeding.  OK, so that seems unlikely, but it is the kind of ‘folk wisdom’ unhelpfully put about by women whose babies are a total pain in the arse: “Oh, she must be bright – she’s always so interested in the world” translates roughly as “Oh, please let there be some compensation for my horrific sleep deprivation“.

So where does that leave us?  For all the potential pitfalls, I still think this is a rather nice study.  Preliminary, but rather nice.  But you know what’s really, really annoying?  I think the Daily Mail article is a better reflection of the original paper than the Guardian article (primarily because it quotes Iacovou explaining that this is the first study of its kind and the result needs to be corroborated from other data).  Dammit.

Posted in Uncategorized | Comments Off on Infant feeding again. And peer reviewed journals again. Resulting in an overlong post. Again.

AntiSocial Science: ‘research has revealed’ that we shouldn’t trust a press release

Research has also revealed some fairly predictable stuff about the best laid plans of mice and men.  There was briefly the promise that I might get paid to blog, but it evaporated, so my blog gets the benefit.  This was my ‘try-out’ piece.  It even got vetted by lawyers!  It just never quite hit payroll…  So apologies for the fact that the links are now a little out of date, but here it is: my take on AntiSocial Science (think Bad Science for the social sciences).

Research has revealed women who wear skirts and jackets are viewed as more confident, higher-earning and more flexible than those opting for a trouser suit.”  Thank you to Professor Karen Pine, of the School of Psychology at the University of Hertfordshire for this little gem.  I’d like to blame the Telegraph’s Science Correspondent Richard Gray for this er… oddly specific fashion recommendation, but I’m afraid he’s just the front man.  If you’re casting around for someone to glare at, Pine is your woman.

She, with the help of Mathieson and Brooke Tailors Ltd, put two sets of paired photographs on a webpage and invited people to judge them on confidence, success, trustworthiness, salary and flexibility.  The participants saw the images (of a man in bespoke and also in off-the-peg suiting and of a woman in a trouser and in a skirt suit, both with pixelated faces) for between three and five seconds.

Let’s think about that for a moment, before we move on to the results.  Firstly, yes, this research is sponsored by makers of bespoke suits.  Secondly, I’ll guess that the research participants who judged the suits were students at the Universityof Hertfordshire(but Pine doesn’t specify in her research summary).  The weight that we give to the conclusions drawn from this little investigation rests on the opinions of the research participants.  It’s therefore pretty important that we know who they are.  If they are in fact students, the weight I give to their judgement of suits is negligible: students are rarely called upon to wear suits, and even more rarely called upon to employ someone else wearing one.

The results show that the woman in the picture was rated more positively on confidence, flexibility and salary when wearing a skirt suit.  However, the statistic on salary (paired samples t-test with a sample size of 303) yielded a t-statistic of 1.92, which is outwith the commonly used 5% limit for statistical significance.  There is no mention of the results on success or confidence.  It is possible that there was no relationship between the value and the garment in these cases, or that the relationship favoured trousers.  Either way, I find the reporting a little unconvincing, to say the least.  It’s a bit like those advertising claims that “72% of Made-Up-Magazine readers agree this product makes their hair shinier”.

This doesn’t stop Gray (or Lynn Davidson in the Daily Mail) of course.  Still, I’m not prepared to let them carry the whole can for this piece of frippery.  Yes, they shouldn’t be reproducing what amounts to a press release from a tailoring company, but by employing an academic ‘beard’ Mathieson and Brooke have lent their enterprise an air of respectability.  It’s clever, but it’s hardly social science.

Posted in Journalism | Tagged | Comments Off on AntiSocial Science: ‘research has revealed’ that we shouldn’t trust a press release

Volunteering, social class and some other things

The NCVO/VSSN Researching the Voluntary Sector Conference took place in London last week.  Many of the big names were there, but a lot of the stuff I found most interesting came from Jon Dean, a doctoral researcher at the University of Kent.  He presented in a panel on “narratives” with Georgina Brewis and my Southampton colleague Anjelica Finnegan.  “Narratives” is an odd word: I think I’d be just as happy with ‘stories’, in general…  Brewis talked about the story of volunteering rates in the UK, particularly the un-proven but widely accepted volunteer ‘boom’ of the 1960s.  Finnegan spoke about a later ‘boom’, after the Labour landslide in 1997.  Again, there’s little evidence for it, but it makes a nice story. 

Dean’s topic was a little different: it underlined the importance of a good story.  The volunteer booms covered by Brewis and Finnegan are pretty much unsubstantiated by the stats: they loom large in the field because the stories are powerful enough to stand alone.  Dean is interested in informal volunteering which, it seems fair to say, does not loom large in the field.  His hypothesis is that informal volunteering is often overlooked precisely because it lacks a convincing narrative when compared to formal, organised volunteering.  Volunteering is thought of in terms of work and service, not in terms of mutual aid.  Informal volunteering does not fit this model; and it doesn’t lend itself to a single neat story of its own.

Informal volunteering is caring, lawn mowing, baby-sitting, cleaning, washing and shopping.  It’s DIY, cooking, dog walking and checking up.  It’s done for neighbours or friends or acquaintances (not for family).  It’s not used as a policy instrumentand it’s not generally associated with other positive individual or societal outcomes.  Government is interested only in formal volunteering, to the extent that the national indicators for local authorities (N16 in this case) are based only on formal volunteering.  Dean’s point is that deprived communities, whose members are more likely to be seen volunteering informally than formally, miss out twice.  They miss out on the positive influences of formal volunteering, and also on the government incentives which follow it.  Simultaneously, academics miss out on understanding the positive role of informal volunteering in a struggling community, and government misses its chance to encourage it.

This brings me to an interesting conversation I had with Rebecca Taylor (a ‘sociologist of work’ at TSRC, Birmingham) about class and volunteering.  Rates of formal volunteering are generally lower among working class individuals than middle class ones.  Why?  Time?  Money?  Or is it essentially cultural…?

Posted in Academic research | Tagged , | Comments Off on Volunteering, social class and some other things

Peer reviewed article ‘wrong’ shock

Dave Johns at Slate has written an interesting piece on self-publication and peer review, using a recent study about ‘social contagion’ as a peg.  I wrote a post back in March about some of the pitfalls of peer review.  In it, I wrote about the Social Science Research Network – a self/early publication site used by academics in politics, economics, social statistics etc.  I thought that perhaps the names and institutions of the authors and the number of citations the paper picked up could be used as a signal of the paper’s academic worth.

The social contagion paper Johns is complaining about was published on SSRN before being picked up by pretty much every single media organisation going (including the Washington Post, New York Times, Guardian etc etc).  It makes claims that divorce is catching.  It has been downloaded 779 times (download rank 9270 – for comparison, a really high number of downloads would be around 100,000) but has never been cited.  The authors are high-profile and come from well-respected institutions (Brown, UC San Diego and Harvard) but their paper has failed to attract any academic attention – either good or bad.  I’m tempted to say that citation signalling is doing its job: the paper is insufficiently interesting to even attract criticism, so we should take care that we don’t over-rely on its findings.  Still, it was only published at the end of 2009, so I am reluctant to bang my gavel just yet.

Perhaps this is a simple case of bad science journalism.  A (flawed) paper is picked up by (bored) journalists and instantly becomes fact.  The paper has been picked apart by writers and bloggers so the record is slowly being set straight.  Maybe we shouldn’t worry?  Not so fast.  Fowler and Christakis also published a very similar piece on social contagion (this time of obesity) in a famous and respected peer reviewed journal called the New England Journal of Medicine.  This social contagion article has picked up no fewer than 239 citations.  So it’s peer reviewed (that’s one signal) and well cited (another).  Can we feel confident in it?  Dave Johns says not, and there are plenty of other people including some in my blogroll who agree with him.  Johns relies quite heavily on a (peer reviewed, naturally!) paper by Russell Lyons which critiques Fowler and Christakis’ approach. I find Lyons’ paper extremely persuasive – but I’m no-one special.  How can a journalist know who to believe and what to print?

What can we conclude?  Journalists shouldn’t write about things they don’t understand?  Every news outlet should employ a statistician as a fact-checker?  It would be nice… unlikely, but nice!  Of course, academics should take care when speaking to journalists but part of the problem here is that researchers are under pressure to demonstrate the ‘impact’ of their research.  A key plank of ‘impact’ is dissemination of research findings in such a way that they can be picked up by users, such as policy-makers.  The media is an important intermediary – have a look at ESRC’s impact evaluation pages to see what I mean.  Academics are pretty much encouraged to talk to the press and the end results are not at all surprising.

In fact, I’m not sure that the end results are particularly terrible either.  Peer reviewed but problematic statistics were published in a simplified way in newspapers all over the world.  This made other researchers mad enough to do something about it and has resulted in a slew of public criticism.  Yes, it’s unfortunate that millions of newspaper readers still think that obesity is contagious, but it’s hardly the end of the world.  And, whichever way you look at it, people really shouldn’t believe everything they read in newspapers, should they?  The problem remains, though, that peer review has been somewhat ineffective… so it really worth the candle?

Posted in Journalism | Tagged | 2 Comments