Survey Data Not Be All And End All – Shock

Apparently there’s more to research than survey data – who knew?

Those who have been paying attention will remember that I have used the National Child Development Study (NCDS) to examine the relationship between volunteering and political activity for the first paper of my PhD.  It turns out that, as well as 50 years of fascinating survey data on a cohort of people born in a single week in 1958, the NCDS also includes the transcripts of 220 semi-structured interviews (each lasting around an hour and a half) given when the cohort members were aged 50.  This collection of 220 interviews is known as the Social Participation and Identity Study (SPIS).

I’ve just taken a short break from my PhD research to do a collaborative (ESRC-funded) research project using this lovely stuff.  I collaborated with Jane Parry (Southampton) and Katherine Brookfield (soon to arrive at Edinburgh) on a mixed-methods report looking at a sample of 50 of the semi-structured interviews (called “50 at 50” – geddit?).  Our full report will be available shortly at   It focuses on participation: in clubs, associations, voluntary organisations etc.  It covers the differences between participation data collected in surveys (quantitative) and in open-ended interviews (qualitative).  We also looked at people’s motivations for involvement and the barriers to participation that they face.  Finally, we included a chapter on participating in a cohort study (actually, more accurately, I should say that Katherine and Jane did that – I stuck mostly to the numbers and to formal social and civic participation).

For now, I’ll stick to the differences between quantitative and qualitative data when it comes to studying participation.

We used NCDS quantitative data to make a ‘sample’ of 50 out of the SPIS 220.  The quantitative data covers the whole of the cohort members’ 50 years, but we used only data from the five adult waves (cohort members were surveyed at age 23, 33, 42, 46 and 50).  We singled out:

  • non-participants,
  • perennial participants, and
  • frequent participants.

Non-participants answered ‘no’ to a slew of participation questions at every wave – they didn’t even go to discos when they were 23!  Perennial participants answered ‘yes’ to ONE question per wave about active involvement with a formal/membership organisation.  Frequent participants were active weekly in either (a) many spheres at once, or (b) voluntary work

Looking only at the quantitative survey data, perennial participants and non-participants are indistinguishable in terms of gender, social class, education, marital status, childrearing and care for elders.  Frequent participants (nb: only 8 of them in our sample) look noticeably better educated than their less-involved peers. They were also:

  • healthier
  • less engaged in caring for elderly parents
  • more likely to vote
  • more likely to be self-employed.

For those familiar with the literature, this should be raising some eyebrows.  We would generally expect volunteers, for example, to be better educated, and from the professional/managerial social classes.  We can certainly see some of those differences among the frequent participants, but not among the perennials.  The take-home, however, in spite of these small differences, is that it’s pretty hard to pick someone’s likely participation status from the rest of their NCDS data.

Looking at this from other side (the dark side?), the qualitative data shows that, as you might expect, frequent participants really are very busy people.  John, one of the frequent participants*, launches into a description of the sports coaching activities he and his wife engage in with the words: “My wife gets up at just after three.”  Frankly, just reading the interview is exhausting.  While both data sources generally agree for the frequent participants, the qualitative interviews show that the majority of non-participants are more engaged that their quantitative data suggests.  Out of 21 non-participants, just seven failed to come up with any participatory activity at all in their qualitative interviews.  Furthermore, some perennial participants do really very little to ‘earn’ that tag.  Some perennials had just attended church services, or played golf at a club – and this put them in the same category as others with much more active records.  Finally, some types of participation were all but absent from the qualitative interviews: political acts and trade union activity were far more prominent in the quantitative than the qualitative data.  (By the way, this last comes under the general heading of ‘if you don’t ask, you don’t get’ – there was little chance that cohort members would be prompted to describe their trade unionism or political acts, given the interview schedule.)

Survey timing clearly played a role here.  Some of the difference between quantitative and qualitative data is down to “events, dear boy, events”.  The gap between quantitative and qualitative data collection was anywhere between one month and two years!  Alison, for example, referred to a changing economic backdrop:  “I think probably we’ve felt the credit crunch crunching over the last few months, so we probably do stay at home a lot more”.  Iona had been involved in various activities and organisations at the time of the NCDS quantitative interview.  However, between then and the SPIS qualitative interview, she was diagnosed with cancer and had started a programme of treatment.  As a result, her participation and volunteering commitments had been suspended (“since September I’ve had somebody else’s life really”).  In that case, the time gap was only three months, but it was very significant for the findings.  The quantitative data identified Fiona as a frequent participant on the basis of her participation in multiple organisations at age 50. However, the qualitative interview data indicated that she rarely participated in any organisations. Highlighting the role of timing, in her SPIS interview Fiona mentioned that work commitments, and the demands of studying part time for a new qualification, had, in the past two years, forced a reduction in her participation activities.

There is also a strong question effect.  The NCDS participation questions are generally predicated on ‘joining’ or membership.  Before cohort members have a chance to tick a box describing their activities, they must first pass a ‘membership’ hurdle.  The survey asks first “Are you a member…?” and only then “How often do you join in the activity of…?”.  In the case of religious activity, this is visible in the quantitative data alone.  Three cohort members (Iain, Susan and Anwen) said that they attended weekly services but were not members of a religious organisation.  When quantitative and qualitative data are compared, it’s clear that ‘membership’ is perceived differently by different cohort members.  For example, Michael (a perennial participant, according to the quantitative data) appeared to identify online contact with several clubs and receiving email newsletters, both rather passive forms of engagement, as ‘joining in’ while Paul (a non-participant) did not identify his membership and subsequent attendance at National Trust properties as ‘joining in’.  The membership hurdle also rules out participatory activities like evening classes and episodic volunteering.  For example, Janet (a quantitative non-participant) was involved informally with a children’s charity through her husband.  She says:

“My husband is a taxi driver and there’s a–, the only volun–, the only charity work that I do is help–, he takes part in a–, it’s a one day kiddies’ outing to PLACE IN SCOTLAND… so we’ve got to blow up the balloons and get everything ready but it’s–, it’s a couple of weeks of preparation, getting everything in, then you’re up at four, five in the morning to put this stuff onto the taxi and things like–, so that’s–, that’s about it.”  The interviewer then asks: “Have you been doing that for quite a while?”  She replies: “Oh, 20-odd years now…  It’s–, it’s maybe not a lot but it’s a lot to me [coughs] ‘cause we pay for it all ourselves as well.”

This certainly sounds like volunteering – but Janet’s quantitative NCDS record suggests no such activity.  The survey questions did not capture her episodic, family-linked volunteering (perhaps partly because she seemed reluctant even to use that word).

Including religious activity as a form of participation has also affected the findings. Some religious participants are very active: running choirs, or Sunday schools, or involved in church administration.  Others just go to the service.  Iain, for example, said: It’s a sanctuary type thing and really, that’s all it is. If somebody were to say to me, “What did the priest say last week?” I couldn’t tell you what he says. I listen in wee bits, but the majority of the time it’s just me and that’s what I get out of chapel.“  This is about as non-participatory as participation gets!  Louise, on the other hand, revealed in her qualitative interview that she was spending 1.5 to 2 days a week on her church-related activity.  Depending on how the cohort member views their own activity, these people can look pretty much identical in the quantitative data.

So the message here…?

  • When you use a survey question, you rely heavily both on question-order and on the respondent’s interpretation.
  • Unless you are interested purely in church-going, it may be better to ask specific questions about church-related participation.
  • If you don’t ask, you don’t get.

And finally – don’t for a moment think that this has put me off using survey data; it’s just made me determined to use it better.


*All names are pseudonyms.

This entry was posted in Methods, PhD and tagged , , , , , , , . Bookmark the permalink.

2 Responses to Survey Data Not Be All And End All – Shock

  1. Essi Lindstedt says:

    Interesting blog. I think there is an issue in relation to observing church participation, and as I think you suggest at the end, it’s important to be clear on what your interest is. So if you are interested in whether they contribute to the social life or social action of the church, to be specific about that. But from a religious point of view there is a sense that ‘something happens’ during worship, and that that itself is a form of participation, in a way that sitting on a bench at a golf club probably is not. I find it difficult to dismiss someone who finds sanctuary at a church service as not participating but I agree that it is not volunteering. Sometimes those who are ‘busy’ may feel supported and enabled by those who are quiet and who held up the spiritual element of church life. Rather in the way that an organiser needs people to organise, and performers need an audience, institutions need a variety of contrasting dispositions among their members even if they don’t themselves recognise that formally.

    • Vicki Bolton says:

      Essi thank you for reading and commenting! It’s very kind – and very useful, especially as I try to work out what language best describes the things I’m interested in.

      It was fascinating to me how diverse religious participation is. It shouldn’t have come as a galloping shock, of course, but I was so focused on the diversity of social participation that I wasn’t really paying proper attention elsewhere. You’re right that the key is to be clear what you’re interested in: and that was one of the problems we had with this study. We inherited the idea of social participation from the people who designed the interviews (the Centre for Longitudinal Studies) and we didn’t examine it as much as we could have done. For me (and aside from volunteering), social participation is marked by an active engagement with other people in real life: if you haven’t spoken to anyone, you haven’t been a social participant. The quote from ‘Iain’ hinted at quite a detached attitude to church-going, but it’s hard to imagine him being able to slip away from a service without speaking! One of the things which interested me is that Iain ticked the same survey boxes as someone much more active in the social/administrative/activist life of their church.

      To allow the subject to drift slightly… One person actually told the interviewer how disappointing it was not to be asked more about his faith. I wonder whether people were reluctant to speak about their faith or the spiritual element of their lives – it’s not something that’s well-addressed in British public life, I don’t think. So it’s extremely interesting to me that Iain talked about sanctuary – he was in a minority, even among the churchgoers, in using any ‘faith’ words.

      And to allow it to drift even more… I’m reading Quirkology by Prof Richard Wiseman. He has a chapter on spiritual and supernatural experiences and wondered whether the very large organ pipes, which emit infrasound, may be contributing to the experience of churchgoing by vibrating the church/pews/people almost imperceptibly. Fascinating stuff.

Comments are closed.