Before this study, our evidence was almost entirely anecdotal. We knew that:
- Sometimes moderators recognise the same faces;
- There are occasional horror stories of 'professional respondents' who make a living out of going to lots of groups;
- Every fieldwork company has occasion to censure recruiters for re-using the same people.
But do these anecdotes represent 'rare problems' unfortunate but inevitable glitches in an otherwise sound system or is there an endemic problem?
The aim of this research was to address only this issue it was specifically not about:
- The importance of repeat attendance. The notion of 'groupies' is generally regarded with dismay, but equally there is evidence that some prior experience of attending groups has advantages (Gabbott & Cordwell; Hayward & Rose, etc).
- Fraud (instances where respondents claim to be something they are not). This is an issue but it's hard to prove. In car groups, you can ask for registration documents but it's harder with things like 'watches four out five episodes of The Bill'. It's probable, however, that fraud increases with repeat attendance given the difficulties of fitting a different set of recruitment criteria each week.
Oh, and bear in mind that there is no industry-wide agreed standard on what to expect each company has its own view of how many previous groups it's ok for respondents to have attended, and the draft Recruitment Guidelines suggest that they should be project specific. Is there an argument for a 'normative' standard, to act as a default to apply unless there are good reasons to change it?
But let's return to the survey and the search for some basic facts. With the help of quant guru John Samuels, AQR helped initiate this study conducted by BMRB. The survey was based on a random-location sample of 10,000 respondents at over 1,000 sampling points including all major conurbations. Put another way, the area covered by the survey included all the places where we do groups.
The research can be looked at in two ways. First, in terms of the sample as a whole, 3% of the total sample had taken part in a group in the past year. Of these, the vast majority (75%) had been to just one group and a further 18% had been to two or three. So the great majority of respondents are not 'groupies'. The survey, however, found a small number 1% of all the people who'd been to groups attending 16-plus groups in the past year! Bear in mind, though, that this represents three people responding.
The second way of looking at the results is to imagine what 'Repeat Attendance' would mean if these people were spread evenly across groups (which is most unlikely given the purposive way we recruit). Then we would 'get' four first-timers, two who had been a couple of times that year and two who were real regulars. Even here, though, it's possible that even the '16+ brigade' are 'legitimate' repeat attendees for example, doctors or IT consultants in sectors where the accepted recruitment practices permit multiple attendance.
Whatever the case, we have to accept that this nucleus of regular-repeat attendees exists and that, although you are unlikely to get more than one in a typical group, moderators need to be alert to them. We are fortunate that Gabbott & Cordwell have shown us how to manage these 'groupies'.
More importantly, perhaps, awareness of their existence reminds us all (clients and researchers alike) that, if we don't allow sufficient time for recruitment, recruiters will call upon this pool of willing respondents to fill the group. Indeed, it may be no co-incidence that repeat attendance is most common among 'hard to recruit' targets single adults with no kids, and the most 'popular' targets ABC1s under 35.
The research demonstrates that the people who go to groups do lots of quant research, too. Hardly surprising, but as a consequence it seems that qual also faces the more general market research industry issue that of falling response rates.
So should this survey worry us at all? Is it only a problem for other people's recruitment? Can we handle it with agreed norms, better notice and more realistic quotas? If you think the issue is threatening to undermine qual credibility shouldn't we start to tackle it before it flattens us?