The "bad apples" debate
AQR is working on the ‘Best Practice‘ Guidelines to help resolve the issue of fraudulent respondents
Its amazing how history repeats itself. The March issue of Research magazine featured a cover story on what it termed Bad Apples, the fake and/or serial respondents who crop up in the course of qualitative work.
A search through AQRs archives will reveal that this is an old chestnut, but one that has never been satisfactorily resolved. Thats why, following the Research article, the topic was put up for discussion on our website.
Members were asked to consider four central themes. First, is a central database thought to be the answer, combined with identity checks? If so, who should bear the cost of its creation and implementation? Secondly, is the root cause linked to the speed with which research and field agencies are asked to recruit candidate? And if this is the case, is the situation ever going to get any better? Thirdly, how much is the case of the ever-more-niche respondent exacerbating the issue, and finally, is there a dearth of discussion between clients and agencies in relation to recruitment briefs and the recruitment process?
One month on and, at the time of writing, there have been 25 responses to date with another seven letters on the topic featured in the April issue of Research. And if you still havent added your comment to the AQR website, its not too late.
The majority thought fraudulent respondents were the biggest problem, but adopted a more relaxed attitude towards those who come under the repeat heading (see site for further details). The points raised included:
- The trend towards conducting groups in viewing facilities, and its impact on the problem of fake respondents
- The dearth of new respondents
- The need to better value, respect and cost recruitment
- The merits of going down the ID route
- The impossible pressures - time and cost - that researchers and recruiters find themselves operating under
- The Continental experience
So where does the debate go from here?
At the April AQR Committee meeting the issue was thrashed over once more and it was agreed that it did need resolution.
The Committee decided to formulate a set of best practice guidelines, endorsed and hopefully written by both clients and researchers, which will offer advice on how to deal with respondents — and recruiters — who arent perhaps participating as they should.
This will focus on problems that can occur with the original selection of respondents, maybe recommending random spot checks, to those that occur at the groups in question, even if this boils down to the researchers initiating post-mortems with recruiters the following day or shortly thereafter.
We dont live in a perfect world, said AQR chair Fiona Jack. Sometimes the clients specifications are difficult to adhere to. There is always an element of time pressure, but we do need to try and create a shared understanding of limitations, manage peoples expectations better, and for everybody to make it their business to raise the bar.
There is no question that the introduction of national ID cards, if and when it happens, will make a difference. Respondents will then be able to be checked when they arrive at groups to make sure they are who they say they are. But this wont resolve all problems. What is to stop participants finding a way round the system? Or exaggerating in groups?
As the Committee pointed out, it is human nature to behave differently according to your surroundings. A focus group is no exception, so all those participating — from the client, to the moderator and the respondents — will tend to present themselves in a different light to normal. In other words, there is no norm.
No matter what the original article in Research magazine would have us believe, bad apples are spotted for the most part by experienced researchers because they stick out from the rest. They may not be outed during the group, but their contribution will be viewed in a different light.
It is not the function of AQR to build a database of these bad apples. Cost is not the issue so much as it is not the Associations remit to police this area.
Which is why the recruitment guidelines are being scanned, to see if there is any way that they could be tightened up, while the Committee sets out to write the new best practice guidelines. These, when they are ready, will appear on the AQR web site and, in future, will be circulated with the guidelines.
A common aim
Will it be enough? Well, if it achieves a dialogue between researchers and clients as to the problems, and their causes, it will have moved this whole debate to a higher level. Those on both sides of the fence do, after all, want to achieve the same aim: answers, insights and understanding.
Did it need an article in Research magazine to set the ball rolling? If the AQR archives are anything to go by, it was time for this issue to be aired again and just happenstance that it happened this way. The challenge for the Committee — and AQR members as a whole — is to go some way towards resolving it at long last so that we dont, in 10 years time, go through the whole process all over again.
This article was first published in InBrief magazine, May 2006
Copyright © Association for Qualitative Research, 2006