The Association for Qualitative Research
The Hub of Qualitative Thinking

Online: View from US

Martin Redfern charts the growth of qualitative research online. But for all its benefits, he warns, it also demonstrates all that is good and bad about the Internet itself.

Research buyers and suppliers in the US are not immune to the current mania for transferring almost all aspects of life and business on to the Internet. In the current context, to be off-line is akin to being unsophisticated and behind the times.

This is especially true for companies at the cutting edge of new technologies — and the researchers who work for them. In the last three years, we have witnessed a significant move toward online research in North America, with both positive and negative results.

Online qualitative research stands as a prime example of both the good and the bad in the rush to go online. It would be quite wrong to believe that the Internet offers a superior medium for most qualitative research, but it would be equally wrong to ignore the real advances in research methodologies that underlie the hyperbole.

In truth, online qualitative research offers a new and exciting tool which, when used appropriately, will allow us to conduct research that was simply impossible in the past.

The primary advantages of online qualitative research, as it is currently implemented, are the opportunity to bring geographically disparate people together and the opportunity to show them interactive or static stimuli in real time and gauge their reactions.

The geographic limitation is so basic to traditional focus groups and in-person interviews that we have taken it for granted. While some have pursued the use of conference calls or videoconferencing, the traditional approach of conducting separate ‘live’ groups in separate markets has been dominant.

Of course, the appeal of the new approach is not that researchers can examine different markets within one focus group. Discerning regional-differences will still require multiple region-specific groups, whether traditional or on-line. Instead, the advantage lies in bringing highly specialized individuals together, who are too rare in the population to support a traditional focus group in a single centralized facility. These might, for example, include rural physicians, users of an experimental drug, very high-value investors, or franchisees.

Exposing respondents to visual and audio stimuli has always been a key advantage of qualitative research over the telephone-based quantitative methodologies that dominate market research. This advantage is enhanced in most online environments because each respondent sees up close, on their screen, a clear reproduction of the stimuli, without the inherent costs (and potential pitfalls) of high quality colour duplication.

More importantly, however, on-line environments can offer easy access to interactive stimuli, the most obvious example being multi-level web sites. As those who have qualitatively tested web sites via traditional means can attest, there is a great deal of equipment, supervision, and luck required to obtain a truly useful result. In contrast, any online focus group implementations will easily allow respondents to chat while viewing a web site, either on their own or under the direction of a moderator.

This is best illustrated by example. A common way to test a new web site with consumers is either to provide each respondent in a focus group with a laptop or to conduct in-depth interviews sequentially using a single computer. The computers include either a live Internet link or (more reliably) a copy of the new web site stored on the hard drive.

This approach has some advantages, as discussed later, but also has some real disadvantages. First, equipment and frequently technical assistance must be shipped from location to location. Second, inevitable technical failures mean a significant inconvenience to researchers, respondents and clients alike when sessions are cancelled.

Finally, the web site cannot be modified quickly to test new ideas, especially if it is being tested in multiple locations simultaneously. In comparison, online testing uses a centralized server including a single centralized copy of the web site to be tested. The web site can be changed quickly at that location. Finally, technical problems mean far less inconvenience for all involved if a session must be cancelled or rescheduled. Thus, online testing provides real, practical advantages over in-person testing.

Equally importantly, there are many other ways in which interactive visual stimuli can be tested online qualitatively, and they will all benefit from the advantages discussed above. Some will be better implementations of current in-person methodologies, while others will be applications that have not even occurred to us yet.

One can imagine advertising testing where respondents choose which of two scenes they think is most effective, and then choose between two following scenes, and so on, thereby creating the advertising themselves. Also, it would be possible to conduct interactive experiments with elements (such as temporary isolation, different stimuli, and ‘planted’ respondents) that would extend experimental design well beyond the techniques currently possible when respondents are present in person.

The geographic and technical merits of online qualitative work are compelling, but they are subject to two broad caveats. The first is technical. The second is methodological.

Technically, online qualitative research is in its infancy. This is not through any lack of industry or intelligence on the part of the researchers or programmers. It is because the installed technology base upon which it relies (the computers which respondents own and the means by which they connect to the Internet) will not yet support the sort of high-end research designs which we would want to implement.

Bandwidth limitations prevent us from testing full-length video clips in a format which provides acceptable quality. To test even moderately complicated web sites or creative elements we must limit our respondents to those with high-speed connections, with a concomitant degradation of our sample’s “representativeness”. Yet, to do otherwise would result in on-line discussions that dragged on interminably while waiting for elements to download to respondent’s computers.

Even now, limitations in speed (at both ends) may result in conversations that lag, meaning either long gaps in the conversation or, worse, responses to questions which arrive confusingly after another question has already been asked.

Looking ahead, there is little doubt that on-line qualitative work will have to provide more ‘human’ interaction than text boxes and typing can provide. Voice and video will be needed if the depth of online findings is ever to match those of in-person research.

This is not simply a question of bandwidth — although a great deal more of that will be required -- but also a question of reaching a point where most respondents have both voice and video-capable computers. Indeed, one suspects that research companies may find themselves providing this technology to panels of potential respondents, much as has occurred in other research domains.

Even at this early stage, it is necessary to ensure that respondents have the necessary software (such as browsers and media players) installed in advance before the sessions can begin, and that they have adequate bandwidth to support the limited uploads and downloads required.

A final technical challenge is that of security. Those familiar with traditional methods of testing new commercially-sensitive concepts are familiar with the process of numbering each item given to respondents and ensuring that no storyboards, mock-ups, or prototypes are taken away by respondents.

Furthermore, being present in person invokes, we assume, a certain reticence on the part of respondents to do anything untoward like stealing product examples or misrepresenting their occupation to avoid screening criteria intended to prevent industrial espionage.

In the online environment, this presents a real challenge. First, electronic media, whether images, video, or web site contents, can be ‘captured’ by any respondents with a moderate amount of technical know-how. Some software packages that purport to prevent this are now available. Second, the somewhat anonymous culture of the Internet will tend to make dishonesty -- such as claiming not to work for the press or a competitor -- somewhat easier.

Both these factors increase the chances of inadvertently losing control of commercially sensitive concepts before they are ready for the public. While the chances of such breaches are, in absolute terms, relatively small, the consequences are serious enough to give any researcher pause for thought.

Methodologically, there are serious limitations to online qualitative research, many of which stem from the technical limitations noted above.

Of perhaps greatest concern, however, is the extent to which participants in online focus groups represent the target population. Even in North America and Europe, Internet access does not generally exceed 60% of households, with a significant skew towards more affluent and better-educated people, making online research highly suspect as a source of general public opinion.

Nonetheless, many target populations -- most notably in business-to-business research -- are sufficiently ‘wired’ to allay these fears. Furthermore, research targeted specifically at the online population is ideally suited to online research. This may include testing potential web sites. Even so, the sample limitations imposed by the medium may frequently become onerous, requiring a return to more traditional (and representative) methodologies.

The fact that current approaches rely heavily on text input, by the moderator and respondents, places a very high premium on typing skills and literacy. Typing, which it should be remembered was a specialty skill a generation ago, is certainly a common ability among many people today, but typing quickly is not.

The inability of respondents to express themselves in text at anything near the rate they communicate in speech places inherent limitations on the amount of information which can be exchanged within a given timeframe. Respondents know this, and will begin to respond in shorter and shorter phrases, without embellishment, in order not to be left behind.

Even more importantly, many people cannot express themselves in text as they can in words. Researchers who use mail surveys or written exercises in focus groups will recognize that grammar, spelling and clarity are not universal skills. These limitations may prevent otherwise extremely helpful respondents from contributing fully to the discussion, whether due to these limitations or embarrassment about these limitations.

By their textual nature, online focus groups are literate and reasoned interactions. This, critically, is an artifact forced on them by the research medium, not a reflection of the respondents’ true natures. Qualitative researchers know that there is a great deal of learning to be found in the non-verbal communication which respondents exhibit. Do they slouch and look bored? Do they nod when someone else makes a point? Do they laugh derisively?

Without the audio and video links discussed earlier, such cues are lost entirely in an online context. Instead, the emotional reactions we see are all mediated by the format of text and the filtering that occurs between a person’s reaction and its commitment to the screen. To seasoned moderators, online focus groups will initially seem rather sterile affairs. To be fair, certain conventions, known as ‘emoticons’, will allow Net-savvy respondents to use punctuation or abbreviations to express non-verbal reactions, but these reactions are still filtered, not spontaneous.

Like most new technologies, there is a tendency for online research to emulate the old approaches, even as it purports to break out of old-style thinking. Online qualitative research mimics, in most ways, the environments and formats with which qualitative researchers and their clients are comfortable. Software programmers have gone to significant lengths to create virtual emulations of the tried-and-true focus group facility. These environments are usually replete with waiting rooms, observer rooms, and the familiar format of a single moderator speaking to a group of strangers, guided by an agenda designed in advance to cover areas of particular interest. Some of the old formats are preserved because they are good and useful, honed through years of ‘live’ qualitative experience. Others are preserved simply because we have not yet had the courage to examine in depth how the Internet could allow us to conduct qualitative research in new and entirely innovative ways.

In the final analysis, qualitative online research is an exciting new tool with a great deal of potential. It will allow us to conduct traditional research more easily and conduct other research which was previously impossible. The geographic scope and interactivity of this research medium hold considerable promise. The intelligent use of this approach will require caution, however, and a willingness to acknowledge the technical and methodological limitations of online qualitative work.

We must, however, look beyond how to make online qualitative research do the work of traditional qualitative approaches. Instead, we must discover how to create better qualitative research via the Internet than we could before. Improving technology will take us some distance in that direction, but it is innovative thinking by qualitative researchers that will finally bring this methodology to maturity.


Martin Redfern
Copyright © Association for Qualitative Research, 2001