Testing quallie skills
Lorraine Davidson puts a computer-aided analysis system to the test and asks: is it worth the effort?
Clients may select particular quallies to work with depending upon the perceived value they bring to the process of analysis and interpretation. To outsiders, these analytical processes may seem little more than subjective judgement on the part of the researcher. This is, of course, true to some degree, although we'd argue this to be grounded in the words and thoughts of subjects and supported by sound reasoning and seasoned intuition.
But as professionals, is there a need to re-appraise our approach to analysis and interpretation to demonstrate that we are actually adding value for clients and that our reasoning is indeed sound?
Working on a variety of commercial projects, often as sole moderator with responsibility for distilling and presenting the findings, I became intrigued by the idea of challenging myself as an experienced qualitative researcher, both on behalf of clients and for personal development.
An opportunity arose to compare the outputs of my traditional approach to analysis and interpretation (i.e. revisiting audio and visual data, making extensive notes and allowing conclusions to crystalise) with use of Computer Assisted Qualitative Data Analysis (CAQDAS), in this case Xsight. Could this possibly enhance client outputs in order to merit the necessary time and cost investment on my part?
The project took a different approach to other studies in this field by documenting the introspective, personal journey of one practitioner as she sought to measure her existing skills alongside use of technology (in this instance, with first time use of CAQDAS). Feedback was also gathered from a sample of qualitative researchers around the world on their use, or experience, of CAQDAS.
Four focus groups were held on shopping for clothing and attitudes to particular retail chains among adults (over 40s) and children (under 12s), both of which were outside the core fashion segments, to enable views to be compared and contrasted. A variety of projective techniques were employed. I firstly undertook the traditional analysis and, after a period of time, began the analysis using XSight, allowing time to learn the package.
The study revealed some interesting findings and has resulted in a notable change in my working practices. Use of CAQDAS brought transparency to the formulation of insights, specifically, being able to track points at which these were derived, with links back to particular sections on-screen. The whole cognitive process of interpretation could therefore be readily documented and potentially recalled at a future date.
Adopting learnings from this exercise therefore and applying them to traditional analysis, I now keep a project diary to document areas which lead to particular conclusions, as these unfold in my own mind. This enables me to have an alternative pair of eyes, similar to another team member, but on a much more personal level.
Importantly, however, I did not feel that use of CAQDAS (XSight), on this initial occasion, brought any significant difference to the conclusions that I made through the traditional off-screen approach.
Still, on-screen file organisation did lead to a neater approach; data was readily retrieved and the whole thought process could be picked up where it was left off, even a few days before. Neither was the cost prohibitive given overall training and time input.
Some disadvantages were, however, noted. For example, there was the initial tendency to gather too much data by using the query tool at every opportunity, resulting in not being able to see the wood for the trees. Furthermore, visual material generated through creative collage work did not lend itself to CAQDAS analysis, although proficiency with the software may of course develop with further use.
I gained feedback from 132 qualitative researchers, by means of an eSurvey in June 2008, of whom 79% were micro organisations similar to myself. This suggested that listening to tapes/videos, transcribing and analysing from notes was the prime method of analysis for focus groups and in-depth interviews.
Interestingly, use of CAQDAS (a variety of packages) was less than 5% overall. Key reasons for this low usage were that many researchers did not feel it would provide additional value to their work and those who had tried it felt that it generated too much data, thereby cluttering the process of analysis.
The key barriers to use among the 108 respondents who hadn't used it relate to the anticipated expense of gathering transcripts, as well as time and budget constraints for training and ongoing use of the software. Importantly, however, there was a desire for some to differentiate the skills of the qualitative researcher, in the eyes of their clients, from what they regard as a more scientific approach.
Given the strength of these sentiments and the often tight turnarounds required for commercial projects, use of CAQDAS in commercial qualitative research is not expected to have significantly increased since the previous survey was undertaken.
So is CAQDAS worth the effort? In my opinion, this exercise helped me to understand more about my own thought processes, while also providing re-assurance that my skills are sufficiently developed to identify all salient points and even additional areas that were not revealed by use of CAQDAS.
CAQDAS may not be my preferred choice of analysis but I may re-visit it for longer term projects, to crosscheck that nothing has been missed and to keep a thorough record for future work or, indeed, to learn other packages. For some clients, it may also provide confidence in a more "objective" approach. True value, however, was derived from the personal challenge, while keeping abreast of technological developments.
Copyright © Association for Qualitative Research, 2010