I have growing concerns about what we have allowed qualitative research to become — and indeed I fear that we may be collaborating in sowing the seeds of destruction of our own industry.

I want to start with Mad Men, the award-winning TV show about advertising in the early 1960s. Fans of the show will know that the central characters are a couple who represent both the Metropolitan and Suburban personalities of early 1960s America. Don Draper is a creative director in a Madison Avenue agency; his wife, Betty hangs out in the suburbs at Ossining about 35 miles north of Manhattan in their perfect home.

What is fascinating about these two — and those around them — is how little self-awareness they have. They are veneers of human beings, somehow locked in a world they aspire to have an impact upon, but blind to themselves and the consequences of their actions. Stuff happens to them that they are genuinely shocked by, as if they had no part in precipitating it. This is much like the advertising world that I first came to know in London around 1969.

Think Joni Mitchell

Into this barren Valhalla came qualitative research, with its autobiographical nature and its capacity to view people in 3D: thoughts, feelings and behaviour. Alongside the autobiographical wave in popular music (think Joni Mitchell or John Lennon on Julia), we were able to meet and connect with people at a whole new level.

I won't pretend that every adman came along to group discussions, as they were known, but the discourse changed. It became possible to observe, discuss and portray aspects of ordinariness with compassion. From these origins the Golden Age of British Advertising was born. People like Ridley Scott, Alan Parker, David Putnam and Charles Saatchi got very creative, and very wealthy.

Much qualitative endeavour in those days was simply meeting people who had never been met before, whose opinions and lives were starting to matter to those who wished to influence them. Qualitative research stuck out above the surface like a proud nail as an alternative to stereotyped, overly made-up "housewives" who looked like awkward fashion models as they pretended to do the laundry in a 30-second commercial for detergent, while "scientists" cut up shirts and compared results.

We would never have had "slice of life" advertising with all its nuances and richness without qualitative explorations. We had sat for hours in tiny living rooms in Sheffield, flooded with the aroma of tea and biscuits and stood terrified in Glasgow pubs as we searched for the room that had been allocated for the group.

Whatever else we did, we got face to face with people, so that we could know them and portray them realistically. I have had the privilege of listening to such people — and found myself changed by this listening.

Now though, I fear we are moving away from people. We are going online or we are using topic guides in which the product, service or advertising gets much more attention. Research today is more about products than people.

There are several contributors to this growing distance between researchers and respondents but let me mention just the following five:

  • The bureaucratization of research. There are standard procedures, venues, processes, and templates for analysis, presentation and reports. All in all it adds up to a kind of PowerPoint World; slick and quick yes, but authentic — I don't think so.

  • The DIY Factor. As research became better known and more widely accepted, it acquired a kind of "anyone can have a go" character. After all, we all have conversations don't we — why shouldn't your clients" questions be just as useful as yours?

  • The unspoken benefit we get from not having to do groups face-to-face in out of the way places. If you can moderate the group from your desk, in front of your PC, you don't have to go out of your way to be with people — and neither do they. Surely everyone wins? Even the esteemed client can tune in from his desk and welcome the fact that his bill is smaller. However, as a climate for deep inquiry, it feels remarkably like the pretending in Mad Men that we saw way back there in the early 60s.

  • The continued shortage of advanced training and expertise for qual researchers. To my knowledge there is only one advanced training and there should be more. Do researchers appreciate the limitations to disclosure created by professional relationships — the type you initially get when you pay people for their attendance and participation? Do they know how to move from professional to personal, even private? What are the tools and processes you need? How much do researchers know about the psychodynamics of groups — aspects of group process that are always present and, unless raised into consciousness, will drive the group from the background?

  • If your qualitative work is about simple cause and effect relationships, these things may not concern you: but most is not. Most decisions and choices are derived from an interplay of observable and private reactions and understanding the dynamics of this relationship is often at the centre of a research brief, as it is at the centre of a decision to buy or not. Herein is the nub of the matter, to truly understand how choices and decisions are made it's necessary to experience both surface and beneath to figure out how they stand in relation to each other.

It is the deeper stuff that is far more likely to determine behaviour, so without it our answers are bound to be superficial and insubstantial — a growing criticism of qualitative work that most of us are not doing nearly enough to rebut. If we don't do something about it our proud nail will be flattened and the world will walk over it without noticing we are there. Herein lie the seeds of our destruction.