Becoming Aware of Research Errors – Dialogue with a Novice
“What’s the topic of your research?”, I asked when discussing her project. “Bilateral stimulation. Psychotherapists train in EMDR, because their therapeutic approach doesn’t work”, she replied without hesitation. Something in her tone suggested that she had already figured it all out.
“So you’d like to use a quantitative methodology?” Her response surprised me: “No, I prefer qualitative enquiries.” This is a mistake that novice researchers often make. They start with their preferred methodology or method, instead of the research question. It’s the other way around. Because the research question determines, which methodology and methods are appropriate.
Discussing this further, she became more inquisitive. “What do I need to do?”, she asked. “Once you’ve got an area you’re interested in, you must think about your research question. Then you consider how to investigate it.” “Easy”, she replied. “BUT, your approach also depends upon certain assumptions”, I said. “Hmmm, I haven’t thought about this”, she responded, suddenly sounding unsure.”
Like other novices, she hadn’t considered the foundations of research. Philosophical assumptions about reality and knowledge form its basis. Both methodology and method rest on philosophical presuppositions.
“What’s your ontological and epistemological perspective?”, I asked. “My what???” It was obvious that she had no idea what I was referring to. “Well, researchers have their own views of what constitutes reality, something that’s called ontology. And what they accept as knowledge, that’s epistemology. These views guide their thinking.” “Oh”, she sighed.
Elaborating, I added: “Ontology and epistemology form part of what social scientists call a paradigm. Have you heard of Thomas Kuhn, who wrote about paradigms?” “Never heard of Kuhn”, she uttered. She hadn’t come across the term paradigm either. “A paradigm guides scientists and tells them how to solve problems. When it stops to be useful, a paradigm must be replaced. Her facial expression indicated confusion. So I tried to explain more: “A paradigm describes a shared world view across a discipline. It’s informed by philosophical assumptions about the nature of reality.” “Aha”, she muttered, “I’ve got a vague idea now.”
“Do you believe that there’s an objective reality that exists independently of you?” She looked perplexed. “Or do you believe that there’re many realities, which are dependent on context, time and space? And are personally and socially constructed?” I waited and allowed her to ponder this. “One reality”, she responded eventually, “can you say a bit more about epistemology?”
“How do you know what you know? That’s epistemology”, I said, “what do you accept as knowledge? How do you know that something is true? What’s your evidence when saying that psychotherapists have trained in EMDR, because their original psychotherapeutic approach wasn’t effective?”
She thought about this without revealing what her epistemological perspective was. Instead, she said: “It’s obvious, otherwise therapists wouldn’t train in EMDR, would they? That’s why I trained in it.” She smiled confidently. “Aha, so you don’t have other evidence?” She looked at me, slightly cross. “You generalise, taking your own experiences and assume that they tally with the motivation of every psychotherapist.” “Yeah, and?” Her inference didn’t surprise me. Jumping to such unwarranted conclusion is a common mistake.
“In order to prove this hypothesis, you must use a quantitative methodology”, I uttered. She didn’t seem convinced. “What’s methodology?”, she wanted to know. “Methodology is an approach, which determines how to study something.” “Great”, she said, “I’ll use a quantitative methodology.”
“Okay. What methods could you use to answer your research question?” “Are method and methodology not the same?” “No”, I replied. Often novices don’t know the difference between the two. “Methods are means for gathering data and form part of your methodology”, I said.
“How about if I use a qualitative methodology?” She had read a chapter on qualitative research. Telling me enthusiastically about her ideas, she then stated: “I want to carry out in-depth interviews and use interpretative phenomenological analysis, IPA.”
“You can’t”, I said wryly. Choosing a methodology, which doesn’t fit with the research question is wrong. Everything is connected. In other words, specific methodologies and methods are associated with particular paradigms. One can’t arbitrarily mix and match. The positivist paradigm relies on a quantitative approach and methods. Using a survey design would be a possibility. To test her initial hypothesis, it would be suitable.
In contrast, IPA sits within the qualitative paradigm and is associated with constructionism or constructivism. IPA is used to study subjective experience. Experience is regarded as a legitimate area of enquiry, which provides useful knowledge about a topic. Interpretative phenomenological analysis is derived from Husserl’s philosophy of phenomenology, his study of consciousness and self-awareness. Moreover, it draws on hermeneutics, the study of interpretation. Initially elaborated by Heidegger and Gadamer, IPA however can’t provide answers to research questions about causality.
“Why is IPA unsuitable?!?”, she demanded again, distrustful of my objection. “Well, as we discussed a moment ago, research is guided by philosophical beliefs about the nature of reality and what counts as knowledge”, I said. “Your hypothesis is in line with positivism. So you must use a scientific method. Experiments or randomised controlled trials, for example. These methods are the only means to establish with certainty how, why, for whom and under which conditions something works or happens. Since your research aims to prove a causal relationship, namely that ineffective psychotherapeutic modalities are the reasons why psychotherapists train in bilateral stimulation, you can’t use IPA.” She remained doubtful.
Understanding the basic tenets, i.e. which methodology and method fit with which epistemological and ontological assumption, is crucial to a successful research project. I tried to make her understand:
“Are you not suggesting that psychotherapists have noticed that CBT, existential, psychodynamic, or any other form of therapy, they originally trained in, doesn’t produce the desired outcome?”, I queried. “Yeap.” “And that this causes them to train in EMDR, or to be more precise, to use bilateral stimulation?” “Yes, that’s what I want to demonstrate”, she acknowledged. “So you must use both a quantitative approach and method.”
There was another difficulty she hadn’t considered. “You can’t equate EMDR only with bilateral stimulation. There’s more to EMDR than eye movements”, I said. “Really?” “Yes, it’s like saying ‘interpretation is psychodynamic or group analytic therapy’.” She paused: “This hasn’t crossed my mind. It’s more complex than I’ve thought.” Research always is. Novices tend to underestimate the amount of thinking and groundwork a good research project requires.
“You’re interested in bilateral stimulation only?”, I clarified. “Yes.” Wandering again into the world of positivism, I suggested: “If you’d like to research one variable, that is, bilateral stimulation, you could carry out a survey. It could give you some insight into why psychotherapists trained in bilateral stimulation. The analysis would be relatively simple, you could use descriptive statistics and calculate means (averages), for example. SPSS does it for you, so you don’t need to worry about complicated stats.”
“Okay, how many participants would I need?” “You’d need a large sample, ideally all psychotherapists, who trained in bilateral stimulation.” Not wanting to recruit so many participants and wavering again, she asked about the advantage of quantitative research. “Others could replicate your study. If all of you’ll achieve the same results, your results aren’t only valid, but also verified.”
Wrinkling her forehead, she muttered: “I don’t like surveys and numbers. I’d rather study psychotherapists’ experiences. Can’t I do that?” “Sure. But then you need to change your research question.” “What???? And I thought that I’m making progress!”, she exclaimed.
It often takes time to come up with a solid research question. Impatience and not aligning methodology and methods with their underlying epistemological and ontological assumptions, result in fatally flawed research projects. Being too hasty initially doesn’t pay off. So tolerating frustrations and recognising that devising a sound research question takes time, is as valuable as learning about philosophical underpinnings.
“Perhaps I believe in multiple realities and subjective knowledge after all”, she wondered. “Would that be in line with IPA?” “Yes, if you’d like to explore psychotherapists’ experiences.”
“But I want to know why other therapists have chosen to train in bilateral stimulation as well”, she insisted. It became clear that she hadn’t grasped the foundations of qualitative research either. “A qualitative methodology is unsuitable when you’d like to establish causation and try to generalise. Generalisability means that you can universally apply the results from your study. For qualitative research, generalisability is a problem. Because qualitative researchers tend to explore the idiosyncratic characteristics of a small, selected group. Therefore, their findings are only transferable to a similar group.” “Hmmm”, she looked confused.
“Use IPA and others could build on your research”, I suggested. “How?” She seemed motivated to familiarise herself with IPA. “Think about where you can recruit, how many psychotherapists (participants) you need, interview them by asking open-ended questions and then analyse your data by identifying patterns, themes etc.” “Sounds also complicated”, she responded. We were both silent for a moment.
“Actually, when I think about it, I do believe that there’s only one, independent reality. A physical world does exist. But I also like constructionism. Is there another approach I could use?”, she asked. “Well, it sounds like you’re a critical realist after all. Critical realism rests on assumptions about an independent reality and socially constructed knowledge. You could use mixed methods, both quantitative and qualitative.” She smiled.
“You’d have to rephrase your research question”, I said. “Fine”, she responded. “How much is known about the relationship between choice of psychotherapy training and bilateral stimulation?”, I asked. She hadn’t searched the literature and didn’t know. Unfortunately, that’s also a common mistake. Either not knowing the research literature or having conducted incomplete, arbitrary searches, but making bold statements about a topic nevertheless. A thorough literature review prevents repeating what other researchers have done, and tears later on.
“Let’s assume that little is known about your topic”, I suggested. “Initially, your research project could be qualitative. You could investigate psychotherapists’ experience. Once you’ve found similar themes and patterns in your data, you could use a quantitative method to validate these.” “Not a bad idea”, she said, “tell me more.”
“Well, your findings may give you some insight into the decision-making process of psychodynamic therapists. However, those findings would only apply to those therapists. However, psychological studies suggest that decision-making processes are universal. If that’s the case, your findings could apply to other therapists.” “Ahhh!”, she exclaimed. Suddenly she sounded very interested. “A subsequent, quantitative study could offer you hard data to support this”, I said.
Trying to digest our conversation, she wanted to know more about qualitative methodologies. “Naturalistic inquiry is an important principle of qualitative research”, I explained. “This means that research takes place within real world settings. You ask open questions to gain insight into people’s experiences.”
“Fair enough”, she replied when I recommended reading several seminal papers and chapters from handbooks. Then her anxiety came again to the fore. “What about recruitment?”, she asked, “how many participants would I need?”
There’re so many strategies a researcher may use. However, there’s beauty in simplicity. “Convenience or purposeful sampling strategies are suitable. You could visit psychotherapy websites created by professional bodies to identify participants. Typically, you’d need 30 or less participants when using a qualitative approach.” “Poooh, I’ve to recruit so many participants???” She had hoped that 3-5 psychotherapists would be sufficient. “You don’t want to make the mistake and recruit too few participants”, I said. “And journals wouldn’t publish your findings.” She nodded: “Lot’s to think about.”
Indeed. Hopefully, you’re aware of common research mistakes. Wishing you a ‘researchful’ time.
Useful literature
K. Denzin, & Y. S. Lincoln (2001) (Eds.), Handbook of qualitative research, 2nd edition. Thousand Oaks, CA: Sage Publications.
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.
Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.
Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences. Thousand Oaks, CA: Sage.