Quantitative Unease

A column dedicated to demystifying psychotherapy research – love it, hate it, or both…at least try to know what it’s all about!

In Search of a Systematic Review

It was one of those cold and dull winter mornings in 2010. We came together to listen to and discuss the findings of a systematic review, which had been commissioned by the IGA and GAS to assess the evidence for the efficacy (effects in controlled conditions) and clinical effectiveness of group analysis and analytic group psychotherapy. The faces of many group analysts were as grim as the title of the research conference: Can group psychotherapy survive NICE? Examining the evidence. The atmosphere was tense. There were traces of trauma and despair in the air. Some analysts vehemently questioned whether group analysis had to become “quantitative”, others grasped that research knowledge and research evidence were important. It was survival analysis in action, both in a statistical and analytical sense.

Some had faith in group analysis and luckily, our profession has more than just survived NICE (National Institute of Clinical Excellence). Research has been integrated into group analytic practice and training. The sobering recommendations and discussions on that grey afternoon in London contributed to this development. Nevertheless, not everybody may search for systematic reviews when flicking though journals. Perhaps it has never crossed their mind as to why it may be beneficial to read a systematic review instead of individual outcome studies.

The beauty of systematic reviews lies in comprehensiveness and parsimony. Easier and quicker to conduct than randomised controlled trials, a review gives you an overview of several primary studies. When you want to evaluate outcome research in group analytic psychotherapy, a systematic review also gives you a more accurate and less biased view. And it’s economical, it saves you time. The findings from many articles are printed in one paper, which means that you can assess their quality by reading one article.

There’s another advantage. By conducting a systematic review, you can study group analysis empirically without destroying its essence. Knowing this might alleviate professional annihilation anxiety and group ‘incohesion’. Aggregation and massification are inhibited, preventing aggression or a flight response. Hence, you can introject curiosity and embrace a positive, positivist research attitude.

Tempted to carry out a systematic review? If so, you’ve to decide first what research question you intend to answer. Let’s say that you aim to determine the effects of long-term group analysis for depressed females. My creativity and intellect have been stifled, this isn’t an adequate question. A knowledgeable group analyst will undoubtedly notice this. The research question should be very specific, clinically relevant and novel. But the female depression question isn’t new, it’s therefore unsuitable. Unless you want to update an existing systematic review. Look at previous reviews and their recommendations to come up with clinically useful ideas instead of trying to resurrect dead topics. Does long-term group analytic psychotherapy improve low mood in female teenagers with a history of alcohol misuse more effectively than CBT is a better question.

Next you must think about inclusion criteria. Obviously, you include group analytic therapy studies that report on depression and low mood in females. That’s not all. Further defined eligibility criteria determine which studies you include or exclude. Do you include results from mixed groups, female and male adolescents? And how do you define long-term, 56 sessions, two years or more? Then there is sample size. How many participants must a study have? Which adolescent depression scales or mood inventories do you deem acceptable as outcome measure? Validity and reliability are important to consider when assessing measures. Reliability isn’t just a good character trait, mood rating scales should be reliable as well.

Do you only include studies, which were conducted after 2009, the ‘hey years’ (heyday) of group psychotherapy, or do you include articles that were published before this date? Once you’re clear about this, you make other decisions regarding inclusion criteria. Are qualitative studies acceptable? Traditionally, studies in systematic reviews must have a rigorous design and randomised controlled trials are considered the gold standard. Therefore, systematic reviews are almost at the top of the pyramid of evidence. Cohort studies, case report and case studies, a methodology inherited from psychoanalysis, are at the bottom of the hierarchy. But you can, of course, systematically review qualitative research studies or do a review of reviews. Blackmore et al.’s (2009) sobering but also encouraging systematic review includes these, partially due to the lack of high-quality trials.

Then you systematically and comprehensibly search electronic databases for eligible studies, otherwise your review wouldn’t be classed as a systematic review. PsycINFO, PubMed/Medline, EMBASE, CINAHL are good starting points. Also search PROSPERO, which is an international database of prospectively registered systematic reviews in health and social care.

You must use specific and sensitive search terms (e.g. depression, low mood, group analysis, group analytic therapy, adolescent females) when searching electronic databases. Don’t get disheartened if your search locates thousands of studies, there will be duplicates. Many studies are conducted in English. If you speak other languages, you can include Spanish, French, Norwegian, Danish or other peer-reviewed journal articles. Journals have different standards and impact factors. The higher the impact factor, the better. Why? Since the papers in those journals are cited frequently in other articles, the probability that group analysis will spread, if you publish your review in one of these journals, is high. This would combat the contagious despair and helplessness amongst group analysts, which I encountered at the conference more than a decade ago.

Prejudice is difficult to overcome, not just in research. To minimise bias, unpublished manuscripts should be included in your review. It can be difficult to find them, hence, it’s a good idea to search the Cochrane Database of Systematic Reviews and the Central Register of Controlled Trials. Researchers register their studies there. You may find a really interesting study, which has not been published. Editors are human and therefore biased as well. Sometimes they don’t publish even sound studies, either because these report negative findings (e.g. group therapy didn’t improve depression), or the results aren’t statistically significant. Your review will be ‘positively’ biased when unpublished, but noteworthy, findings are excluded. Also use Higgins et al.’s (2011) Cochrane Collaboration’s Tool for assessing risk of bias in randomised trials.

Once you have identified all studies that meet your inclusion criteria, sift through them by screening titles and reading relevant abstracts. Normally the number of studies you can exclude at this point is substantial and you’re left with a manageable amount of potentially relevant papers for full-text screening. Review the reference lists of those articles to identify potentially eligible papers that you might have missed. After having excluded all unsuitable studies, read and examine all suitable papers to extract the data for your review. You can devise a brief checklist, which contains all relevant criteria (e.g. source, characteristics of sample, study design, intervention characteristics, outcome/measures) to do this. Or you can evaluate the quality of randomised controlled trials by using quality rating proformas or tailored checklists found in published reviews and at http://www.prisma-statement.org. If information is missing in included papers, extract it from another article.

The entire procedure can be presented in a flowchart (e.g. number of records identified through database search, additional records identified through reference lists, results after removal of duplicates, records screened, full-texts assessed for eligibility, articles included in the review, records excluded). Your own systematic review is taking shape. Knowledge of statistics is as invaluable as clinical expertise.

When assessing the quality of primary studies, inspecting effect sizes is a good idea, because they tell you how effective group psychotherapy is for female adolescents compared to males. Often, but not always, Cohen’s effect size is used, with values ranging from 0.2 to 0.8. The larger the effect size, the better the statistical and clinical difference is between groups (females versus males). However, you also have to consider the number of participants in studies. Small trials tend to overestimate treatment effects, so be aware of that.

Finally, after you’ve assessed the quality of the included studies, you summarise, synthesise and interpret your data. It’s an appraisal not an analytic interpretation. Tabulated data and a narrative synthesis can be used to present the characteristics and details of your primary studies.

Be mindful that your review has to be reproducible, so be clear and provide detailed information. The value of your review doesn’t only depend on the quality and quantity of the included studies, but also rests on your methodology (clear search strategy, adequate inclusion/exclusion criteria etc.).

Write up your review of the retrieved papers, using the abstract, introduction, methods, results, discussion, conclusion, recommendations and references format. Be precise and acknowledge limitations and bias, both in the included studies and in your review. Leave the (statistical) meta-analysis for group supervision, or discuss it on a rainy day at another research conference. Until then, enjoy the success of a systematic review and celebrate its resurrection.

Seasonal Greetings.

References

Blackmore et al. (2009). A systematic review of the efficacy and clinical effectiveness of group analysis and analytic/dynamic group psychotherapy. Centre for Psychological Services Research, School of Health and Related Research, University of Sheffield, UK.

Blackmore et al. (2012) Report on a Systematic Review of the Efficacy and Clinical Effectiveness of Group Analysis and Analytic/Dynamic Group Psychotherapy. Group Analysis, vol. 45(1), 46-69.

Higgins et al. (2011) The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928 doi: 10.1136/bmj.d5928

Susanne Vosmer
s.vosmer@gmail.com