Quantitative Unease

A column dedicated to demystifying psychotherapy research – love it, hate it, or both…at least try to know what it’s all about!

Experiments And Experimental Designs

It’s unfortunate that experiments have such a bad reputation. They’ve a distinct advantage. Experiments are the only way of showing that there’s a direct causal relationship between a particular treatment and measured behaviour. How do we demonstrate this? By manipulating the independent variable, the variable we suspect to be the causal one. If it does produce changes in the dependent variable, the one we measure, a causal inference can be made.

The condition in which the suspected causal variable is present, is called experimental, and the one in which it’s absent, control condition. We compare experimental with control condition to see if there’s a difference between scores. In order to make causal inferences, we’ve to be sure that the independent variable really is the only thing that we’re changing. Extraneous variables, rival explanations, must be eliminated. So those possibilities must be kept constant. Uncontrolled variables that change along with the independent variable are called confounders. A good experiment should not have confounding variables. Choosing enough subjects is also important. Unrelated to subjectivity, so-called subjects are participants in experimental studies. These are the basics of experimental design. Let’s have a look at two common ones, the between and within subjects design.

In the between subjects design, different subjects are chosen, who are assigned to conditions. One or more independent variables are used. An experiment with one independent variable, and an experimental and control group, is the simplest design.

How does this work in practice? If you’d like to test the impact of silence on anxiety, you randomly allocate the same number of subjects into two groups.  Anxiety is your dependent variable. It depends on, is affected, by the independent, treatment variable (silence). You ask everybody to fill in the Beck Anxiety Inventory to measure their anxiety. In your control group, you conduct your psychotherapy session as you always do. The experimental group is exposed to long silences.

After the sessions, you measure anxiety again and compare scores between the two groups. Looking at the mean (average) score, you find that scores have increased under the condition of long periods of silence. You conclude that silence caused higher anxiety. For this to be true, increased anxiety must be due to silence and nothing else. However, your results may still be open to interpretation. Comparing the mean, you discover the odd outlier, one score is very high. Generalisation might be problematic. Your experiment needs to be replicated and run again to be certain that silence causes anxiety.

You can also match subjects. You assess subjects’ anxiety before allocating them to experimental conditions. Then you allocate an equal number of people with low anxiety to each group to guarantee an equal spread. Whether a person with low anxiety goes into the normal or silent psychotherapy sessions is decided through randomization.

There’re other aspects to consider. Random error can’t be completely removed. For example, anxiety may be influenced by group dynamics. It’s impossible to control for everything. But if you run enough subjects, those effects should cancel each other out. Systematic errors also bias results. Such errors happen when testing one group in the evening and the other in the morning. Anxiety may be higher at a particular time of the day. There might also be an individual difference between people’s anxiety. Some may feel more anxious during silence than others. To minimise individual differences (extraneous variation), you can use a within subjects design. When designing experiments, use this first. Only turn to alternatives when you discover unsurmountable obstacles.

For the within subjects design you need fewer people. The same subjects are tested on at least two different occasions. Order effects, leading to improvement or deterioration in performance, is a disadvantage of this design. Running the silent session first may affect anxiety levels in the other session. Subjects could still be anxious and their anxiety therefore be higher in the normally conducted psychotherapy session. A confounder arises from the order in which you run your groups. Order effects can’t be eradicated. But randomly allocating subjects to your groups can counterbalance them, because a random sequence should spread order effects more or less equally around the normal and silence group. It works best when you’ve more than two groups, although randomization and counterbalancing only transform variation into unsystematic variation. They don’t eliminate variation. When studying the impact of 15-minute with 40-minute silence, you’ve two experimental conditions. You can also compare three or more different conditions and use one as control group. This may reduce variation.

If you’ve become interested in quantitative research and are tempted to design an experiment, think of all the intriguing techniques that could be examined to demonstrate a causal link between technique and behaviour change.

Wishing you a good autumn.

Susanne Vosmer
s.vosmer@gmail.com