Here's a common puzzle that comes up all the time in discussions of replication in psychology. I call it the stimulus adaptation puzzle. Someone is doing an experiment with a population and they use a stimulus that they created to induce a psychological state of interest in that particular population. You would like to do a direct replication of their study, but you don't have access to that population. You have two options: 1) use the original stimulus with your population, or 2) create a new stimulus designed to induce the same psychological state in your population.
One example of this pattern comes from RPP, the study of 100 independent replications of psychology studies from 2008. Nosek and E. Gilbert blogged about one particular replication, in which the original study was run with Israelis and used as part of its cover story a description of a leave from a job, with one reason for the leave being military service. The replicators were faced with the choice of using the military service cover story in the US where their participants (UVA undergrads) mostly wouldn't have the same experience, or modifying to create a more population-suitable cover story. Their replication failed. D. Gilbert et al. then responded that the UVA modification, a leave due to a honeymoon, was probably responsible for the difference in findings. Leaving aside the other questions raised by the critique (which we responded to), let's think about the general stimulus adaptation issue.
If you use the original stimulus with a new population, it may be inappropriate or incongruous. So a failure to elicit the same effect is explicable that way. On the other hand, if you use a new stimulus, perhaps it is unmatched in some way and fails to elicit the intended state as well. In other words, in terms of cultural adaptation of stimuli for replication, you're damned if you do and damned if you don't. How do we address this issue?
Manipulating psychological states across cultures, languages, and contexts is a tricky business. There are two key principles not to forget here. First, identifying the linking function between a particular state or construct that you want to target and the stimulus. Having a better idea of your stimulus generation mechanism – whether it's sampling words from a list, writing stories according to some principle, or any other formal statement of procedure – can allow you to generate a range of stimuli more easily and allow others to adapt your work to other contexts more easily. Second, identifying the predictions of the manipulation – what difference in state do you want to elicit – can allow you to verify that prediction. These principles lead to a couple of concrete suggestions.
1. Use manipulation checks. If the original stimulus was designed to elicit a particular psychological state, the manipulation check provides the key confirmation of that state. If you have a stimulus that is designed to make participants feel disgusted, lonely, or confused, confirm that it did that! If the original paper used a manipulation check with their stimulus, then a replication of the manipulation check result (in terms of magnitude of effect) should be a sufficient solution to ensure that the new stimulus elicited the state of interest. If there was no manipulation check in the original, though, it can still be useful to include a manipulation check in the replication. (Ideally, the check should be given after the dependent variable is measured, to avoid demand characteristics.) Even if there wasn't a manipulation check in the original, a successful check in the replication still provides some evidence that you elicited the state you wanted to elicit.
2. Run both versions. This strategy is a stopgap, but it can be helpful in countering the objection that you made the wrong decision in adapting a stimulus. With enough statistical power, most patterns of results in this experimental design (stimulus X manipulation) are helpful. If both stimuli produce the result, that's great. If one or the other does and you see an interaction, then you know that the adaptation choice was critical. Only in the case of the failure of both stimuli are you still vulnerable to the joint critique that A) the original stimulus didn't fit your context, and B) you didn't adapt the stimulus correctly. But at least it looks like you tried. Still, if you're in this situation, I think the best response is probably the last.
3. Diversify your stimuli. The issue of stimulus adaptation comes up most frequently when there is only one stimulus item in an experiment. As many people have written – in domains from psycholinguistics (Clark, 1973) to social psychology (Judd, Westfall, & Kenney, 2012) – one stimulus item simply does not license generalization to a broader class. In order to license such a generalization, a sample of stimuli is required, just as a sample of participants is required to license a generalization to a population.
Thus, probably the strongest response to the stimulus adaptation issue is 1) to create a set of stimuli that are argued to elicit the state of interest, and 2) to then use a manipulation check to confirm that these stimuli create a distribution on that state with a mean that is different from some control stimuli. Of course, doing this can be hard in between-subjects research. But the alternative – in both original studies and replications – is unpalatable for precisely the reasons described above.
No comments:
Post a Comment