An update on automatic belief encoding

Last fall I wrote about our experiences investigating a very influential paper about automatic mind reading by Kovács, Téglás, & Endress (2010; KTE).  In brief, we replicated the intial findings, but we found that they were due to a confound in the experimental design: the timing of the manipulation check was confounded with condition, and appeared to be producing the observed reaction time effect.

One limitation of our work, however, was that we were unable to access the original stimuli and so we relied on our own reconstructions. These reconstructions (there were two) were different in a number of respects that may have decreased our chances of seeing the same effect KTE observed. So we were pleased to learn that Agnes Kovács and her co-authors had posted a subset of her original stimuli (see the bottom of this page). While they didn't post the full video set, we were nevertheless able to modify the videos to create the necessary conditions and rerun our experiment.

As a reminder, the task was detecting a ball behind a box. The consistent pattern we found in our experiments was a crossover interaction between the agent's belief that a ball was present and the participant's belief. This crossover didn't obviously follow from any kind of belief-tracking account, since why would detection of the ball be slow when the agent AND the participant believe it is present. Here are KTE's original data and the theoretically-predicted belief tracking effect:

And here are our original data from a bunch of separate experiments. As you can see, we get the crossover interaction in just about every case, not the belief-tracking pattern that KTE predicted:

And here are the data from the same experiment, run using KTE's stimuli (all methods identical to those used in Experiments 1a and b, above):

So this pattern of RTs looks quite similar and strongly supports the presence of a crossover interaction. These findings provide further support for our account of the original KTE results, and further argue against this paradigm measuring beliefs. 

(Thanks to my coauthors Desmond Ong, Jonathan Philips, and Rebecca Saxe. Desmond did the experiment, and all three reviewed this post).