One limitation of our work, however, was that we were unable to access the original stimuli and so we relied on our own reconstructions. These reconstructions (there were two) were different in a number of respects that may have decreased our chances of seeing the same effect KTE observed. So we were pleased to learn that Agnes Kovács and her co-authors had posted a subset of her original stimuli (see the bottom of this page). While they didn't post the full video set, we were nevertheless able to modify the videos to create the necessary conditions and rerun our experiment.
As a reminder, the task was detecting a ball behind a box. The consistent pattern we found in our experiments was a crossover interaction between the agent's belief that a ball was present and the participant's belief. This crossover didn't obviously follow from any kind of belief-tracking account, since why would detection of the ball be slow when the agent AND the participant believe it is present. Here are KTE's original data and the theoretically-predicted belief tracking effect:
And here are the data from the same experiment, run using KTE's stimuli (all methods identical to those used in Experiments 1a and b, above):
So this pattern of RTs looks quite similar and strongly supports the presence of a crossover interaction. These findings provide further support for our account of the original KTE results, and further argue against this paradigm measuring beliefs.
(Thanks to my coauthors Desmond Ong, Jonathan Philips, and Rebecca Saxe. Desmond did the experiment, and all three reviewed this post).