Is mind-reading automatic? A replication and investigation

tl;dr: We have a new paper casting doubt on previous work on automatic theory of mind. Musings on open science and replication.

Do we automatically represent other people's perspective and beliefs, just by seeing them? Or is mind-reading effortful and slow? An influential paper by Kovács, Téglás, and Endress (2010; I'll call them KTE) argued for the automaticity of mind-reading based on an ingenious and complex paradigm.  The participant watched an event – a ball moving back and forth, sometimes going behind a screen – at the same time as another agent (a Smurf) watched part of the same event but sometimes missed critical details. So, for example, the Smurf might leave and not see the ball go behind the screen.

When participants were tested on whether the ball was really behind the screen, they appeared to be faster when their beliefs lined up with the Smurf's. This experiment – and a followup with infants – gave apparently strong evidence for automaticity. Even though the Smurf was supposed to be completely "task-irrelevant" (see below), participants apparently couldn’t help "seeing the world through the Smurf’s eyes." They were slower to detect the ball, even when they themselves expected the ball, if the Smurf didn’t expect it to be there. (If this short description doesn't make everything clear, then take a look at our paper or KTE's original writeup. I found the paradigm quite hard to follow the first time I saw it.)

I was surprised and intrigued when I first read KTE's paper. I don't study theory of mind, but a lot of my research on language understanding intersects with this domain and I follow it closely. So a couple of years later, I added this finding to the list of projects to replicate for my graduate methods class (my class is based on the idea – originally from Rebecca Saxe – that students learning experimental methods should reproduce a published finding). Desmond Ong, a grad student in my department, chose the project. I found out later that Rebecca had also added this paper to her project list.

One major obstacle to the project, though, was that KTE had repeatedly declined to share their materials – in direct conflict with the Science editorial policy, which requires this kind of sharing. I knew that Jonathan Philips (Yale) and Andrew Surtees (Birmingham) had worked together to create an alternative stimulus set, so Desmond got connected with them and they generously shared their videos. Rebecca's group created their own Smurf videos from scratch. (Later in the project, we contacted KTE again and even asked the Science editors to intervene. After the intervention, KTE promised to get us the materials but never did. As a result, we still haven't run our experiments with their precise stimuli, something that is very disappointing from the perspective of really making sure we understand their findings, though I would stress that because of the congruence between the two new stimulus sets in our paper, we think the conclusions are likely to be robust across low-level variations.)

After we got Jonathan's stimulus set, Desmond created a MTurk version of the KTE experiment and collected data in a high-power replication, which reproduced all of their key statistical tests. We were very pleased, and got together with Jonathan to plan followup experiments. Our hope was to use this very cool paradigm to test all kinds of subtleties about belief representation, like how detailed the participants' encoding was and whether it respected perceptual access. But then we started taking a closer look at the data we had collected and noticed that the pattern of findings didn't quite make sense. Here is that graph – we scraped the values from KTE's figures and replotted their SEM as 95% CIs:

The obvious thing is the difference in intercept between the two studies, but we actually don't have a lot to say about that – RT calculations depend on when you start the clock, and we don't know when KTE started the clock in their study. We also did our study on the internet, and though you can get reliable RTs online, they may be a bit slower for all kinds of unimportant reasons.

We also saw some worrisome qualitative differences between the two datasets, however. KTE's data look like people are slower when the Smurf thinks the ball is absent AND when they themselves think the ball is absent too. In contrast, we see a crossover interaction – people are slow when they and the Smurf think the thing is absent, but they are also slow when they and the Smurf think the thing is present. That makes no sense on KTE's account – that should be the fastest condition. What's more, we can't be certain that KTE wouldn't have seen that result, because their overall effects were so much smaller and their relative precision given those small effects seemed lower.

I won't go through all the ways that we tried to make this crossover interaction go away. Suffice it to say, it was a highly replicable finding, across labs and across all kinds of conditions that shouldn't have produced it. Here's Figure 3 from the paper:

Somewhere during this process, we joined forces with Rebecca, and found that they saw the crossover as well (panels marked "1c: Lab" and "2b: Lab 2AFC"). So Desmond and Jonathan then led the effort to figure out the source of the crossover.

Here's what they found. The KTE paradigm includes an "attention check" in which participants have to respond that they saw the Smurf leave the room. But the timing of this attention check is not the same across belief conditions – in other words, it's confounded with condition. And the timing of the attention check actually looks a lot like the crossover we observed: In the two conditions where we observed the slowest RTs, the attention check is closer in time to the actual decision that participants have to make.

There's an old literature showing that making two reaction time decisions right after one another makes the second one slower. We think this is exactly what's going on in KTE's paper, and we believe our experiments demonstrate it pretty clearly. When we don't have an attention check, we don't see the crossover; when we do have the check, but it doesn't have a person in it at all (just a lightbulb), we still see the crossover; and when we control the timing of the check, we eliminate the crossover again.

Across studies, we were able to produce a double dissociation between belief condition and the attention check. To my mind, this dissociation provides strong evidence that the attention check – and not the agent's belief – is responsible for the findings that KTE observed. In fact, KTE and collaborators actually also see a flat RT pattern in a recent article that didn't use the attention check (check their SI for the behavioral results)! So their results are congruent with our own – this also partially mitigates our concern about the stimulus issue. In sum, we don't think the KTE paradigm provides any evidence on the presence of automatic theory of mind.

Thoughts on this process. 

1. Several people who are critical of the replication movement more generally (e.g. Jason Mitchell, Tim Wilson) have suggested that we pursue "positive replications," where we identify moderator variables that control the effects of interest. That's what we did here. We "debugged" the experiment – figured out exactly what went wrong and led to the observed result. Of course, the attention check wasn't a theoretically-interesting moderator, but I think we did exactly what Mitchell and Wilson are talking about.

But I don't think "positive replication" is a sustainable strategy more generally. KTE's original N=24 experiment took us 11 experiments and almost 900 participants to "replicate positively," though we knew much sooner that it wouldn't be the paradigm we'd use for future investigations (what we might have learned from the first direct replication, given that the RTs didn't conform to the theoretical predictions). 

The burden in science can't fall this hard on the replicators all the time. Our work here was even a success by my standards, in the sense that we eventually figured out what was going on! There are other experiments I've worked on, both replications and original studies, where I've never figured out what the problem was, even though I knew there was a problem. So we need to acknowledge that replication can establish – at least – that some findings are not robust enough to build on, or do not reflect the claimed process, without ignoring the data until the replicators figure out exactly what is going on.

2. The replication movement has focused too much for my taste on binary effect size estimation or hypothesis testing, rather than model- or theory-based analysis. There's been lots of talk about replication as a project of figuring out if the original statistical test is significant, or if the effect size is comparable. That's not what this project was about – I should stress that KTE's original paradigm did prove replicable. All of the original statistical tests were statistically significant on basically every replication we did. The problem was that the overall pattern of data still wasn't consistent with the proposed theory. And that's really what the science was about.

This sequence of experiments was actually a bit of a reductio ad absurdum with respect to null-hypothesis statistical testing more generally. Our paper includes 11 separate samples. Despite having planned to have >80% power for all of the tests we did, the sheer scope means that a good number of them would not come out statistically significant, just by chance. So we were in a bit of a quandary – we had predictions that "weren't satisfied" in individual experiments, but we'd strongly expect that to be the case just by chance! (The probability of 11 statistical tests, each with 80% power, all coming out significant is less than .1).

So rather than looking at whether all the p-values were independently below .05, we decided to aggregate the RT effect size on the key effect using meta-analysis. This analysis allowed us to see which account best predicted the RT differences across experiments and conditions. We aggregated the RT coefficients for the crossover interaction (panel A below) and the RT differences for the key contrast in the automatic theory of mind hypothesis (panel B below).  You can see the result here:

The attention check hypothesis clearly differentiates between conditions where we see a big crossover effect and conditions where we don't. In contrast, the automatic theory of mind hypothesis doesn't really differentiate the experiments, and the meta-analytic effect estimate goes in the wrong direction. So the combined evidence across our studies supports the attention check being the source of the RT effect.

Although this analysis isn't full Bayesian model comparison, it's a reasonable method for doing something similar in spirit – comparing the explanatory value of two different hypotheses. Overall, this experience has shifted me much more strongly to more model-driven analyses for large study sets, since individual statistical tests are guaranteed to fail in the limit – and that limit is much closer than I expected.

3. Where does this leave theory of mind, and the infant literature in particular? As we are at pains to say in our paper, we don't know. KTE's infant experiments didn't have the particular problem we describe here, and so they may still reflect automatic belief encoding. On the other hand, the experience of investigating this paradigm has made me much more sensitive to the issues that come up when you try to create complex psychological state manipulations while holding constant participants' low-level perceptual experience. It's hard! Cecilia Hayes has recently written several papers making this same kind of point (here and here). So I am uncertain about this question more generally.

That's a low note to end on, but this experience has taught me a tremendous amount about care in task design, the flaws of the general null-hypothesis statistical approach as we get to "slightly larger data" (not "big data" even!) in psychology, and replication more broadly. All of our data, materials, and code – as well as a link to our actual experiments, in case you want to try them out – are available on github. We'd welcome any feedback; I believe KTE are currently writing a response as well, and we look forward to seeing their views on the issue.

---
(Minor edit: I got Cecilia Heyes' name wrong, thanks to Dale Barr for the correction).

Response to Marcus & Davis (2013)

tl;dr: We wrote a response to a critique of our work. Some more musings about overfitting.

Last year, Gary Marcus and Ernie Davis (M&D) wrote a piece in Psychological Science that was quite critical of probabilistic models of higher-level cognition. They asked whether such models are selectively fit to particular tasks to the exclusion of counter-evidence ("task selection") and whether the models are hand-tweaked to fit those particular tasks ("model selection"). On the basis of these critiques, they questioned whether probabilistic models are a robust framework for describing human cognition.

It's never fun to have your work criticized, but sometimes there is a lot that can be learned from these discussions. For example, in a previous exchange with Ansgar Endress (critique here, my response here, his response here), I got a chance to think through my attitude towards the notion of rationality or optimality. Similarly, in Tom Griffiths' and colleagues response to another critique, they have some nice discussion of this issue.

In that spirit, a group of Bayesians whose work was mentioned in the critique have recently written a response letter that will be published in the same journal as M&D's critique (after M&D get a chance to reply). Our response is very short, but hopefully it captures our attitude towards probabilistic models as being a relevant and robust method – not the only one, but one that has shown a lot of recent promise – for describing higher-level cognition. Here I want to discuss one thing that got compressed in the response, though.

One of the pieces of work M&D critiqued was the model of pragmatic reasoning that Noah Goodman and I published a couple of years ago (I'll call that the FG2012 model). Our one-page paper reported only a single study with a very restricted stimulus set, but there is actually a substantial amount of recent work on this topic that suggests such models do a good job at describing human reasoning about language in context; I posted a bibliography of such work a little while ago.

M&D criticized a particular modeling decision that we took in FG212 – the use of a Luce choice rule to approximate human judgments. They pointed out that other choices (that could have been justified a priori) nevertheless would have fit the data much worse. Summarizing their critique, they wrote:

"Individual researchers are free to tinker, but the collective enterprise suffers if choices across domains and tasks are unprincipled and inconsistent. Models that have been fit only to one particular set of data have little value if their assumptions cannot be verified independently; in that case, the entire framework risks becoming an exercise in squeezing round pegs into square holes." (p. 2357)

I agree with this general sentiment, and have tried in much of my work to compare models from different traditions across multiple datasets and experiments using the same fitting procedures and evaluation standards (examples here, here, and here). But I don't think the accusation is fair in the case of FG2012. I'll go through a couple of specific interpretations of the critique that don't hold in our case, and then argue that in fact the "model selection" argument is really just a repetition of the (supposedly separate) "task selection" argument.

Problem: numerical overfitting -> Solution: cross-validation. A very specific critique that M&D could be leveling at us is that we are overfitting. The critique would then be that we tuned our model post-hoc to fit our particular dataset. In a standard case of overfitting, say for a classification problem, the remedy is to evaluate the model on held-out data that wasn't used for parameter tuning. If the model performs as well on the out-of-sample generalization, then it's not overfit. Our (extremely simple) model was clearly not overfit in this sense, however: It had no numerical parameters that were fit to the data.

Problem: post-hoc model tweaking -> Solution 1: pre-registration. Another name for overfitting – when it concerns the researcher's choice of analytic model – is p-hacking. This is closer to what M&D say: Maybe we changed details of the model after seeing the data,  in order to achieve a good fit. But that's not true in this case. As datacolada says, when someone accuses you of p-hacking, the right response is to say "I decided in advance." In this case, we did decide in advance – the Luce choice rule was used in our 2009 CogSci proceedings paper with a predecessor model and a large, independent dataset.*

Problem: post-hoc model tweaking -> Solution 2: direct replication. A second response to questions about post-hoc model choice is direct replication. Both we and at least one other group that I know of have done direct replications of this study – it was a very simple MTurk survey, so it is quite easy to rerun with essentially no modifications (if you are interested, the original materials are here**). The data look extremely similar.*** So again, our model really wasn't tweaked to the particulars of the dataset we collected on our task.

What is the critique, then? I suspect that M&D are annoyed about the fact that FG2012 proposed a model of pragmatic reasoning and tested it on only one particular task (which it fit well). We didn't show that our model generalized to other pragmatic reasoning tasks, or other social cognition tasks more broadly. So the real issue is about the specificity of the model for this experiment vs. the broader empirical coverage it offers.

In their response, M&D claim to offer two different critiques: "model selection" (that's the one we've been discussing) and "task selection" (the claim that Bayesian modelers choose to describe the subset of phenomena that their models fit, but omit other evidence in the discussion). In light of the discussion above, I don't see these as two different points at all. "Model selection," while implying all sorts of bad things like overfitting and p-hacking, in this case is actually a charge that we need to use our models to address more different tasks. And if the worst thing you can say about a model is, "it's so good on those data, you should apply it to more stuff," then you're in pretty good shape.

---
* Confession: I've actually used Luce choice in basically every single cognitive model I've ever worked on. At least every one that required linkage between a probability distribution and a N-alternative forced-choice task.

** Contact me if you want to do this and I can explain some idiosyncrasy in the notation we used.

*** I'm not sure how much I can share about the other project (it was done by a student at another institution whose current contact info I don't have) but the result was extremely close to ours. Certainly there were no differences that could possibly inspire us to reject our choice rule.

(Thanks to Noah Goodman, Gary Marcus, and Ernie Davis for reading and commenting on a draft of this post).

Semantic bleaching in early language

M is now 15 months, and her receptive vocabulary is quite large. She knows her vehicles, body parts, and a remarkable number of animals (giraffe, hyena, etc.).  Not coincidentally, we spend a large amount of our time together reading books about vehicles, body parts, and animals – at her initiation. Her productive language is also proceeding apace. As one friend astutely observed, she has a dozen words, nearly all of them "ba."*

I've noticed something very interesting over the last three months. Here's one example. When M first discovered the word "da," she used it for several days, with extreme enthusiasm, in a way that seemed like was identical to the word "dog." The form-function mapping was consistent and distinctive: it would be used for dogs and pictures of dogs, but nothing else. But then over the course of subsequent days, it felt like this word got "bleached" of meaning – it went from being "dog" to being "wow, cool!"

The same thing happened with her first extended experience with cats, at a visit to my parents' apartment. She started producing something that sounded like "tih" or "dih" – very clearly in response to the cats. But this vocalization then gradually became a noise of excitement that was seemingly applied to things that were completely different. Perhaps not coincidentally, our visit was over and we didn't have any cat-oriented books with us, so she couldn't use the word referentially. Now that we're back in the land of cat books, the word is back to having a "real" meaning.

This looks to me to be a lot like the phenomenon of "semantic bleaching," where words' meanings get gradually broadened to incorporate other meanings (like the loss of the "annus" – year – part of the meaning of anniversary). This kind of bleaching typically happens over a much longer timescale as part of grammaticalization, the process by which content words can become particles or affixes (e.g. as in the content verb "go" becoming a particle you can use to describe things in the future like "going to X"). But maybe it is happening very quickly due to the intense communicative pressure on M's extremely small lexicon?

The idea here would be that M only has a very small handful of words. If they don't fit the current situation, she adapts them. But mostly the communicative context she's adapting to is, "hey, look at that!" or "let me grab that!" So the process of bleaching words out from "dog" to something more like "cool!" could actually be a very adaptive strategy.

I've checked the research literature and haven't found anything like this described. Any readers know more?

---
* It's really more like ball ("ba"), balloon ("ba", but possibly just a ball), bus ("ba"), perhaps baby ("ba") and bottle ("ba") as well, dog ("da"),  yes ("dah"), cat/kittie ("dih"), truck ("duh"), daddy ("dada"), yum ("mum-mum"), more ("muh"), hi ("ha-e"), and bye ("ba-e"). Some of these are speculative, but I think this is a pretty good estimate.

Probabilistic pragmatics bibliography

Pragmatics is the study of human communication in context. A tremendous amount of experimental and theoretical work has been done on pragmatics since Grice's seminal statement of the cooperative principle. In recent years, a number of people have been working on a new set of formal models of pragmatics, using probabilistic methods and approaches from game theory to quantify human pragmatic reasoning. 

This post is an incomplete bibliography of some of the recent work following this approach. My goal in compiling this bibliography is primarily personal: I want to keep track of this growing literature and the different branches it's taken. I've primarily included research that is either formal/computational in nature, or based directly on formal models. Please let me know in the comments or by email if you have work that you would like added here.

Probabilistic Models and Experimental Tests
One flaw in this literature is that right now there's no one good paper to look at for an intro. The first paper on this list is (IMO) a good introduction, but it's only a page long, so if you want details you have to look elsewhere. 

• Frank, M. C., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. Science, 336, 998.
• Approximate inference: Vogel, A., Goméz Emilsson, A., Frank, M. C., Jurafsky, D., & Potts, C. (2014). Learning to reason pragmatically with cognitive limitations. Proc CogSci. 
• Levels of implicature: Degen, J. & Franke, M. (2012). Optimal Reasoning About Referential Expressions. Proc SeineDial.
• Deriving implicatures from autonomous agents. Vogel, A., Potts, C. & Jurafsky, D. (2013). Implicatures and nested beliefs in approximate Decentralized-POMDPs. Proc ACL.

Game Theoretic Approaches

This section is a very incomplete list of some of the great work on this topic in the game theory tradition. Note, Michael Franke is someone different from me. 

• Review of the game-theoretic approach: Franke, M. (2013). Game Theoretic Pragmatics.
  Philosophy Compass 8.3, pp. 269-284.
• Franke, M. and Jäger, G. (2012). Bidirectional Optimization from Reasoning and Learning in Games. Journal of Logic, Language and Information 21.1, pp. 117-139
• Franke, M. (2011). Quantity Implicatures, Exhaustive Interpretation, and Rational Conversation. Semantics & Pragmatics 4.1, pp. 1-82.
• Golland, D., Liang, P. & Klein, D. (2010). A game-theoretic approach to generating spatial descriptions. Empirical Methods in Natural Language Processing.

Extensions to Other Phenomena
Many of these models have been applied primarily to reference resolution but many other linguistic phenomena seem amenable to the probabilistic pragmatics approach.

• To non-literal language: Kao, J. T., Wu, J., Bergen, L., & Goodman, N. D. (2014). Nonliteral understanding of number words. Proceedings of the National Academy of Sciences.
• To syllogism: Tessler, M. H. & Goodman, N. D. (2014). Some arguments are probably valid: Syllogistic reasoning as communication. Proc CogSci.
• To scalar implicature (and interactions with knowledge/belief): Goodman, N. D. and Stuhlmueller, A. (2013). Knowledge and implicature: Modeling language understanding as social cognition. Topics in Cognitive Science.
• To cost-based (horn) implicatures - note, there's an in-prep version of a much longer version of this on the authors' websites: That's what she (could have) said: How alternative utterances affect language use. L. Bergen, N. D. Goodman, and R. Levy (2012). Proc CogSci.
• More cost-based implicature: Cost-Based Pragmatic Inference about Referential Expressions
  Judith Degen, Michael Franke and Gerhard Jäger (2013). Proc CogSci.
• To negation: Nordmeyer, A. E., & Frank, M. C. (2014). A pragmatic account of the processing of negative sentences. Proc CogSci.
• Interactions with grammar: Potts, Christopher. 2013. Conversational implicature: interacting with grammar. Ms., Stanford University. 

Connections to Language Acquisition

• Initial suggestion about connections with word learning: Frank, M. C., Goodman, N. D., Lai, P., & Tenenbaum, J. B. (2009). Informative communication in word production and word learning. Proc CogSci.
• Followup with evidence: Frank, M. C., & Goodman, N. D. (2014). Inferring word meanings by assuming that speakers are informative. Cognitive Psychology.
• Smith, N., Goodman, N. D., & Frank, M. C. (2013). Learning and using language via recursive pragmatic reasoning about other agents. Neural Information Processing Systems.
• Review article: Frank, M. C. (2014). Learning words through probabilistic inferences about speakers’ communicative intentions. In Language in Interaction. Studies in honor of Eve V. Clark. Arnon, I., Casillas, M., Kurumada, C. & Estigarribia, B., Eds. Amsterdam: John Benjamins. 

Connections with Pedagogy and Teaching
There are many interesting and as-yet-unexplored connections between pragmatics and teaching. 

• Shafto, P., Goodman, N., & Frank, M. C. (2012). Learning from others: The consequences of psychological reasoning for human learning. Perspectives in Psychological Science, 7, 341-351. 
• Followup with more data: Shafto, P. Goodman, N. D., & Griffiths, T. L. (2014). Rational reasoning in pedagogical contexts. Cognitive Psychology.
• Frank, M. C. (2014). Modeling the dynamics of classroom education using teaching games. Proc CogSciShafto, P. and Goodman, N. D. (2008). Teaching games: statistical sampling assumptions for learning in pedagogical situations. Proc CogSci.

Sharing research using RMarkdown

(An example of using R Markdown to do chunk-based analysis, from this tutorial.)

This last year has brought some very positive changes in the way my lab works with and shares data. As I've mentioned in previous posts (here and here), we have adopted the version control tool git and the site github for collaborating and sharing data both within the lab and outside it. I'm very pleased to report that nearly all of our publications for 2014 have code and data openly shared through github links.

In the course of using this ecosystem, however, I've come to think that it's still not perfect for collaboration. In particular, in order to view analysis results from a collaborator or student, I need to clone into the repository and run all of their analyses, regenerating their figures and working out what they were intending in their code. For simple projects, this isn't so bad. But for anything that requires a modicum of data analysis, it really doesn't work very well. For example, I shouldn't have to rerun all the data munging for an eye-tracking project on my local machine just to see the resulting graphs.

For that reason, we've started using R Markdown for writing our analyses and sharing them with collaborators. R Markdown is a method for writing chunks of code interspersed with simple formatted text. Plots, tables, etc. are inserted inline. This combo then can be rendered to HTML, PDF, or even Word formats. Here's a nice tutorial – the source of the sample image above. The basics are so simple, it should only take about 5 minutes to get started. And all of this can be done from within RStudio, which is a substantially better IDE than the basic Mac R interface.*

Using R Markdown, we write our analyses in a (relatively) comprehensible way, explaining and delineating sections as necessary. We then can compile these to HTML and share them using RPubs, a service that is currently integrated with the R Markdown functionality in RStudio. That way we can just send links to one another (and we can republish and update with new analyses as needed).

Overall, this workflow means that we have full version control over all of our analyses (via git), but also have a friendly way to share with time-crunched or less tech-savvy collaborators. And critically, the overhead to swap to this way of working has been almost nonexistent. Several of our students in the CSLI undergraduate summer internship program this summer completed projects where all their data analysis was done this way. No ecosystem is perfect, but this one is a nice balance between reproducibility and openness on the one hand and ease of use on the other.

----
* I can't help mentioning that it would be nice if the internal plotting window was a quartz window that could save vector PDFs. The quartz() workaround is very ugly when you are working in OS X full-screen mode.

** Right now, all RPubs documents are shared publicly, but that's not such a big deal if you're used to working in a primarily public ecosystem using github anyway.

More on the nature of first words

About two weeks ago, M – now 13 months old – started using "dada" to refer to me. She has been producing "da" and "dada" as part of her babble for quite a while, but this was touching and new. It's a wonderful moment when your daughter first calls to you using language, not just a wordless cry.

Of course, congruent with what happened with "brown bear," I haven't heard much "dada" in about a week. She still seems to understand it (and likely did before producing it), but the production really seems to come and go with these first words. Now she's big into balls and appears to produce the sequence "BA(l)" pretty consistently while pointing to them. (I'm writing "BA(l)" because there's a hint of a liquid at the end, in contrast to the punctate "ba" that she uses for dogs and birds that we see at the park).

I want to comment on something neat that happened, though. In the very first day of M's "dada" production, we saw two really interesting novel uses of the word, both supporting my previous discussion about the flexibility of early language.

The first use was during a game we often play with M where she unpacks and repacks all the cards in my wallet. A couple of years ago, I lost my credit cards several times, and the bank started putting my photo on my card. (I think they do this for folks who are at high risk for identity theft). During the wallet-unpacking game, M took one of the cards, pointed to the photo of me (a small, blurry, old photo at that), and said "dada."

Kids do understand and recognize photos and other depictions early in life. My favorite piece of evidence for children's picture understanding is a beautiful old study by Hochbert & Brooks (1962). They found that their own child, after being deprived of access to drawings and photos until the age of 19 months, nevertheless had very good recognition objects he knew from both kinds of images, the very first time he saw them.* M's generalization of "dada" to my photo thus might not be completely surprising, but it certainly supports the idea that the word was never dependent on me actually being there.

The second example, reported by my wife, is even more striking.  When I had stepped out of the house for a moment, M pointed to the bedroom door where I had been and said "dada" – as though she was searching for me. This kind of displacement – use of language to describe something that is absent  – is argued to be a critical design feature of language in a really nice, under-appreciated article by Hockett (1960). Some interesting experiments suggest that even toddlers can use language to learn about unseen events, but I don't know about systematic studies of the use of early words to express displaced meanings. M's use of "dada" to refer to my absence (or perhaps to question whether I was present but unseen) suggests that she already is able – in principle – to use language in this way.

More broadly, in watching these first steps into language I am stunned by the disconnect between comprehension and production. Production is difficult and laborious: M accomplishes something like "brown bear" or "dada" but then quickly forgets or loses interest in what she has learned.** But the core understanding of how language works seems much more mature than I ever would have imagined. For M, the places where she shows the most ability is in understanding language a signal of future action. When we say "diaper time" or "would you like something to eat?" she apparently takes these as signals to initiate the routine, and toddles over to the changing pad or dinner table. But when we're in the middle of the routine, saying "diaper" doesn't inspire her to point to her diaper.

Again and again I am left with the impression of a mind that quickly apprehends the basic framework assumptions of the physical and social world, even as carrying out the simplest actions using that knowledge remains extraordinarily difficult.

----

* Needless to say, this was an epic study to conduct. Drily, H&B write in their paper that “the constant vigilance and improvisation required of the parents proved to be a considerable chore from the start—further research of this kind should not be undertaken lightly.”

** On a behaviorist account of early language, M would never forget "dada" – I was so overjoyed that I probably offered more positive reinforcement than she could even appreciate.

(minor updates and typo fixes 7/29)

Exploring first words across children

(This post is joint with Rose Schneider, lab manager in the Language and Cognition Lab.)

For toddlers, the ability to communicate using words is an incredibly important part of learning to make their way in the world. A friend's mother tells a story that probably resonates with a lot of parents. After getting more and more frustrated trying to figure out why her son was insistently pointing at the pantry, she almost cried when he looked straight at her and said, “cookie!” She was so grateful for the clear communication that she gave him as many cookies as he wanted.

We're interested in early word learning as a way to look into the emergence of language more broadly. What does it take to learn a word? And why is there so much variability in the emergence of children's language, given that nearly all kids end up with typical language skills later in childhood?

One way into these questions is to ask about the content of children's first words. Several studies have looked at early vocabulary (e.g. this nice one that compares across cultures), but – to our knowledge – there is not a lot of systematic data on children's absolute first word.* The first word is both a milestone for parents and caregivers and also an interesting window into the things that very young children want to (and are able to) talk about.

To take a look at this issue, we partnered with Children’s Discovery Museum of San Jose to do a retrospective survey of children's first word. We're very pleased that they were interested in supporting this kind of developmental research and were willing to send our survey out to their members! In the survey, we were especially interested in content words, rather than names for people, so for this study, we omitted "mama" and "dada" and their equivalents. (There are lots of reasons why parents might want these particular words to get produced – and to spot them in babble even when they aren't being used meaningfully).

We put together a very short online questionnaire and asked about the child's first word, the situation it occurred in, the age of the child, the age of the utterance, and the child's current age and gender. The survey generated around 500 responses, and we preprocessed the data by translating words into English (when we had a translation available) and categorizing the words by the MacArthur-Bates Communicative Development Inventory (CDI) classification, a common way to group children's vocabulary into basic categories. We did our data analysis in R using ggplot2, reshape2, and ddply.

Here's the graphic we produced for CDM:

We were struck by a couple of features of the data, and the goal of this post is to talk a bit more about these, as well as some of other things that didn't fit in the graphic.

First, the distribution of words seemed pretty reasonable, with short, common words for objects ("ball," "car"), animals ("dog," "duck" – presumably from bathtime), and social routines ("hi"). The gender difference between "ball" and "hi" was also striking, reflecting some gender stereotypes – and some data – about girls' greater social orientation in infancy. Of course, we can't say anything about the source of such differences from these data!

Another interesting feature of the data was the age distribution we observed. On parent report forms like the CDI, parents often report that their children understand many words even in infancy, with the 75th percentile being reported to know 50 words at 8 months. While there is some empirical evidence for word knowledge before the first birthday, this 50 word number has always been surprising, and no one really knows how much wishful thinking it includes. The production numbers for the CDI are much lower, but still have a median value above zero for 10-month-olds. So is this overestimation? Probably depends on your standards. M, Mike's daughter, had something "word-like" at 10 months, but is only now producing "hi" as a 12-month-old (typical girl).

One possible confound in this reporting would be parents misremembering the age at which their child first produced a word, perhaps reporting systematically younger or older ages (or even ages rounded more towards the first birthday) as the first word recedes into the past. We didn't find evidence of this, however. The distribution of reported age of first word was the same regardless of how old the child was at the time of reporting:

Now on to some substantive analyses that didn't make it into the graphic. Grouping our responses into broad categories is a good way to explore what classes of objects, actions, etc., were the referents of first words. While many of the words we observed in parents’ responses were on the CDI, we had to classify some others ad-hoc, and still others we were unable to classify (we ended up excluding about 50 for a total sample of 454, 42% female). Here's a graph of the proportions in each category:

So no individual animal name dominated, but overall they were most frequent, followed by "games and routines" (including social routines like "hi" and "bye") and toys. People were next, followed by animal sounds.

There are some interesting ways to break this down further. Note that girls generally are a few months ahead, language-wise, so analyses of age and gender are a bit confounded. Here's our age distribution broken down by gender:

As expected, we see girls a bit over-represented in the youngest age bin and boys a little bit over-represented in the oldest bin.

That said, here are the splits by age:

and gender:

Overall, younger kids are similar to older kids, but are producing more names for people. Older kids were producing slightly more vehicle names and sounds, but this may be because the older kids skew more male (see gender graph, where vehicles are almost exclusively the provenance of male babies). The only big gender trends were 1) a preference for toys and action words for the males and 2) a general broader spread across different categories. This second trend could be a function of boys' tendency to have more idiosyncratic interests (in childhood at least, perhaps beyond).

Overall, these data give us a new way to look at early vocabulary, not at the shape of semantic networks within a single child, but at the variability of first words across a large population. We invite you to look at the data if you are interested! 

---
Thanks very much to Jenni Martin at CDM for her support of our research!

* What does that even mean? Is a word a word if no one understands or recognizes it? That seems pretty philosophically deep, but hard to assess empirically. We'll go with the first word that someone else, usually a parent, recognized as being part of communicating (or trying to communicate).