Preregister everything

Which methodological reforms will be most useful for increasing reproducibility and replicability? I've gone back and forth on this blog about a number of possible reforms to our methodological practices, and I've been particularly ambivalent in the past about preregistration, the process of registering methodological and analytic decisions prior to data collection. In a post from about three years ago, I worried that preregistration was too time-consuming for small-scale studies, even if it was appropriate for large-scale studies. And last year, I worried whether preregistration validates the practice of running (and publishing) one-offs, rather than running cumulative study sets. I think these worries were overblown, and resulted from my lack of understanding of the process.

Instead, I want to argue here that we should be preregistering every experiment do. The cost is extremely low and the benefits – both to the research process and to the credibility of our results – are substantial. Starting in the past few months, my lab has begun to preregister every study we run. You should too.

The key insights for me were:

1. Different preregistrations can have different levels of detail. For some studies, you write down "we're going to run 24 participants in each condition, and exclude them if they don't finish." For others you specify the full analytic model and the plots you want to make. But there is no study for which you know nothing ahead of time. 
2. You can save a ton of time by having default analytic practices that don't need to be registered every time. For us these live on our lab wiki (which is private but I've put a copy here).  
3. It helps me get confirmation on what's ready to run. If it's registered, then I know that we're ready to collect data. I especially like the interface on AsPredicted, that asks coauthors to sign off prior to the registration going through. (This also incidentally makes some authorship assumptions explicit). 

Minimal nativism

(After blogging a little less in the last few months, I'm trying out a new idea: I'm going to write a series of short posts about theoretical ideas I've been thinking about.)

Is human knowledge built using a set of of perceptual primitives combined by the statistical structure of the environment, or does it instead rest on a foundation of pre-existing, universal concepts? The question of innateness is likely the oldest and most controversial in developmental psychology (think Plato vs. Aristotle, Locke vs. Descartes). In modern developmental work, this question so bifurcates the research literature that it can often feel like scientists are playing for different "teams," with incommensurable assumptions, goals, and even methods. But these divisions have a profoundly negative effect on our science. Throughout my research career, I've bounced back and forth between research groups and even institutions that are often seen as playing on different teams from one another (even if the principals involved personally hold much more nuanced positions). Yet it seems obvious that neither has sole claim to the truth. What does a middle position look like?

One possibility is a minimal nativist position. This term is developed in Noah Goodman and Tomer Ullman's work, showing up first in a very nice paper called Learning a Theory of Causality.* In that paper, they write:

... this [work] suggests a novel take on nativism—a minimal nativism—in which strong but domain-general inference and representational resources are aided by weaker, domain-specific perceptual input analyzers.

This statement comes in the context of the authors proposal that infants' theory of causal reasoning – often considered a primary innate building block of cognition – could in principle be constructed by a probabilistic learner. But that learner would still need some starting point; in particular, here the authors' learner had access to 1) a logical language of thought and 2) some basic information about causal interventions, perhaps from the infant's innate knowledge about contact causality or the actions of social agents (these are the "input analyzers" in the quote above).