My lab's AI policies and norms

I – like every other academic in the world perhaps – am trying to come to terms with the effects of AI on research, teaching, and everything else. This post is about AI use in research. 

We recently had a lab meeting about how we are using AI in our research. People expressed lots of amazement about how much you can do with AI coding agents especially. I've personally been having a lot of fun using agents to help me pursue my research agenda (example, this preprint). But students also expressed a lot of disorientation, bewilderment, and skepticism. It's a challenging time to be a trainee scientist! There are many new opportunities for discovery, but the ground is shifting under our feet and it's hard to be sure exactly what's required, what's permissible, and what best practices are. 

My north star in all of this is the idea that science is not just for making more PDFs. Our scientific process should be organized around robust and transparent routes to discovery. Used appropriately, AI can lead to faster progress and stronger research. Used inappropriately, it can be a route towards generating papers that no one understands and slowing down communication. Here are a few principles we discussed in our meeting that I'm putting forward as current lab policies. 

Policies

No AI-authored text in papers. We will write all the words in our papers. This step maintains human understanding and direction in all of the research we do. It also goes some way to preventing inadvertent errors or fabrications by models, though other checks are critical. To be clear, AI-authored text is fine for some applications (e.g., software documentation or administrative tasks). But let's keep it out of scientific papers – at least, for the moment. 

Help me, but don't touch the paper!

As I’m discussing paper writing with more folks in the age of Claude Code and other coding assistants, we’re having more discussions about how to use these tools in writing. 

Per my earlier post, I don't want AI tools to "take away the good part" of my job, I want them to help with the bad parts. I'm finding it transformative that coding agents can help me iterate quickly through data munging tasks, navigate cluster scheduling challenges, and prototype experiments, just to name a few examples. They can also be very valuable for manuscript preparation, including finding typos, helping with formatting and reproducibility, and code review.

Agents are especially useful for Rmarkdown or Quarto-based reproducible writing. Writing reproducible papers is a great way to avoid typos and to quickly update your manuscript with new analyses. I teach this content and find it extremely helpful in my work. But getting a reproducible paper to compile can be very annoying, and I would like agents to help me (and my students and collaborators, who struggle with this step sometimes!). 

Here's my simple rule: agents should not directly touch the text of the paper. Paper text should be written by a human.* If agents write text, it's still far too easy for them to make things up (especially references) and to write text that is superficially correct but unmoored to the actual human research activities. 

So, how do you collaborate with an agent on a paper and guarantee it doesn't touch the text? Here is one workflow that's been working for me:

• The paper is written in a git repository. You commit all changes you make to the paper. 
• You ask the agent to do the work on a branch and make a pull request, following strict rules not to change text. 
• You review merge the PR and have clear visibility (and an audit trail) for any changes. That way, if the agent does touch the text, you know where and when that happened.

Here's a Claude Code skill file that instantiates this workflow.

---

*Norms may shift in the future but for now, that's where I'll draw the hard line. 

Datapages for reusable (and pretty!) data sharing

[This post is joint with Mika Braginsky, a long-time collaborator on data-sharing and data-viz.]

Data sharing is both a critical scientific need and, increasingly, a mandate by many research funders. The FAIR principles – that data should be findable, accessible, interoperable, and reusable – are a critical guide to how data are shared. Yet even FAIR-compliant datasets in approved repositories are often shared in ad-hoc formats that are hard to reuse or to integrate with other data. In contrast, the most impactful datasets tend to be disseminated thoughtfully through dataset-specific or community-specific platforms. These “domain-specific data repositories” (this was our term from a previous blogpost!) create opportunities for creating data standards and ontologies that fit the needs of a particular community, research problem, or instrument type. They also allow opportunities for engagement through interactive visualizations. But custom repositories and pretty websites with nice visualizations are costly and complicated to create.

We are introducing a set of open-source tools and templates for easily creating datapages, interactive websites to disseminate data for broad reuse. Datapages are easy to deploy for a single project, but extensible enough to host large collections of related datasets. You can learn more and get started at https://datapages.github.io/.

Using AI to improve (not automate away) academic research

Everyone seems to be consumed with AI anxiety. Graduate students are wondering if they will be replaced by assistants, or if they themselves are using AI enough or using it "right". Researchers are wondering what it means to produce research if agents can write whole papers. Everyone is wondering how we will keep up with a literature that is moving ever faster. 

Everyone is feeling the pressure to do *more*: do more projects, produce more papers, review more papers. This has already resulted in negative impacts on the research space, for example the problems that conferences have in getting quality, non-automated reviewing for the huge volume of submissions they receive. 

We should think about what we can do that is *different.* We should try to use automation to be more efficient at the annoying parts of our jobs while leaving more time for discovering new knowledge. The key (fast-evolving, unresolved) issue is how AI models will change the frontier of what is scientifically possible. This varies from field to field and changes day by day, but my sense is that the rise of semi-autonomous agents will be very interesting for scaling up social and behavioral science.

An LLM-backed "socratic tutor" to replace reading responses

My hot take on college-level teaching is that reading responses are mostly a terrible assignment, and they're even worse in the age of AI. I'm piloting something a bit different with my co-instructor right now: a "socratic tutor" bot that asks students to answer open-ended socratic questions about a specific text and "passes" them when they show sufficient comprehension. Initial feedback from students in a first trial has been extremely positive, so I am thinking more about how this could be useful in the future, as well as some of the potential problems. LLMs are far from a panacea for education – they cause way more problems than they solve, at the moment! – but this might be an interesting use case. 

As an instructor, one major challenge is that you want people to read the assigned reading and engage with it so that what you do in class can build on this content in a meaningful way; some students would prefer not to (or just don't have time, or whatever). How do you solve this problem? Weekly quizzes are possible but they're time-consuming to make and give and annoying to grade; plus they reinforce a memorization mindset, rather than inviting students to engage. 

The humble reading response is a frequent alternative: you ask students to respond to, critique, or build on their readings, usually in a short response ranging from a paragraph to a page. At their best in a well-prepared seminar, the instructor reads these beforehand, synthesizes them, and calls on individual students to share their reactions. But in a larger course, often this synthesis is impossible – and so the reading response becomes an assignment that no one wants to write and that can be tedious to read at the level they deserve. Even worse, if you're not getting called out on your reaction, it's possible to "respond" to a reading without having read it. And that's even before you can ask an AI to write a response to a text that it has ingested at some point (or that you've pasted into its chat window). What do we do? 

Book review: Elusive Cures

 I'm normally an avid fiction reader, but this summer I've been on a non-fiction kick. I just finished listening to Nicole Rust's new book, Elusive Cures. The premise of the book is the simple, important question: why haven't we made more progress on understanding brain disorders using basic neuroscience? Rust's argument is that the kind of "domino chain" causal model that we use to understand many neural systems is simply mismatched to the nature of how complex systems work. Rust is a cognitive neuroscientist who is known for her work on vision and memory, but she does not lean on these areas in the book, instead broadly surveying the neuroscience of disorders including Alzheimer's, Parkinson's, and depression. 

Although I'm mostly a cognitive scientist these days, Rust's description of the forward causal model for neuroscience immediately felt familiar from from my grad school neuroscience training. These kinds of causal systems are the ones we've made the most progress on in cognition as well: we have pretty strong models of how visual object recognition, reading, and language processing unfold in time. In contrast, processes that unfold interactively over time, such as mood, are much harder to understand this way. 

I have often been skeptical of the application of complex dynamical systems theory to cognition, though Rick Dale's nice intro for the Open Encyclopedia of Cognitive Science did win me over somewhat. I agree that cognition is a complex dynamical system, but in practice such formalisms can often feel unconstrained. Many researchers using dynamical systems theories don't – for whatever reason – engage in the kind of systematic model comparison and evaluation that I believe is critical for cognitive modeling. 

I heard that same kind of skepticism in Rust's own writing, which made it even more compelling when she made the case for the critical importance of understanding the brain as a complex dynamical system. Her discussion of the role of homeostasis in brain systems in particular was inspiring. It made me wonder why we don't apply the concept of homeostasis more to reason about social systems as well – for example, how communities maintain their educational standards in the face of policy changes or interventions. It's always a pleasure when a book sparks this kind of reflection. 

In sum, I strongly recommend Elusive Cures. I found it thought-provoking and broad, with good descriptions of both individual research findings and sweeping trends. The book feels like the rare "popular" book that also effectively makes a forceful scientific argument. 

Two summer book recommendations

After a long stint primarily reading fiction, I've been on a non-fiction kick recently and just read two books that I would definitely recommend!

Persuasion in Parallel (2022) by Alexander Coppock, a political scientist at Yale, is a scholarly monograph on how political persuasion works. It's a delightful combination of large-scale replications, strong emphasis on effect estimation and causal inference, and really thoughtful discussion of mechanisms. It starts from a replication and re-analysis of Lord, Ross, and Lepper (1979), the seminal work on political persuasion, and goes on to replicate a whole host of more recent studies. Across all of them, the key take-home is that arguments about controversial topics (e.g., gun control, abortion, etc.) operate very similarly across people with very different views: they cause small changes in attitude in the direction of the arguments, regardless of the recipient's initial views. 

In statistical terms, there's very limited heterogeneity across groups in the effect of persuasive arguments. I really appreciated evidence on this heterogeneity question because often students' intuition in psychology is that everything differs based on sociodemographic characteristics; yet this intuition is rarely quantified or challenged. Coppock's analyses take a really important step in this direction. 

The book is short and quite readable (especially given how data-rich it is). It's also very up front about the limitations of the work. There's also a thought-provoking final chapter on Bayesian inference and rational models of belief change that makes a number of connections to computational cognitive science that I enjoyed. Despite being an academic, I am not the sort of person who will sit down on a weekend with a monograph from another discipline for fun; this book was an exception for me because of how interesting, important, and thorough the work is.

On a heavier note, Doctored (2025), by Charles Piller (an investigative reporter with Science), is a screed about scientific misconduct in Alzheimer's research. I'm intimately familiar with replication issues in psychology, but I was still totally horrified to read about the impacts of scientific fraud in the Alzheimer's field. Piller makes a very well-researched and thorough case, working with experts on fraud and scientific reviewers. While a critique of the book by an Alzheimer's authority questions how central the fraudulent work was to the field (Lancet review), I was convinced by the later chapters of Doctored that show how pervasive image falsification has been within the Alzheimer's research enterprise. It's just awful to think that many people have been in dangerous clinical trials due to research misconduct. 

The book was clearly written very fast as there is some redundancy between chapters and a bit of unnecessary stage-setting around various researchers' grandparents (perhaps reflecting a pivot from an earlier vision of the book where certain people were more important to the narrative). But the substance of the scientific critique is so compelling – and honestly terrifying – that I was more than happy to overlook a few minor weaknesses in the prose. Definitely recommend.