Friday, October 23, 2020

Against reference limits

Many academic conferences and journals have limits on the number of references you can cite. I want to argue here that these limits make no sense and should be universally abolished

To be honest, I kind of feel like I should be able to end this post here, since the idea seems so eminently sensible to me. But here's the positive case: If you are doing academic research of any type, you are not starting from scratch. It's critical to acknowledge antecedents and background so that readers can check assumptions. Some research has less antecedent work in its area, other research has more, and so a single limit for all articles doesn't make sense. More references allow readers to understand better where an article falls in the broader literature.

Some objections and responses. 

Aren't there space limitations? No, there aren't. Some journals still operate based on a set "page budget" that the publisher puts in place. This is silly as absolutely no one reads paper journals any more. If this weren't already clear before the pandemic, it's clear now. No one has sent an issue of Cognition or Psych Science to my house but life goes on.  

In my high profile, glossy journal, you should only cite important references and not try to be complete. When you remove references from an article, typically you cut the three papers you might have cited to just one. That one is probably the original positive claim; it's more likely to be from a famous old guy and it's less likely to be a newer finding, a meta-analysis, or a reference that provides additional context. This lack of context feeds the "rich get richer" cycle of citations and it hurts readers who should see multiple sources of evidence on an issue.

My review journal is aimed at students and we don't want to overwhelm them with references. I guess the argument is this: If you have 70 references and you cut them to 30 as a function of the journal limits, then students know what citation to look at. To me this seems crazy. First of all, no student is going to track down all 30 references; they are inevitably looking at a subset, probably the references for one claim. And for that one claim, they deserve the same context as a researcher does – don't just send them to the original paper without also giving them the critique, the meta-analysis, or the newer non-replication. If you want to curate, then have the bibliography be annotated (as, for example, Nature Reviews Neuroscience does). Let the author call out the important references, rather than removing dissent and diversity from the bibliography.

It's only a conference paper/abstract, you don't need references and they count against the space limitations anyway. Most computer science conferences now do not count references against page limits, and increasingly abstracts for developmental psych conferences do not either. Fundamentally, you are probably looking at conference papers or abstracts on the web – so the documents you look at should be able to have hyperlinks in them (and that's all references are, anyway, is hyperlinks to other papers). Let authors add a reference section! And while we're at it, we should have a technical solution (e.g., a regular expression) to count words outside of citations. Why dock people words for appropriate scholarly procedure?

Unlimited references encourage (self-)citation packing. It's true that if citations were unlimited, in principle you could pack the reference section with tons of irrelevant citations or, maybe more realistically, with self-citations. But first of all, most journals already have unlimited citations and no one does this (well, almost no one). Second, citation packing is something that can be dealt with by editors and reviewers. Finally, if someone is hell-bent on self-citation and you have a reference limit, they will use all of their references to cite themselves anyway. But if you give them unlimited references they might actually cite the relevant work in addition to their own. Self-citation is a real issue, but limiting references is the wrong policy tool to deal with this problem.

OK, I hope I've convinced you. Let everyone cite to their heart's content. Don't limit references, and don't count them towards page and word limits in submissions. 

No comments:

Post a Comment