Thursday, January 26, 2017

Paper submission checklist

It's getting to be CogSci submission time, and this year I am thinking more about trying to set uniform standards for submission. Following my previous post on onboarding, here's a pre-submission checklist that I'm encouraging folks in my lab to follow. Note that, as described in that post, all our papers are written in RStudio using R Markdown, so the paper should be a single document that compiles all analyses and figures into a single PDF. This process helps deal with much of the error-checking of results that used to be the bulk of my presubmission checking.

Paper writing*

  • Is the first paragraph engaging and clear to an outsider who doesn't know this subfield?
  • Are multiple alternative hypotheses stated clearly in the introduction and linked to supporting prior literature?
  • Does the paragraph before the first model/experiment clearly lay out the plan of the paper?
  • Does the abstract describe the main contribution of the paper in terms that are accessible to a broad audience?
  • Does the first paragraph of the general discussion clearly describe the contributions of the paper to someone who hasn't read the results in detail? 
  • Is there a statement of limitations on the work (even a few lines) in the general discussion?

Analytic code 

  • Are libraries sourced and constants defined at the top of the doc
  • Are code blocks short and legible?
  • If there is complicated code, have you walked a collaborator or friend through it to catch obvious weirdness?
  • If there is extensive data preprocessing, have you considered factoring to a script where you can check outputs and store an intermediate tidy data frame? 
  • For long/complex analyses, have you considered factoring helper functions to a separate script? 


  • Are all fonts > 9 pts (as an absolute minimum) when rendered into the document?
  • Is your aspect ratio for panels a bit wider than it is high? (maybe around the golden ratio, ~1.6:1 or 3:2).
  • Are your axes titled appropriately?
  • Is your caption clear and informative? Does it describe all aspects of the plot, including error bars?
  • Is the legend in a useful place? Have you considered direct labeling of the plots using directlabels::geom_dl() or putting the legend below the plot to save space?
  • Have you used a good color palette? consider langcog::scale_colour_solarized()
  • Are you using an unobtrusive theme like theme_bw()? have you considered using something minimalist like ggthemes::theme_few()?


  • Is there a github repository?
  • Does it have all the data and code, including writeup? 
  • Is the repo organized clearly, with "raw_data," "analysis," and "writeup" directories? (plus "helper" etc. as needed). 
  • Is there an R project, and is it packratted for package versioning? 
  • Can someone else re-render your paper from scratch (get a buddy to try this)? if that takes a while, can you cache intermediate data?
  • Is there a that briefly describes what the project is and how the repo is organized?

Authorship, formatting, and miscellaneous

  • Have all authors ok'd submission?
  • Have you checked on grant acknowledgements?
  • Are all section headings maximally informative?
  • Are results and methods described in the past tense?
  • Are there links to your preregistration, experiment code, and data/analysis repository?
  • Did you spell check, since R Studio doesn't have inline spellchecking?
  • Did you check for referent-less "this" (my pet prescriptive peeve, as in "This shows that ...")?

Good luck on your submissions!

*Note: These may seem a bit formulaic, but A) CogSci papers are very short (6 pages), and B) often a writing formula is a good starting point for creative departures.


    1. These are very useful. I suggest using child Rnw's to layer the chunks within the main document. That way the text is not interrupted by code chunks.

      Also, I don't understand why you have formatting comments. Those are handled by the latex style file?

    2. Thanks! We use Rmd not Rnw so I am not sure whether the child suggestion works the same way. FWIW, I find knitr+rmarkdown substantially easier than the old sweave format.

      Re: formatting comments, what are you thinking about? Nothing in there is really stuff that would be handled by the text style.

      1. Maybe I misread something about formatting.

        However, note that the Rnw format is also used with knitr (not only sweave, which I think is history now). I write my papers not in Rmd but in Rnw files, using knitr.

        I prefer this Rnw to Rmd mainly because I get better control over latex that way. I only use Rmd for vignettes, which I write when documenting the data analysis as a package.