tag:blogger.com,1999:blog-4297242917419089261.post6709983955874262632..comments2024-03-09T01:05:10.754-08:00Comments on Babies Learning Language: Post-publication peer review and social shamingMichael Frankhttp://www.blogger.com/profile/00681533046507717821noreply@blogger.comBlogger8125tag:blogger.com,1999:blog-4297242917419089261.post-74554186348134759352014-05-23T11:43:48.810-07:002014-05-23T11:43:48.810-07:00I totally agree. Open comments on a peer-reviewed...I totally agree. Open comments on a peer-reviewed paper are interesting & happen anyway, but are no surrogate for private pre-publication peer review.Joanna Brysonhttps://www.blogger.com/profile/02968914847649268737noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-47755393110185160082014-05-22T10:42:19.307-07:002014-05-22T10:42:19.307-07:00Hugh,
I agree that some PPPR focuses on correcti...Hugh, <br /><br />I agree that some PPPR focuses on correcting over-hyped science and that this is a valuable function. Two points on this:<br /><br />1. Your description above is a vision of PPPR in which commentary plays a fundamentally different role than standard peer review - it is for discussing interpretation and implications of work after publication (e.g. like "news and views") rather than for enforcing methodological standards. I'm fine with that and think it makes more sense in a lot of ways, especially because - as you note - you mostly want to do this for high-profile papers that make strong claims. It's not necessary to PPPR reasonable papers in specialist journals. <br /><br />2. The corrective to an overhyped paper (much like a poor methodological decision) isn't snark. I think we have to be extra careful to be moderate in our PPPR responses because even a slightly-snarky blogpost can inspire much more informal vitriol that leads to the (mis-)perception of "bullying." <br /><br />So I'm definitely in favor of the kind of PPPR that you describe - I just think we need to be very aware of the rationale and consequences if we want to foster a professional role for blogging and tweeting in providing proper and timely evaluation of new science.<br /><br />best, <br /><br />Mike<br /><br />PS: I think the discussion I had with Endress isn't quite PPPR - that's more like traditional academic back-and-forth. But I'm glad you agree we didn't oversell. ;)Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-73826146028622837052014-05-22T09:32:10.877-07:002014-05-22T09:32:10.877-07:00Hey Mike et al,
My impression is that most post p...Hey Mike et al,<br /><br />My impression is that most post publication peer review doesn't focus on science that is simply bad or unreplicable. Rather, it focuses on science that is perceived to be oversold or overhyped. Replicability might not respond to social incentives, but overselling might (as it's a response to a social incentive in the first place).<br /><br />If you publish a paper in a field journal saying "I have provided modest evidence that pushing your tongue into your cheek promotes flippant behavior" then you are unlikely to suffer the trauma of a Dutch psychologist tearing into your paper like a pitbull. That's because you've only made a modest claim (by virtue of your language and the setting for publication), and if that modest claim turns out to be incorrect then it only requires modest correction.<br /><br />By contrast, if you use the same weak data to make a strong claim in a flagship journal (and claims made in flagship journals should be strong), then that claim will be widely dispersed and believed. If the evidential foundations of that claim turn out to be weak, a strong corrective is needed (like PPPR).<br /><br />I think we all know that there's a big incentive in science to make overly strong claims about data: It gets you a paper in Nature, a segment on NPR, a big grant, and an academic job. That damages the career prospects of those who are more careful or who aren't willing to overclaim. PPPR might provide some balance, which would be good all round.<br /><br />Ta,<br /><br />Hugh<br /><br />ps, just to be clear, I don't think your rule learning work falls into the over-selling camp. In fact, I thought that that back-and-forth was a really useful discussion.Hugh Rabagliatihttps://www.blogger.com/profile/04074218181207217948noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-16134255420423001952014-05-22T03:33:20.611-07:002014-05-22T03:33:20.611-07:00Mike, good post, I agree. The solution to the repl...Mike, good post, I agree. The solution to the replicability "crisis" is not to shame people who publish results that aren't replicable. It could happen to the best of us, although we should all try to insure it doesn't, of course. If it's fraud that's one thing. But data can do it to you even if you're innocent of "QRPs" and/or intentional fraud. <br /><br />The solution isn't to stop replicating. But if the research community is going to get behind replication, it's going to have to be palatable to the research community. Perception does matter. Anonymoushttps://www.blogger.com/profile/07516834715925519704noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-56048935879120648782014-05-21T15:26:01.335-07:002014-05-21T15:26:01.335-07:00I agree, Kyle.
I have read the posts and comment...I agree, Kyle. <br /><br />I have read the posts and comments on your and Dan Casasanto's work on Language Log. I really like reading discussions about findings like yours (the QWERTY effect), but I found the overall tone of this particular discussion dismissive and distasteful. I stand by what I wrote here: post-publication reviewers need to take care in their critiques - and small issues of tone in posts can make commenters even more extreme in their statements.<br /><br />One thing that I didn't mention in my previous discussion is that open data and materials play a strong role in facilitating the clarity of these discussions. Some authors are very responsive regarding publicizing materials, but nothing will ever be as easy as clicking over to the repository and seeing what the authors did directly. In your discussion with the Language Log folks, I thought several points would have been much facilitated by an open repository of the precise code used for the analyses. <br /><br />MikeMichael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-41293195019138087572014-05-21T14:47:22.895-07:002014-05-21T14:47:22.895-07:00Bloggers have every right to criticize published w...Bloggers have every right to criticize published work, but I wish more of them would acknowledge the power they hold, and take more care by contacting authors for clarification or to ask for data, for example. There's more at stake than feelings. Careers can be damaged by negative blog posts, the inevitable "trial by Twitter", and the resulting permanent scar on the author's google search results. Let's not pretend that doesn't matter these days. Potential employers google applicants.<br /><br />Blogs started as a counter-culture movement, but now if they're not quite the new establishment yet they are at least heading that way. They have considerable reach and power. And we all know what comes with great power, or at least ought to.<br />Anonymoushttps://www.blogger.com/profile/08952112001761910889noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-43753444352597678422013-09-11T14:19:53.292-07:002013-09-11T14:19:53.292-07:00Hi Deborah,
Thanks for the comment. I enjoy read...Hi Deborah, <br /><br />Thanks for the comment. I enjoy reading post-publication comments by you, Dan Simons, and others, and I hope you continue to write them. My suggestion was not at all that you or others should refrain. <br /><br />Instead, I was trying to argue two things:<br /><br />1. Post-publication peer review doesn't have the right incentive structure to replace standard peer review (because most papers won't get comments, and audit pressure doesn't do a great job of enforcing compliance on its own), and<br /><br />2. Positive, synthetic comments *in addition to* critical comments are necessary in order to create culture changes. <br />Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-55076529740889679552013-09-11T12:01:23.600-07:002013-09-11T12:01:23.600-07:00You seem to be coming close to saying that, if you...You seem to be coming close to saying that, if you find problems with a paper that has received massive uncritical media coverage, you should not say anything in public about it, because it will upset the author.deevybeehttps://www.blogger.com/profile/15118040887173718391noreply@blogger.com