tag:blogger.com,1999:blog-4297242917419089261.post7184078301600941895..comments2017-05-22T06:04:45.753-07:00Comments on Babies Learning Language: A moderate's view of the reproducibility crisisMichael Frankhttp://www.blogger.com/profile/00681533046507717821noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-4297242917419089261.post-75260470745785446752015-09-18T10:02:38.087-07:002015-09-18T10:02:38.087-07:00Thanks for the comments! A few followups:
1b. Exc...Thanks for the comments! A few followups:<br /><br />1b. Exclusion of studies from publication is a different (though related problem). My point was that if you publish those four studies together, then their analytic decisions should constrain one another. If you swap e.g. exclusion criteria or dependent variables from study to study in a single paper, then it's pretty obvious what you're doing. <br /><br />2. Maybe I should have said *writing* a paper is a lot of work! I don't think it's just the publication process. We can lower the bars to publication, but it's still not trivial to craft a decent manuscript.<br /><br />3a/b. I'm arguing these points as someone whose career has been about using Bayesian methods! It's not that I don't like Bayesian statistics altogether. But the fact is that the frequentist linear model is easy to use and easy to reason about and it will take a lot of education as well as theoretical development to change that. I've used Bugs and Jags (though not PyMC) and can vouch for the fact that even though these tools generally work well, they are still *way* harder to use and to understand than lm(y ~ x), and the benefits are not always obvious. Michael Frankhttp://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-53009902546513602182015-09-18T09:53:38.624-07:002015-09-18T09:53:38.624-07:00Some quick notes:
1a. I don't think the advoc...Some quick notes:<br /><br />1a. I don't think the advocates of preregistration want to disband exploratory research altogether. <br />1b. > you don't need to pre-register because your previous work naturally constrains your analysis<br />only the previous *published* work constrains the analysis etc. A researcher may decide to leave 2 out of 6 studies out and publish the remaining 4 as a single paper. There is no way to know he left out 2 papers based on the remaining work. That's why pre-registration is necessary. <br />2. > Publishing a paper, even a short one or a preprint, is a lot of work.<br />Agreed. But I see this more as a problem of the current publishing system and the current publishing format. <br />3a.> You don't have to be an expert to understand someone's ANOVA analysis. But if everyone uses one-off graphical models, then there are many mistakes we will never catch due to the complexity of the models. <br /><br />Is this not a consequence of the asymetric focus of the current education on freq stats? If people were taught bayes stats maybe they would find bayes easy and they would be confused by freq methods, no?<br />(btw. bayesian anova does exist and some researchers prefer to estimate graphical models with freq methods, the examples are not well chosen)<br />3b. Stan is very recent - 3 years old, it's not surprising that it has bugs. More mature software like bugs, jags or pymc exists and should be considered Though, from my experience, virtualy, all problems I encountered in Stan were promptly solved by upgrading to the most recent Stan version and I can recommend the software with good conscience.mozgostrojehttps://mozgostroje.wordpress.com/noreply@blogger.com