Commenting on papers after they are published has to be a good idea in some form: more discussion of science and more opportunities to correct the record! But I want to argue against using post-publication peer review, at least in its current form, as the primary method of promoting stronger cultural norms for reliable research.
1. Pre-publication peer review is a filter, while post-publication peer review is an audit.
With few exceptions, peer-review is applied to all submitted papers. And despite variations in the process from journal to journal, the overall approach is quite standard. This uniformity doesn't necessarily lead to perfect decisions, where all the good papers are accepted and bad papers are rejected. Nevertheless, peer review is a filter that is designed to be applied across the board so that the scientific record contains only those findings that "pass through" (consider the implicit filter metaphor!).
When I review a journal article I typically spend at least a couple of hours reading, thinking, and writing. These hours have to be "good hours" when I am concentrating carefully and don't have a lot of disruptions. I would usually rather be working on my own research or going for a walk in the woods. But for a paper to be published, a group of reviewers needs to commit those valuable hours. So I accept review requests out of a sense of obligation or responsibility, whether it's to the editor, the journal, or the field more generally.
This same mechanism doesn't operate for post-publication review. There is no apparatus for soliciting commenters post-publication. So only a few articles, particularly those that receive lots of media coverage, will get the bulk of the thoughtful, influential commentary (see e.g. Andrew Gelman's post on this). The vast majority will go unblogged.
Post-publication peer review is thus less like a filter and more like an audit. It happens after the fact and only in select cases. Audits are also somewhat random in what attracts scrutiny and when. There is always something bad you can say about someone's tax return - and about their research practices.
2. Post-publication peer review works via a negative incentive: social shaming.
People are generally driven to write thoughtful critiques only when they think that something is really wrong with the research (see e.g. links here, here, here, and here). This means that nearly all post-publication peer review is negative.
The tone of the posts linked above is very professional, even if the overall message about the science is sometimes scathing. But one negative review typically spurs a host of snarky follow-ons on twitter, leaving a single research group or paper singled out for an error that may need to be corrected much more generally. Often critiques are merited. But they can leave the recipients of the critique feeling as though the entire world is ganging up against them.
For example, consider the situation surrounding a failure to replicate John Bargh's famous elderly priming study. Independent of what you think of the science, the discussion was heated. A sympathetic Chronicle piece used the phrase "scientific bullying" to describe the criticisms of Bargh, noting that this experience was the "nadir" of his career. That sounds right to me: I've only been involved in one, generally civil, public controversy (my paper, reply, my reply back, final answer) and I found that experience extremely stressful. Perceived social shaming or exclusion can be a very painful process. I'm sure reviewers don't intend their moderately-phrased statistical criticisms to result in this kind of feeling, but - thanks to the internet - they sometimes do.
3. Negative incentives don't raise compliance as much as positive cultural influences do.
Tax audits (which carry civil and criminal penalties, rather than social ones) do increase compliance somewhat. But a review of the economics literature suggests that cultural factors - think US vs. Greece - matter more than the sheer expected risk due to audit enforcement (discussion here and here).* For example, US audit rates have fluctuated dramatically in the last fifty years, with only more limited effects on compliance (see e.g. this analysis).**
Similarly, if we want to create a scientific culture where people follow good research practices because of a sense of pride and responsibility - rather than trying to enforce norms through fear of humiliation - then increasing the post-publication audit rate is not the right way to get there. Instead we need to think about ways to change the values in our scientific culture. Rebecca Saxe and I made one pedagogical suggestion here, focused on teaching replication in the classroom.
Some auditing is necessary for both tax returns and science. The overall increase in post-publication discussion is a good thing, leading to new ideas and a stronger articulation and awareness of core standards for reliable research. So the answer isn't to stop writing post-pub commentaries. It's just to think of them as a complement to - rather than a replacement for - standard peer review.
Finally, we need to write positive post-publication reviews. We need to highlight good science and most especially strong methods (e.g. consider The Neurocomplimenter). The options can't be either breathless media hype that goes unquestioned or breathless media hype that is soberly shot down by responsible academic critics. We need to write careful reviews, questions, and syntheses for papers that should remain in the scientific literature. If we only write about bad papers, we don't do enough to promote changes in our scientific standards.
* It's very entertaining: The tone of this discussion is overall one of surprise that taxpayers' behavior doesn't hew to the rational norms implied by audit risk.
** Surprisingly, I couldn't find the exact graph I wanted: audit rates vs. estimated compliance rates. If anyone can find this, please let me know!