tag:blogger.com,1999:blog-4297242917419089261.post1861715826053046107..comments2024-03-09T01:05:10.754-08:00Comments on Babies Learning Language: Misperception of incentives for publicationMichael Frankhttp://www.blogger.com/profile/00681533046507717821noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-4297242917419089261.post-5430173060327815272016-05-09T01:34:54.299-07:002016-05-09T01:34:54.299-07:00Isn't there quite a bit of variability across ...Isn't there quite a bit of variability across academic systems? Some countries reportedly do require a specific number of publications for junior faculty positions, starting grants, promotion, and so on. My speculation is that that's most clearly the case in countries where bureaucrats or out-of-field faculty play a major role in the decision, but it's hard to know.<br /><a href="http://www.teefury.club/" rel="nofollow" title="teefury">teefury</a>|<a href="http://www.teefury.club/" rel="nofollow" title="teefury coupon">teefury coupon</a>Anonymoushttps://www.blogger.com/profile/15037409903012211585noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-1891407755580432562016-05-08T14:57:08.100-07:002016-05-08T14:57:08.100-07:00Screening applicants is indeed a very hard problem...Screening applicants is indeed a very hard problem. In the searches I've been involved in, they typically try to get two readers for every app, and those folks search for work that catches their interest. It's tough to do that for 40 candidates, but way easier than doing it for 150. <br /><br />I'm not trying to argue that quantity of publications and their outlets *doesn't* matter. That would be silly. I also think it's likely that those signals are over the entire field decent but highly imperfect *correlates* of quality research. <br /><br />What I'm arguing is that candidates often *overweight* quantity due to availability bias (even over its actual importance), and so they should try to correct for that bias in discussion/evaluation.Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-43859615079597683082016-05-08T14:56:45.578-07:002016-05-08T14:56:45.578-07:00I agree about branding. That's a tough problem...I agree about branding. That's a tough problem and there has been some interesting research on this:<br /><br />http://advances.sciencemag.org/content/1/1/e1400005.shortMichael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-59217930190272242522016-05-06T04:28:43.166-07:002016-05-06T04:28:43.166-07:00Glad to hear that the hiring committees that you s...Glad to hear that the hiring committees that you served on were doing such a great job. Unfortunately, your experience may not be representative of other committees. I have talked to many senior researchers who regularly serve on hiring committees and almost all of them agree that quantity of publications and rank the journals published in are the most important factors determining whether you make it on the short list. Given that departments are typically inundated with large numbers of applications, that's actually not surprising because no one has the time to read articles from all those applicants to see how engaging their writing is and how deep their ideas. (How did you solve this problem?) Granted, once a candidate is on the shortlist, publications don't seem to matter so much but that's hardly comforting for those who didn't make it on that shortlist.<br /><br />This is how things work in the US, the UK is a different matter because of the REF evaluations. I've seen job ads from UK universities that made it perfectly explicit that publication metrics are the most important factor in the hiring decision. Given that publications in the UK more or less directly translate to money for the universities, this is not surprising either.Titus von der Malsburghttps://www.blogger.com/profile/14370107382857806010noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-33151492222767281492016-04-27T23:55:49.466-07:002016-04-27T23:55:49.466-07:00Hiring professors is something I've been invol...Hiring professors is something I've been involved in here in Germany, and one thing that should not be undervalued is the personality of the candidate---will people be able to work with them? A socially inept hire can cause havoc in a department even if he/she brings in millions of Euros in funding or tons of publications.<br /><br />In Germany, we get about 30-40 minutes with each candidate called in for a talk, and it's almost impossible to get a feeling for the person that quickly (however, I have evolved tricks to get a quick Stichprobe). <br /><br />Another thing worth mentioning is that the "branding" matters. A potential hire coming from Stanford will be taken much more seriously a priori than from a no-name university in the middle of nowhere. The priors are set by such cues and these are very hard to override with actual data on the person.<br /><br />I see this most dramatically in the reviewing process for funding; as a reviewer, I have seen funding agencies automatically fund a researcher who's too big to fail, even after a reviewer identified serious errors in their papers. Similarly, in hiring, after a certain threshold of achievement has been reached (and this can be as much a random walk to the top as a talent-driven rise), you will be golden no matter what. There can come a point when the science and quality ceases to matter.<br /><br />Shravan Vasishthhttps://www.blogger.com/profile/05926656325558456592noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-34859759081683549182016-04-26T20:29:02.644-07:002016-04-26T20:29:02.644-07:00This is an interesting question. I think at the ex...This is an interesting question. I think at the extreme, nobody's saying that quantity is the only thing that matters. Of course quality matters, of course any decent hiring committee is going to read publications as part of its process. I think the more interesting argument is, how *much* does quantity matter? And is it getting more weight than it should? That's a harder question, because "how much weight should it get?" is a fairly subjective question.<br /><br />When I think about this issue, there are a few things that come to mind. One is how expectations of publication records have grown. When I was in grad school on a search committee, the expectation was that one first-authored paper in a good journal was good enough. My strong impression today is that it would be pretty damn hard to get a tenure-track job at an R1 university with just 1 publication. When I talk to folks who've been around longer than me they report having a similar impression - the bottom-line expectation of quantity has gone up. That is going to have a particularly strong effect on grad students.<br /><br />A second thing is that I think quantity probably matters less the further through any selection process you go - maybe because we get more sophisticated about how we make decisions, but also maybe because by that point we've already sucked out a lot of the quantity variance by selecting on it. The cut from 200 initial applications to a "closer read" list of 20 probably looks a lot different from the cut from 3 interviews to 1 job offer. Once you've already made your cuts on quantity, restriction of range and Berkson's paradox and survivorship bias are going to make it look relatively less important. It continues to matter, but its relative weight is probably greatest at the earliest stages of selection. But to someone coming out of grad school, they're (legitimately) worried about making it through that first cut.<br /><br />A third thing is that things vary. Like yours, my department also has a pretty strong ethos of reading and discussing papers. One of the things I discovered when I was putting out feelers about that N-best evaluation thing is that how departments make decisions varies widely. Tal Linzen above mentioned some other countries count publications, but I've heard about departments in the U.S. that do the same thing - like, literally having points systems based on numbers of publications in various tranches of journals. And less formally, I hear about people being told that they need X number of publications per year or something like that to have a safe chance at tenure.<br /><br />Again, none of this means that we are ignorant of quality. Of course we aren't. Nor should we ignore quantity altogether — as you say, people have to be productive. The question though is, are we giving quantity the right emphasis, and not too much? And maybe it's less a matter of yes vs. no, than a matter of what parts of the decision process we are talking about. Quantity is easier to detect and easier to agree on than quality. My own hunch is that the places where we're most at risk of overweighting it are where we are pressed for time and attention (like first cuts) or where decision-makers have less expertise to evaluate quality (like as you move from committee to department to dean's office). If you're further along in the process, or if you've got enough quantity to satisfy those decision-makers, then it can feel like quality matters more. But that may not be equally true for everyone at all times.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-73885909172378236792016-04-26T09:25:16.760-07:002016-04-26T09:25:16.760-07:00Yes, agreed - I think this is true and lines up ni...Yes, agreed - I think this is true and lines up nicely with the idea that the more the decision-makers have a stake in the person, the less numbers matter. Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-34035977032606388572016-04-26T09:24:41.286-07:002016-04-26T09:24:41.286-07:00Thanks for the comment, Xenia. I'm sorry to he...Thanks for the comment, Xenia. I'm sorry to hear about these negative experiences. There are always bad actors in any field, as well as some bad incentive structures (I am learning from comments that strict numerical cutoffs are more prevalent outside the US). The one consolation I have is that there tend to be fewer bad actors in academia than outside, in part because the incentives for success are far smaller. :) Most of us do this because it's fun. Michael Frankhttps://www.blogger.com/profile/00681533046507717821noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-90861016486765937592016-04-26T00:41:52.941-07:002016-04-26T00:41:52.941-07:00Isn't there quite a bit of variability across ...Isn't there quite a bit of variability across academic systems? Some countries reportedly do require a specific number of publications for junior faculty positions, starting grants, promotion, and so on. My speculation is that that's most clearly the case in countries where bureaucrats or out-of-field faculty play a major role in the decision, but it's hard to know.Anonymoushttps://www.blogger.com/profile/14864640787642051975noreply@blogger.comtag:blogger.com,1999:blog-4297242917419089261.post-24192356557765026312016-04-25T23:29:59.583-07:002016-04-25T23:29:59.583-07:00Thanks, Michael, for this interesting and encourag...Thanks, Michael, for this interesting and encouraging blog post!<br /><br />I really like to hope that you’re right: that the majority of hiring (and funding) decisions are based on a careful evaluation of the candidates’ potential, rather than the number of papers. I am sure that this is how it works a lot of the time - in fact, I got my current post-doc position with “only” two published papers.<br /><br />However, there are clear examples that this is not always the case. It may differ, e.g., across countries, but there are definitely some cases where it was the number of papers that determined an (unfavourable) outcome:<br /><br />1) A friend of mine got rejected for a university Early Career Researchers fellowship (up to 3 years post PhD). She managed to get some insider info from the selection committee, and found out she was not even considered because she had less than 10 publications.<br />2) A relative of mine did not publish during her PhD, mainly because of health problems (caused, at least in part, by working in a “pimp’s” lab). I know her well enough to confidently say that she’s exceptionally intelligent and hard-working. Now she cannot even get an RA job.<br />3) Her “pimp” - though I have no sympathy for them - is under pressure, because if they don’t publish a certain amount of papers per year, they get demoted to a part-time position.<br />4) Two colleagues from another university quit their permanent jobs in protest, because the university wanted to increase their teaching load, while expecting them to publish the same amount, to manipulate the university’s (calculated) research output. <br />5) Taken from a blog I read just the other day (http://blogs.lse.ac.uk/impactofsocialsciences/2016/03/14/addicted-to-the-brand-the-hypocrisy-of-a-publishing-academic/; about impact factors rather than the number of publications): “And very often, the committee members will say something along the lines of “Well, Candidate X has got much better publications than Candidate Y”…without ever having read the papers of either candidate. The judgment of quality is lazily “outsourced” to the brand-name of the journal”.<br /><br />It is possible that these are just isolated instances - and I would be very happy if someone convinced me of that. However, even if they are, there is often the perceived pressure: I can’t think of any senior scientist who has given me career advice, who has’t told me that I need to publish a lot to get a job and funding in the future, and that my current output is probably not sufficient. And in many stories that I've heard about "pimps", ECRs are drilled with the credo "quantity matters, not quality". Whether or not this is justified, I think the perceived pressure alone causes a threat to quality of science.Xenia Schmalzhttps://www.blogger.com/profile/02238923475669435076noreply@blogger.com