Impact factors are hooey
One of the major metrics scientists use to rank journals is the “Impact Factor“. This is a real number calculated in mysterious ways by the Thomson Corporation. The problem with it is twofold: first of all, it’s not accurate; secondly, it’s held in far too high of a position of importance (especially given point one).
This came up in my thoughts because of a comment from Sciencewoman in response to my recommendation that she look into publishing her latest paper in an OA journal:
PA: I looked at the directory [of open access journals] and the only appropriate OA journal had a significantly lower impact factor. Again, as a pre-tenured person, I need to be aware of those things.
This disheartens me because I think that it is precisely amongst the young faculty that a preference for OA publishing can take hold.
So, what is wrong with impact factors? I hardly know where to start… Indeed the subject has been covered extensively elsewhere, and I’ll refer the reader to papers from Nature, Seglen, & Postma.
Let’s look at this case specifically. Sciencewoman is worried that publishing her work in a journal with a low impact factor will reflect negatively on the paper - the tenure committee won’t think that it’s “prestigious”. First of all, this is a fallcious argument. Impact factors are an aggregate of citation counts for that journal (more or less), and therefore reflect only on the journal as a whole, not individual papers within it. For instance, let’s say I wrote a really smashing paper in the Journal I Just Made Up, which was cited by every scientist twice. Since all the other papers in JIJMU were random scraps I found in rubbish bins, they weren’t cited at all. Just because JIJMU would have a fantastic impact factor (hauled up by my honestly brilliant paper) that doesn’t mean that my neighbor’s grocery list is also groundbreaking research. This is an extreme example, but I think you get my point.
There are more specific problems with impact factors as they relate to OA journals. These journals tend to be younger, and therefore are inherently shortchanged when it comes to impact factors which are tallied over several years. They often serve niche fields, in which the overall impact factor might be lower (the numbers across fields cannot really be directly compared, because of different audience sizes and referencing traditions). To be fair, there are certain ways that OA journals have an advantage when it comes to impact factors - it’s been shown that OA papers tend to be referenced more often than their closed counterparts.
The fact is that the impact factor as a metric of research quality is embarassingly bad, and I’d be happy if we could just do away with the notion altogether. I definitely think that a young faculty member up for tenure should be able to make a convincing argument that the committee should look past a flawed number and appreciate the desire to make the work accessible to the largest community possible, by publishing it openly.
April 16th, 2008 at 2:57 pm
The young faculty member could make a convincing case for the tenure-related benefits of OA (e.g. citation advantage, maximal efficiency of the research cycle), and an equally convincing case that the tenure committe should be rewarding OA publication in any case, on the basis that publicly funded researchers have a responsibility to make their work as widely available as possible.
She could also use a number of alternative indices of research impact (h index, Eigenfactor, etc) — none are perfect but in aggregate they could demonstrate that her research is of a higher quality than might be indicated by the deeply flawed Impact Factor.