When it comes to academic evaluation, is it not uncommon for scholars to succumb to the delusion that academic merit is accrued solely by publishing in high-profile journals. While the extent to which an article is cited and recognized by the scholarly community is undeniably a significant criterion for academic evaluation, the purpose of citation indexes should not be overstated. Citation indexes are foremost used to recommend articles through citation and retrieval services; as such, they are primarily intended to be gauges of relevance, and secondarily as evaluation mechanism. In fact, the original idea behind the creation of citation indexes was not at all to rate journals, institutions and scholars. Rather, as Eugene Garfield observed in his 1955 article “Citation Indexes for Science”, they are simply expected to facilitate better understanding and cataloguing of scientific history and the structure of scientific research.

 

To this point, citation indexes enable us to chart the connections and intersections of scientific papers in various fields of science, as well as to show the individual and collective achievements that have been made in each field. With the ease of access to indexes scholars enjoy today, they can advance their discipline by building on the contributions of their peers internationally, rather than repeating research that has already been done elsewhere. However, it is clear that they provide only a very cursory and indirect evaluation of an article’s content—the algorithms assess and categorize information that is much too general to draw a detailed, pointed sense of an articles’ merit.

 

Moreover, as tools of academic evaluation, citation indexes have several notable flaws. First, they only demonstrate superficial connections in the intricate process of citation. Second, citation ratings only show the degree to which an article is cited; this alone is insufficient to determine academic influence and potential—judgment which requires careful peer analysis. Third, an article’s citation rate is affected by various factors such as an article’s availability, date of publication, field, language and author’s language capability, all of which affect other scholars’ ability to or likelihood of accessing and citing it, regardless of its quality. Fourth, journal rankings and impact facter may fluctuate from factors such as expansion or contraction of its total contents, changes to the title of the journal, and even the accuracy of keywords tags. 

 

Evaluation of scholarly publications is much more complicated than the minimal and imperfect measures provided by citation indexes. While generalized quantitative feedback can increase efficiency, we must remember that as the results of a blood test or the image produced by an MRI scan can aid but not replace medical diagnosis, statistical measures offered by citation indexes are no substitute for in-depth evaluation. We must exercise prudence when using quantitative measures to assess specific details within an article, lest we implement a policy of rigidity that inhibits long-term innovation.

 

Shen Guzhao is from Nanjing University.

 

the Chinese version appeared in Chinese Social Sciences Today, No. 281, Mar. 19.

 

Translated by Jiang Hong