By Luisa Veras de Sandes-Guimarães

One of the approaches for evaluating the quality of scientific periodicals, based on a quantitative and performance analysis of the research, involves the use of bibliometric impact indicators. Based on the number of citations, the level of interest of other researchers in published articles can be verified for a given period. Calculating the impact of publications on the scientific community is part of the study and analysis of citations, an area pertinent to Bibliometrics.

Analyzing citations and the impact factor

The creation and dissemination of knowledge in an area is facilitated by the circulation of ideas among research groups. Each author contributes to the body of knowledge in his or her area, creating and adding new information. However, it should be emphasized that an article is never isolated, but rather a contribution to the field literature which forms the base on top of which new studies are created. In this process, citations are extremely important, because they symbolize the recognition of the quality of previously written and published works.

According to Guedes and Borschiver (2005), the analysis of citations has three main applications: in auxiliary libraries for the management of collections; in science it serves the purpose of mapping the performance of authors; and in management it provides a base for making decisions in terms of the funding of research, aid, scholarships, and IT budgets, etc. This technique is also used to evaluate scientific periodicals, calculate author productivity, measure the quality of information, measure information flow, and to indicate scientific structures and trends, etc.

Citations represent a link between the person citing and the document cited. The nature of this relationship is difficult to characterize, due to the multiple reasons that authors cite other authors, like the fifteen given by Garfield: to pay homage to pioneers; to give credit to related works (paying homage to peers); to identify equipment, methodologies, etc.; to provide a series of readings on a subject; to correct the work of others; to correct one’s own work; to criticize previous studies; to justify claims; to mention future works; to promote little known or poorly indexed works or works lacking citations; to authenticate data and classes of constant physical facts, etc.; to identify original publication in which an idea or concept was discussed; to identify original publications or other works that describe a concept or eponymous term; to rebut the works or ideas of other people (negative allegations); to dispute allegations of priority on the part of others (denying credit) (Smith, 1981).

The perfecting of citation analysis has been characterized by the development of new techniques and measures, by research with new tools and the study of units using distinct analyses. These trends have caused an accelerating increase in the quality of studies that use citation analysis. The first citation studies were based, in general, on reference lists in articles published by a select group of periodicals. Citation data was transcribed and manipulated manually. With such a backward process, most of the studies thus had to deal with a limited scope. However, with the introduction of computers and information systems, it was possible to optimize this situation significantly in two ways: by printing data indices using citations from an enormous number of documents, and by analyzing citation data that could be read by machines (Smith, 1981).

According to Araújo (2006) the reference counting method was first used by Gross & Gross in 1927, then by Allen in 1929, followed by Gross & Woodford in 1931. However, it was in the 1960s, with the use of new computational tools that the technique gained new life. 1963 marked a new phase in the history of bibliometics when Eugene Garfield, founder of the Institute of Scientific Information (ISI), created the first citation index, called the Science Citation Index (SCI). Later Garfield created other indices seeking to encompass other areas of knowledge, such as the Social Sciences Citation Index (SSCI) and the Arts & Humanities Citation Index (A&HCI). Garfield also created one of the most relevant concepts in the study of citations, the Impact Factor.

In the beginning of the 1960s, Irving Sher and Eugene Garfield created the Impact Factor for periodicals to help in the process of selecting journals to be included in the Science Citation Index (SCI). Both realized that the SCI needed to have a core group of often cited large publications. However, they also recognized that small, but important review periodicals wouldn’t be selected if the criteria only considered publication itself or the number of citations. Thus they needed a simple method to compare periodicals no matter what their size, and this is why they created the Impact Factor (Garfield, 1999).

A periodical’s impact factor is based on two elements: the numerator, which is the number of citations during the current year of any articles published in a journal over the previous two years, and the denominator, which is the total number of articles published in that periodical over these same two years. Impact Factor data is published for every indexed periodical (today aggregated from the Web of Science database) and has been published annually since 1976 in the Journal of Citations Reports (JCR).

Example of calculating the Impact Factor

2006 IF for the periodical Physical Review Letters
Nº of citations made in 2006 of articles published in:	2005= 2004= Total=	28,078 23,332 51,410
Nº of articles published in:	2005= 2004= Total=	3,694 3,575 7,269
Calculation
Citations made / Number of articles	51,410 / 7,269 =	7.072

 

Source: adapted from Mugnaini and Strehl (2008)

Garfield (1999) believes that citation studies should be normalized taking into consideration variables like the discipline or area of knowledge and citation practices. The density of citations and their half-lives are also important variables. Citation density (the average number of references cited per article) would be significantly less for an article dealing with mathematics than one dealing with life sciences. The half-life (the number of years that span 50% of the citations made in the current year) for a physiology journal would be longer than the period for molecular biology or astronomy periodical. Despite Garfield’s reservations, the index continues to be calculated and interpreted without considering the variables that Garfield has cited (citation half-life and density for each area of knowledge).

Dong, Loh and Mondry (2005) affirm that the immediate availability of the IF and the lack of other recognized quality indicators have contributed to the adoption of IF as the quality indicator for periodicals. However, the authors point out that the IF calculation relies on various factors, including: language and the preferential breadth of databases; the procedures used to collect citations; the algorithm used to calculate the IF; the distribution of periodical citations; citations of invalid articles; publishers’ preferences for certain types of articles; the behavior of citations in diverse disciplines; and the possible influence of periodical editors.

According to the authors the ISI Web of Science databases cover less than a quarter of world scientific periodicals and display a preference for periodicals published in English. Periodicals not published in this language have a relatively low IF due to the sparse coverage of foreign language periodicals in these databases. (Dong et al., 2005). The authors point out that today English is the lingua franca of science, as German was in the 19^th century and the beginning of the 20^th century, and Latin and Greek were in previous centuries.

In addition to this, the IF only considers citations coming from journal articles indexed in the ISI Web of Science. Thus the index doesn’t include citations that appear in books, event articles, theses, dissertations and other periodicals that are not indexed in this database (Harzing & Van der Wal, 2008). This point is detrimental to Brazilian periodicals because in addition to Portuguese being an under-represented language, only a few of these periodicals appear in the database, resulting in an IF that only partially represents the impact of a given publication.

Another issue pointed out by several studies is the difference in the standards for publication and citation in various disciplines (Leite, Mugnaini, & Leta, 2011; Mugnaini & Población, 2010; Harzing & Van der Wal, 2008; Mugnaini & Strehl, 2008; Dong et al., 2005; Amin & Mabe, 2000; Narin, 1976). Articles in growth areas trend to cite much more recent references than more traditional fields of research, in particular theoretical areas and mathematics, leading to substantial variations in IFs between different disciplines. Also the collection of citations for a period of just two years after publication has an important influence on the IF. Research periodicals in areas that are growing rapidly tend to publish articles with a shorter time period between submission and acceptance. A large proportion of works are cited within two years, and thus, the periodical has a high IF. Nonetheless, there are many periodicals in other areas that have longer citation half-lives and many works in these journals are cited for a much longer period than two years after publication.

Mugnaini and Población (2010) conducted a study of the impact of document types (articles, books, annals, theses, etc.) on citations in five scientific journals in distinct areas of knowledge. The authors found that books are more cited in a book on Applied Social Sciences, while in a Collective Health periodical this type of document is cited as often as scientific articles. Citations of international periodicals are prevalent in Physics and Medical reviews. Annals and theses are the focus of Veterinary and IT journals. These findings are important to understanding each area’s culture of scientific communication.

Leite et al. (2011) proposed a new approach to investigate scientific productivity. The International Publication Ratio (IPR) was developed to distinguish between groups with different publication tendencies. The authors used the Lattes database to collect information about the PhDs in the Brazilian scientific community, including their area of knowledge, affiliation and publications. All totalled, they analyzed the résumés of 34,390 researchers and classified their publications into five groups according to their IPR: (1) highly international (between 80.1-100% international publications), (2) mainly international (60.1-80%), (3) intermediate (40.1-60%), (4) mainly domestic (20.1-40%) and (5) highly domestic (0-20%). The IPR data was associated with the researchers’ areas of knowledge.

They found evidence to suggest that international performance is a variable that depends on the area of knowledge. Areas dedicated to questions of international interest, such as Biology, Engineering and Exact and Earth Sciences, represent a large proportion of the researchers with the highest IPR. But this isn’t true for fields essentially devoted to questions that are of domestic or local interest. The use of IPR offers a good example of the importance of the idiosyncrasies of each field in determining the critical factors to be considered when comparing performance in different areas in a scenario where the general evaluation determines the destination of financial resources (Leite et al., 2011).

According to Dong et al. (2005), due to the preference that authors and researchers give to journals with a high IF, editors may be tempted to artificially inflate the IF of their periodical. A very simple way to do this is to ask authors to cite the journal’s articles. In 1997, the periodical Leukemia was accused of trying to manipulate its IF. The periodical asked authors who had submitted articles to cite more articles from their review. A few years later in 2002 there was a similar case in which a periodical editor suggested that more references be included from his periodical. Sevinc (2004) cites four other cases of IF manipulation, and Falagas and Alexiou (2008) cite ten other cases.

Mugnaini and Strehl (2008) point out that for a long time only the data from the ISI Web of Science databases and the indicators of the JCR offered a notion of the impact of periodicals in the academic scientific community. According to the authors, this hegemony became entrenched even in the countries whose scientific periodicals were being underrepresented in these international databases.

However, Mugnaini and Strehl (2008) affirm that this situation is changing, because other databases with indexed citations have been created and thus offer competition to the ISI Web of Science in the production of data to calculate the average impact of scientific periodicals. The authors indicate that the following databases are particularly good in terms of the context of science in developing countries: the Scientific Electronic Library Online (SciELO), Scopus and Google Scholar. In the area of Administration, the Scientific Periodicals Electronic Library (SPELL) stands out with 92 periodicals in the areas of Administration, Accounting and Tourism, and soon it will be publishing impact indicators.

In addition to this, other indices to measure impact have been invented such as the H-index, the G-index, the E-index and, most recently, Altmetrics, which proposes using non-traditional metrics to measure the impact of articles, such as the number of downloads and views, discussions in social networks, and bookmarking favorites, among others.

H Index

The H Index was proposed by the researcher Jorge Hirsch (2005) and initially was intended to supplement the qualitative evaluation of researchers in the area of physics. However, this measurement became popular in other areas of knowledge and today is very heavily used to evaluate the impact of researchers. Beyond this, it has also come to be used to calculate the impact of scientific periodicals, as was subsequently proposed by Braun, Glanzel and Schubert (2006). It’s calculated in the following manner: a scientist/periodical has a value of H, if H of his or her N articles possess at least H citations apiece, and the other articles have less than H citations apiece (Hirsch, 2005). For example, a researcher/periodical with an H Index value of 7 has 7 articles which have at least 7 citations apiece.

As a result, the H index provides a combination of quantity (the number of articles published) and quality (impact, or citations that these documents receive). Thus the H Index is preferable to the total number of citations, because it corrects for one hit wonders, which is used to describe academic authors who have had just one or a few highly cited articles, but do not demonstrate sustained academic performance over a long period of time. The H Index is also preferable to the number of articles published, because it corrects for articles that are not cited and which have had a limited impact on the field. To sum up, the H Index favors academics who publish a constant flow of articles with above average and lasting impact (Harzing & Van der Wal, 2009).

According to the authors, in the same way that the H Index is a measurement of sustained and durable academic performance, the H Index for periodicals proves to be a robust measure of sustained and durable performance by periodicals. The authors made a comparison between the H Index calculated using citations obtained from Google Scholar and the Impact Factor calculated using the Web of Science database for a sample of 838 economics and administration articles, and their results indicate that the former is a broader, more accurate measurement of the impact of periodicals.

References

Amin, M., & Mabe, M. (2000). Impact factor: use and abuse. Perspectives in Publishing, (1), 1-6.

Araújo, C. A. (2006). Bibliometria: evolução histórica e questões atuais. Em Questão, 12(1), 11-32.

Braun, T., Glanzel, W., & Schubert, A. (2006). A Hirsch-type index for journals. Scientometrics, 69(1), 169-173. doi:10.1007/s11192-006-0147-4.

Dong, P., Loh, M., & Mondry, A. (2005). The impact factor revisited. Biomedical Digital Libraries, 2. doi:10.1186/1742-5581-2-7.

Falagas, M., & Alexiou, V. (2008). The top-ten in journal impact factor manipulation. Archivum Immunologiae et Therapiae Experimentalis, 56(4), 223-226. doi: 10.1007/s00005-008-0024-5.

Garfield, E. (1999). Journal impact factor: a brief review. Journal of the American Medical Association, 161(8).

Guedes, V., & Borschiver, S. (2005). Bibliometria: uma ferramenta estatística para a gestão da informação e do conhecimento, em sistemas de informação, de comunicação e de avaliação científica e tecnológica. Anais do Encontro Nacional de Ciência da Informação, Salvador, BA, Brasil, 6.

Harzing, A. W., & Van der Wal, R. (2009). A Google Scholar H-Index for journals: an alternative metric to measure journal impact in economics and business. Journal of the American Society for Information Science and Technology, 60(1), 41-46. doi: 10.1002/asi.20953.

Harzing, A. W., & Van der Wal, R. (2008). Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, 8, 61-73, 2008. doi: 10.3354/esep00076.

Hirsch, J. E. (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences, 102(46), 16569-16572, 2005. doi:10.1073/pnas.0507655102.

Leite, P., Mugnaini, R., & Leta, J. (2011). A new indicator for international visibility: exploring Brazilian scientific community. Scientometrics, 88(1), 311-319. doi: 10.1007/s11192-011-0379-9.

Mugnaini, R., & Población, D. (2010). Multidisciplinaridade e especificidade na comunicação científica: discussão do impacto na avaliação de diferentes áreas. Revista Eletrônica de Comunicação, Informação e Inovação em Saúde, 4(5), 23-30.

Mugnaini, R., & Strehl, L. (2008). Recuperação e impacto da produção científica na era Google: uma análise comparativa entre o Google Acadêmico e a Web of Science. Encontros Bibli, 13(nesp), 92-105. doi:10.5007/1518-2924.2008v13nesp1p92.

Narin, F. (1976). Evaluative bibliometrics: the use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill: Computer Horizons.

Sevinc, A. (2004). Manipulating impact factor: an unethical issue or an editor’s choice. Swiss Medical Weekly, 134(410). 

Smith, L. (1981). Citation analysis. Library Trends, 30(1), 83-106.