Losing objectivity: The questionable use of surveys in the Global Ranking of Academic Subjects

Abstract

The Academic Ranking of World Universities (ARWU) is one of the most well-known university rankings, recognized for its objective and reproducible methodology. In contrast, the Global Ranking of Academic Subjects (GRAS), which ranks institutions by scientific subjects and is also elaborated by Shanghai Ranking Consultancy (SRC), introduces methodological differences that deviate from the ARWU’s objectivity. This is due to the use of SRC’s Academic Excellence Survey to define two of the GRAS’s five indicators. Specifically, the Top indicator counts publications in journals determined by respondents as top tier in their field, and the Award indicator does the same for prizes. An examination of this survey suggests the presence of potential biases, especially in participant selection and journal identification, among which an Anglo-Saxon bias is prominently evident. Likewise, there is a potential risk that the selection of journals in some cases may be influenced, potentially masking conflicts of interest, such as involvement in editorial committees that could sway this selection. As a result, relying on surveys instead of adhering to established bibliometric standards can lead to inconsistencies and subjectivity, especially if not rigorously conducted. Such methodologies pose a risk to the trustworthiness of tools crucial for university policymaking.