A systematic review to evaluate the risk of bias of meta-analyses reporting experimental educational interventions focused on academic performance

Abstract

Introduction: Concerns about the risk of bias (RoB) of Meta-analysis (MAs) have grown in parallel with the exponential increase in the number of publications in science. However, this has not been properly assessed in Education. The aims were to evaluate the RoB of MAs in Education and to identify potential predictors of a lower RoB. Methods: Systematic review. Selection criteria were all MAs of experimental design evaluating the effectiveness of educational interventions on any academic outcome published from 1 January 2009 (year of publication of the first PRISMA guideline) published in English or Spanish, with the exclusion of those with other designs, evaluating other outcomes or not accessible to full text. A systematic search was performed in four databases (ERIC, Web of Science, Scopus, and PubMed) until March 2022. A preregistered protocol was used to extract data on study characteristics, PRISMA compliance, and RoB, based on the AMSTAR 2 instrument, and dichotomized as low vs. high RoB. The study selection and data extraction process were independently conducted by two researchers and disagreements were solved by consensus or by a third researcher. Statistical analysis: A flow-diagram and descriptive tables were tabulated. As a measure of association, odds ratios (OR) and its 95% confidence intervals were calculated by logistic regression analysis with dichotomized RoB as the dependent variable. Results: A total of 69 meta-analyses of studies were identified. Almost 90% (n¼62) of them were rated with a critically low overall confidence level, and almost 70% (n¼49) had a high RoB. Factors related to a low RoB were adherence to PRISMA guideline (OR ¼ 5.5; 95%CI: 1.8–16.6), the most recent studies (OR ¼ 7.4; 95%CI: 1.5– 35.3), a higher number of authors (OR ¼ 1.4; 95%CI: 1.1–1.9), a corresponding author from a European country (OR ¼ 3.7; 95%CI: 1.1–12.8), and publishing in the health educational area (OR ¼ 13.4; 95%CI: 3.6–49.6). Conclusions: Our study raises concerns regarding the methodological quality of published MA in Education. The use of instruments, such as AMSTAR 2 and PRISMA 2020, may improve the quality of future MA in Education.