Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
Metadatos
Mostrar el registro completo del ítemEditorial
Elsevier
Materia
Crowdsourcing data analysis Scientific transparency Research reliability Scientific robustness Researcher degrees of freedom Analysis-contingent results
Fecha
2021-06-17Referencia bibliográfica
Martin Schweinsberg... [et al.]. Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis, Organizational Behavior and Human Decision Processes, Volume 165, 2021, Pages 228-249, ISSN 0749-5978, [https://doi.org/10.1016/j.obhdp.2021.02.003]
Patrocinador
INSEAD; Swiss National Science Foundation (SNSF) European Commission 143411Resumen
In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the
effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic
approach but also the operationalizations of key variables were left unconstrained and up to individual analysts.
For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or
some combination. To maximize transparency regarding the process by which analytic choices are made, the
analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths
in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded,
resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed
empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research
question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain
variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions
play a critical role in driving the reported empirical results, underscoring the need for open data, systematic
robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations
and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal
analyses by data scientists, are discussed.