The Quality of a Scientific Manuscript Given by a Peer-Reviewed. Report in Three Dimensions: Accessibility, Contribution and Experimentation (AccConExp)

Abstract

In this paper, we present a new model (AccConExp) to help the editor evaluate the reviewer's report in the peer-review process. The model provides information about accessibility, contribution and experimentation and analyses the sentiment of these characteristics. Accessibility pertains to the clarity and coherence of the manuscript; Contribution assesses whether the work is original or well justified; and Experimentation reflects the presence of significant comparisons or substance within the paper. For example, with this information, a journal editor can establish whether the paper meets the journal's standards given the polarity of accessibility, contribution or experimentation. The AccConExp model provides a strong and flexible framework for the analysis of reports that emphasise accessibility, contribution and experimentation. Its computational efficiency and scalability with emerging categories render it an essential resource for journal editors and various stakeholders within the academic and research communities. Furthermore, the AccConExp model introduces a novel method for improving the peer-review process by offering a more organised and insightful analysis of reviewers' reports, ultimately resulting in more consistent and high-quality assessments of scientific research. For this, the AccConExp model integrates a theoretical model based on partial least squares-structural equation modelling (PLS-SEM) to acquire new knowledge and a multi-task deep machine learning to explore the knowledge learning with the model PLS-SEM. The PLS-SEM part of the AccConExp model obtains a causal prediction from a set of aspect categories assigned to the reviewer's report to build new knowledge of the report based on accessibility, contribution and experimentation. The causal-exploratory capabilities of the multi-task deep learning model allow the labelling of new report's sentences based on accessibility, contribution, experimentation constructs and sentiment. Once we discover a sentence's construct, a second deep learning machine allows us to obtain its aspect category (clarity, soundness, originality, motivation, substance and meaningful comparison). The AccConExp model has been tested using reviewer reports from ICLR and NeurIPS papers (conferences with high impact in machine learning). The AccConExp model is compared with a multi-task architecture that assigns aspect categories to the report's sentences. The results obtained with the AccConExp model are competitive and allow us to give new information to the reviewer's reports without the effort to generate a new dataset labelled with these new constructs. Also, the AccConExp model's computational efficiency and capacity to adapt to new categories render it an invaluable resource for journal editors and various stakeholders within the academic and research community. The methodology used in this paper can be extended to other research fields to define its constructs, even if the aspects considered in the review's reports area differ from those used in this proposal. We release our codes for more study.