Implicit Reporting Standards in Bibliometric Research: What Can Reviewers’ Comments Tell Us About Reporting Completeness?

The rapid growth in the number of bibliometric studies in recent years has been accompanied by increasing diversity in the quality of the reporting of these studies’ methodologies and results. This ongoing study explores and systematises the quality and completeness of reporting bibliometric research using a bottom-up approach based on open peer review. We first identified 89 bibliometric studies published in library and information science (LIS) journals and conference proceedings and non-LIS journals, and then retrieved the 194 corresponding first-round reviews. From these reviews we extracted 968 reviewer comments pertaining to aspects of reporting the details of the se studies, and inductively classified these comments into 11 broad thematic categories and 68 sub-categories. Our preliminary results find that 77% of comments overall and the majority in each broad category were critical, which could be expected given the purpose of peer review to identify opportunities for improvement. In contrast, comments relating to the provision of study data and to the overall assessment of articles were more likely to be positive. The most common themes of reviewers’ comments were critically appraising the details of the data, methods, visualisations and tables used, and the clarity of the research questions and text. The finalised results will provide a precise and practical outline of concrete items that should be reported in bibliometric research according to the implicit community standard. Our findings will highlight particular features of bibliometric reporting that could be strengthened, complementing existing initiatives to generate guidance for the complete and accurate reporting of bibliometric studies.