The Integrity of Source Code Commenting : Benchmark Dataset and Empirical Analysis

Abstract Code comments are a vital software feature for program cognition &software maintainability. For a long time, researchers have been tryingto find ways to ensure the consistency of code-comment. While doingthat, two of the raised problems have been dataset scarcity and languagedependency. To address both problems in this paper, we worked on adataset creation made using C# projects; there are no annotated datasetsyet on C#. 9,310 code-comment pairs of different C# projects wereextracted from a data pool. 4,922 code-comment pairs were annotatedafter removing NULL, constructor, and variable. Both method-commentand class-comment were considered in this study. We employed twoevaluation metrics for the dataset, one is Krippendorff’s Alpha whichshowed 95.67% similarity among the rating of 3 annotators for all thepairs & other is Bilingual Evaluation Understudy (BLEU) to validateour human-curated dataset. A modified model from a previous study isalso proposed, which obtained 96.2% using the performance metric AUC-ROC after fitting the model to our annotated 4,922 code-comment pairs.