IICON : identifying informative comments in online news

Many news outlets are discontinuing their comment sections due to moderation challenges as manual moderation for identifying irrelevant and informative comments is inadequate, costly, and time-consuming. Recognizing informative comments serve as an endorsement, giving both the comments and the news more credibility, engaging more readers, and shaping the discourse around the news article. An alternative to manual moderation is automating comment curation using data-driven methods, which reduce human moderation effort as an assistive tool. Most of these methods are based on term matching, such as TF-IDF or BM25, which do not adequately address the issue of identifying comments that do not significantly share terms with the article but are relevant to its context. This paper presents a framework, IICON, specifically designed for online news that takes a news article and the associated user comments as input and determines the most insightful comments. We develop sparse and dense retrieval models to work within IICON. To evaluate these methods, we create a training corpus of 18K news articles having 1M user comments from the Guardian. We also create an expert annotated testbed benchmark for which experiments show that IICON based on dense and sparse retrieval models performs competitively and outperforms existing methods by 2 percent to 8 percent in the mean precision.