IRPDP_HT2: A Scalable Data Pre-processing Method in Web Usage Mining using Hadoop-MapReduce

Abstract Data preparation is a vital step in the Web usage mining process since it provides structured data for the subsequent stages. Raw server logs should be turned into user sessions in order to generate structured data for pattern discovery phase. In recent decade, popular websites' server log production has risen to many terabytes to petabytes each day. As a result, server logs have Big Data challenges including storage and processing. Data cleaning, user identification, and session identification are data-intensive and deemed-computation-intensive operations in the data pre-processing sub phases. In recent years, Google's MapReduce parallel programming framework for data-intensive applications has grown in popularity as it provides the facility of scalable, flexible, fault-tolerant, distributed, and data-intensive processing over a cluster of nodes. In this paper, we provide IRPDP_HT2, a parallel data pre-processing algorithm using MapReduce that comprises data cleansing, user identification, and session identification approaches. The relevance and scalability of IRPDP_HT2 are also assessed using real web server logs, and it is discovered that the presented approach is both efficient and scalable.