Javascript must be enabled to continue!
Advanced methods for missing values imputation based on similarity learning
View through CrossRef
The real-world data analysis and processing using data mining techniques often are facing observations that contain missing values. The main challenge of mining datasets is the existence of missing values. The missing values in a dataset should be imputed using the imputation method to improve the data mining methods’ accuracy and performance. There are existing techniques that use k-nearest neighbors algorithm for imputing the missing values but determining the appropriate k value can be a challenging task. There are other existing imputation techniques that are based on hard clustering algorithms. When records are not well-separated, as in the case of missing data, hard clustering provides a poor description tool in many cases. In general, the imputation depending on similar records is more accurate than the imputation depending on the entire dataset's records. Improving the similarity among records can result in improving the imputation performance. This paper proposes two numerical missing data imputation methods. A hybrid missing data imputation method is initially proposed, called KI, that incorporates k-nearest neighbors and iterative imputation algorithms. The best set of nearest neighbors for each missing record is discovered through the records similarity by using the k-nearest neighbors algorithm (kNN). To improve the similarity, a suitable k value is estimated automatically for the kNN. The iterative imputation method is then used to impute the missing values of the incomplete records by using the global correlation structure among the selected records. An enhanced hybrid missing data imputation method is then proposed, called FCKI, which is an extension of KI. It integrates fuzzy c-means, k-nearest neighbors, and iterative imputation algorithms to impute the missing data in a dataset. The fuzzy c-means algorithm is selected because the records can belong to multiple clusters at the same time. This can lead to further improvement for similarity. FCKI searches a cluster, instead of the whole dataset, to find the best k-nearest neighbors. It applies two levels of similarity to achieve a higher imputation accuracy. The performance of the proposed imputation techniques is assessed by using fifteen datasets with variant missing ratios for three types of missing data; MCAR, MAR, MNAR. These different missing data types are generated in this work. The datasets with different sizes are used in this paper to validate the model. Therefore, proposed imputation techniques are compared with other missing data imputation methods by means of three measures; the root mean square error (RMSE), the normalized root mean square error (NRMSE), and the mean absolute error (MAE). The results show that the proposed methods achieve better imputation accuracy and require significantly less time than other missing data imputation methods.
Title: Advanced methods for missing values imputation based on similarity learning
Description:
The real-world data analysis and processing using data mining techniques often are facing observations that contain missing values.
The main challenge of mining datasets is the existence of missing values.
The missing values in a dataset should be imputed using the imputation method to improve the data mining methods’ accuracy and performance.
There are existing techniques that use k-nearest neighbors algorithm for imputing the missing values but determining the appropriate k value can be a challenging task.
There are other existing imputation techniques that are based on hard clustering algorithms.
When records are not well-separated, as in the case of missing data, hard clustering provides a poor description tool in many cases.
In general, the imputation depending on similar records is more accurate than the imputation depending on the entire dataset's records.
Improving the similarity among records can result in improving the imputation performance.
This paper proposes two numerical missing data imputation methods.
A hybrid missing data imputation method is initially proposed, called KI, that incorporates k-nearest neighbors and iterative imputation algorithms.
The best set of nearest neighbors for each missing record is discovered through the records similarity by using the k-nearest neighbors algorithm (kNN).
To improve the similarity, a suitable k value is estimated automatically for the kNN.
The iterative imputation method is then used to impute the missing values of the incomplete records by using the global correlation structure among the selected records.
An enhanced hybrid missing data imputation method is then proposed, called FCKI, which is an extension of KI.
It integrates fuzzy c-means, k-nearest neighbors, and iterative imputation algorithms to impute the missing data in a dataset.
The fuzzy c-means algorithm is selected because the records can belong to multiple clusters at the same time.
This can lead to further improvement for similarity.
FCKI searches a cluster, instead of the whole dataset, to find the best k-nearest neighbors.
It applies two levels of similarity to achieve a higher imputation accuracy.
The performance of the proposed imputation techniques is assessed by using fifteen datasets with variant missing ratios for three types of missing data; MCAR, MAR, MNAR.
These different missing data types are generated in this work.
The datasets with different sizes are used in this paper to validate the model.
Therefore, proposed imputation techniques are compared with other missing data imputation methods by means of three measures; the root mean square error (RMSE), the normalized root mean square error (NRMSE), and the mean absolute error (MAE).
The results show that the proposed methods achieve better imputation accuracy and require significantly less time than other missing data imputation methods.
Related Results
GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies
GSimp: A Gibbs sampler based left-censored missing value imputation approach for metabolomics studies
Abstract
Left-censored missing values commonly exist in targeted metabolomics datasets and can be considered as missing not at random (MNAR). Imp...
A New Approach of Outlier-robust Missing Value Imputation for Metabolomics Data Analysis
A New Approach of Outlier-robust Missing Value Imputation for Metabolomics Data Analysis
Background:Metabolomics data generation and quantification are different from other types of molecular “omics” data in bioinformatics. Mass spectrometry (MS) based (gas chromatogra...
Uncovering the consequences of batch effect associated missing values in omics data analysis
Uncovering the consequences of batch effect associated missing values in omics data analysis
ABSTRACTStatistical analyses in high-dimensional omics data are often hampered by the presence of batch effects (BEs) and missing values (MVs), but the interaction between these tw...
Handling Missing Data in COVID-19 Incidence Estimation: Secondary Data Analysis
Handling Missing Data in COVID-19 Incidence Estimation: Secondary Data Analysis
Abstract
Background
The COVID-19 pandemic has revealed significant challenges in disease forecasting and in developing a public health response, ...
Comparison of Methods for Imputing Social Network Data
Comparison of Methods for Imputing Social Network Data
Social network data often contain missing values because of the sensitive nature of the information collected and the dependency among the network actors. As a response, network im...
A framework for testing different imputation methods for tabular datasets
A framework for testing different imputation methods for tabular datasets
AbstractBackground and purposeHandling missing values is a prevalent challenge in the analysis of clinical data. The rise of data-driven models demands an efficient use of the avai...
Genetic Programming for Symbolic Regression on Incomplete Data
Genetic Programming for Symbolic Regression on Incomplete Data
<p><b>Symbolic regression is the process of constructing mathematical expressions that best fit given data sets, where a target variable is expressed in terms of input ...
Enhancing data integrity in Electronic Health Records: Review of methods for handling missing data
Enhancing data integrity in Electronic Health Records: Review of methods for handling missing data
AbstractIntroductionElectronic Health Records (EHRs) are vital repositories of patient information for medical research, but the prevalence of missing data presents an obstacle to ...

