Search engine for discovering works of Art, research articles, and books related to Art and Culture
Javascript must be enabled to continue!
Hadoop Distributed File System for Big data analysis
View through CrossRef
Abstract
Hadoop is framework that is processing data with large volume that cannot be processed by conventional systems. Hadoop has management file system called Hadoop Distributed File System (HDFS) that has NameNode and DataNode where the data is divided into blocks based on the total size of dataset. In addition, Hadoop has MapReduce where the dataset is processed in Mapping phase and then reducing phase. Using Hadoop for big data analysis has been revealed important information that can be used for analytical purpose and enabling new products. Big data could be found in many different resources such as social networks, web server logs, broadcast audio streams and banking transactions. In this paper, we illustrated the main steps to setup Hadoop and MapReduce. The illustrated version in this work is the latest released of Hadoop 3.1.1 for big data analysis. A simplified pseudo code is provided to show the functionality of Map class and reduce class. The developed steps are applied with a given example that could be generalized with bigger data.
Title: Hadoop Distributed File System for Big data analysis
Description:
Abstract
Hadoop is framework that is processing data with large volume that cannot be processed by conventional systems.
Hadoop has management file system called Hadoop Distributed File System (HDFS) that has NameNode and DataNode where the data is divided into blocks based on the total size of dataset.
In addition, Hadoop has MapReduce where the dataset is processed in Mapping phase and then reducing phase.
Using Hadoop for big data analysis has been revealed important information that can be used for analytical purpose and enabling new products.
Big data could be found in many different resources such as social networks, web server logs, broadcast audio streams and banking transactions.
In this paper, we illustrated the main steps to setup Hadoop and MapReduce.
The illustrated version in this work is the latest released of Hadoop 3.
1.
1 for big data analysis.
A simplified pseudo code is provided to show the functionality of Map class and reduce class.
The developed steps are applied with a given example that could be generalized with bigger data.
Related Results
Enhancing Big Data Security in Hadoop using Machine Learning
Enhancing Big Data Security in Hadoop using Machine Learning
In the era of Big Data, where vast amounts of information are generated and analysed to extract valuable insights, ensuring the security of data has become paramount. Hadoop, as a ...
Secure Cloud Data with Attribute-based Honey Encryption
Secure Cloud Data with Attribute-based Honey Encryption
Abstract
Encryption is a Technique to convert plain text into Cipher text, which is unreadable without an appropriate decryption key. Hadoop is a platform to process and st...
Creating Secure File Systems in Open-Source Operating Systems
Creating Secure File Systems in Open-Source Operating Systems
The relevance of this study is determined by insecure data storage on personal computers, as it is the main operating system that performs authentication and file access control. B...
Compressive structural bioinformatics
Compressive structural bioinformatics
We are developing compressed 3D molecular data representations and workflows (“Compressive Structural Bioinformatics”) to speed up mining and visualization of 3D structural data by...
Compressive structural bioinformatics
Compressive structural bioinformatics
We are developing compressed 3D molecular data representations and workflows (“Compressive Structural Bioinformatics”) to speed up mining and visualization of 3D structural data by...
Hadoop Tools
Hadoop Tools
As the name indicates, this chapter explains the various additional tools provided by Hadoop. The additional tools provided by Hadoop distribution are Hadoop Streaming, Hadoop Arch...
YouTube: big data analytics using Hadoop and map reduce
YouTube: big data analytics using Hadoop and map reduce
We live today in a digital world a tremendous amount of data is generated by each digital service we use. This vast amount of data generated is called Big Data. According to Wikipe...