Javascript must be enabled to continue!

A Memory-Aware Spark Cache Replacement Strategy

Spark is currently the most widely used distributed computing framework, and its key data abstraction concept, Resilient Distributed Dataset (RDD), brings significant performance improvements in big data computing. In application scenarios, Spark jobs often need to replace RDDs due to insufficient memory. Spark uses the Least Recently Used (LRU) algorithm by default as the cache replacement strategy. This algorithm only considers the most recent use time of RDDs as the replacement basis. This characteristic may cause the RDDs that need to be reused to be evicted when performing cache replacement, resulting in a decrease in Spark performance. In response to the above problems, this paper proposes a memory-aware Spark cache replacement strategy, which comprehensively considers the cluster memory usage, RDD size, RDD dependencies, usage times and other information when performing cache replacement and selects the RDDs to be evicted. Furthermore, this paper designs extensive corresponding experiments to test and analyze the performance of the memory-aware Spark cache replacement strategy. The experimental data show that the proposed strategy can improve the performance by up to 13% compared with the LRU algorithm in different scenarios.

Journal of Internet Technology

Jingyu Zhang Jingyu Zhang Ruihan Zhang Jingyu Zhang Osama Alfarraj Ruihan Zhang Amr Tolba Osama Alfarraj Gwang-Jun Kim Amr Tolba

網際網路技術學刊

2022

Title: A Memory-Aware Spark Cache Replacement Strategy

Description:

In application scenarios, Spark jobs often need to replace RDDs due to insufficient memory.

Spark uses the Least Recently Used (LRU) algorithm by default as the cache replacement strategy.

This algorithm only considers the most recent use time of RDDs as the replacement basis.

This characteristic may cause the RDDs that need to be reused to be evicted when performing cache replacement, resulting in a decrease in Spark performance.

In response to the above problems, this paper proposes a memory-aware Spark cache replacement strategy, which comprehensively considers the cluster memory usage, RDD size, RDD dependencies, usage times and other information when performing cache replacement and selects the RDDs to be evicted.

Furthermore, this paper designs extensive corresponding experiments to test and analyze the performance of the memory-aware Spark cache replacement strategy.

The experimental data show that the proposed strategy can improve the performance by up to 13% compared with the LRU algorithm in different scenarios.

.

Back

While the CBEA (Cell Broadband Engine Architecture) offers substantial computational power, its explicit multilevel memory hierarchy poses significant challenges to traditional pro...

A Hierarchical Cache Architecture-Oriented Cache Management Scheme for Information-Centric Networking

Information-Centric Networking (ICN) typically utilizes DRAM (Dynamic Random Access Memory) to build in-network cache components due to its high data transfer rate and low latency....

Design and Optimization of 4-way set Associative Mapped Cache Controller

Abstract: In the realm of modern computer systems, the 4-way set associative mapped cache controller emerges as a cornerstone, revolutionizing memory access efficiency. This explor...

Concurrent Evaluation of Web Cache Replacement and Coherence Strategies

When studying Web cache replacement strategies, it is often assumed that documents are static. Such an assumption may not be realistic, especially when large-size caches are consid...

In-Memory Caching for Enhancing Subgraph Accessibility

Graphs have been utilized in various fields because of the development of social media and mobile devices. Various studies have also been conducted on caching techniques to reduce ...

C-Aware: A Cache Management Algorithm Considering Cache Media Access Characteristic in Cloud Computing

Data congestion and network delay are the important factors that affect performance of cloud computing systems. Using local disk of computing nodes as a cache can sometimes get bet...

Analytical characterization of cache replacement policy impact on content delivery time in information‐centric networks

SummaryInformation‐centric networking (ICN) has emerged as a promising candidate for designing content‐based future Internet paradigms. ICN increases the utilization of a network t...

Two novel cache management mechanisms on CPU-GPU heterogeneous processors

Heterogeneous multicore processors that take full advantage of CPUs and GPUs within the samechip raise an emerging challenge for sharing a series of on-chip ...

Email:
Password:

Email:

A Memory-Aware Spark Cache Replacement Strategy

Related Results