A Review of the Parallelization Strategies for Iterative Algorithms

Abstract Iteration-based algorithms have been widely used and achieved excellent results in many fields. However, in the big data era, data that needs to be processed is enormous in terms of both depth (the dimensionality of data) and breadth (the volume of data). Due to the slowdown of Moore's Law, the computing power of single-core CPUs is becoming saturated. The increase in the computational complexity and the bottleneck of the single-core processor’s speed exacerbate the time-consuming problem of iterative algorithms. With the rise of multi-core computers and distributed computing systems, parallelizing and deploying iterative algorithms on such systems can make full use of computing resources and accelerate iterative computation, providing a new idea for solving the aforementioned problems. However, due to the logical dependency between two consecutive iterations in an iterative algorithm, it is difficult to directly implement the concurrent computation of such algorithms. To this end, many studies have been conducted on the parallelization of iterative algorithms in both academia and industry. This paper aims to conduct an in-depth research and analysis of these parallelization strategies. Firstly, the abstract description and classification of iterative algorithms are given. Then four concurrency strategies for iterative algorithms are summarized, including logical units that can be intrinsically concurrently computed, multi-initial state parallel search strategy, data parallelism, and task parallelism. Finally, the paper detailed the convergence of parallel iterative algorithms, focusing on building the mathematical model of asynchronous iterative algorithms, and summarizing the convergence conditions of asynchronous iterative algorithms.