A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters

Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. (2016) A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters. Concurrency and Computation: Practice and Experience, 29 (3). ISSN 1532-0626.

Full text not available from this repository.

Abstract

At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment.

This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.

Item Type: Article
Subjects: Q Science > Q Science (General)
Divisions: Faculty of Social and Applied Sciences > School of Law, Criminal Justice and Computing
Depositing User: Dr Man Qi
Date Deposited: 03 Apr 2018 10:35
Last Modified: 03 Apr 2018 10:35
URI: https://create.canterbury.ac.uk/id/eprint/17178

Actions (login required)

Update Item (CReaTE staff only) Update Item (CReaTE staff only)

Downloads

Downloads per month over past year

View more statistics

Share

Connect with us

Last edited: 29/06/2016 12:23:00