Comparison of Resource Management Frameworks for Processing Big Data.

Sanam Ahmad; Muhammad Murtaza Yousaf; Syed Waqar Jaffry; Shahzad Sarwar; Laeeq Aslam

Author(s)

Sanam Ahmad Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

Muhammad Murtaza Yousaf Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

Syed Waqar Jaffry Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

Shahzad Sarwar Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

Laeeq Aslam Punjab University College of Information Technology, University of the Punjab, Lahore, Pakistan

Abstract

- Big data analytic solutions are usually deployed on computer-cluster or cloud computing environment. Efficient resource management is a challenging task in a cluster or cloud environment. Big data processing frameworks hide the details of resource management from the end user. In this paper, we have described the traditional Hadoop and state-of-the-art resource management frameworks used for big data processing. For the evaluation of resource management framework we have identified a feature vector comprising resource assignment, scheduling, security, and physical systemlevel resources. Using the feature vector, resource management frameworks have been compared in order to identify strengths and weaknesses of each. Identified weaknesses indicate the future research directions for further improvement of respective framework. It has been observed that YARN framework qualifies for the most features in the feature vector.

Publication Details

Page(s) 4-8

DOI DOI not available

Published Journal: Bahria University Journal of Information & Communication Technologies, Volume: 9, Issue: 2, Year: 2016

Request the full text from PASTIC

References

JeffreyDean,SanjayGhemawat, 2008.,Communications of the ACM 51 107 -113

BenjaminHindman,Anthony D.Joseph,Randy H. Katz , Scott,Shenker, August 2011.Mesos: Flexible Resource Sharing for the Cloud”,proceedings of USENIX -

Notes by Facebook engineering, “Under the Hood: Scheduling MapReduce jobs more efficiently with Corona”, November 9, 2012 online: http://www.facebook.com/notes/facebook-engineering/under-the-hood-scheduling-mapreduce-jobs-more-efficiently-with-corona/10151142560538920

BikashSharma,Seung-HwanLim,Chita R. Das, 2012.MROrchestrator: A Fine-Grained Resource Orchestration Framework for MapReduce Clusters”,Proceedings of the 2012 IEEE Fifth International Conference on Cloud Computing 1 -8

LeonardoNeumeyer, 2010.S4: Distributed Stream Computing Platform”,Proceedings of the 2010 IEEE International Conference on Data Mining Workshops 170 -177

VinodKumar,Vavilapalli , Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley,SantaClara, 2013.Apache Hadoop YARN: yet another resource negotiator”,Proceedings of the 4th annual Symposium on Cloud Computing, October 01-03 -

PatrickHunt,Flavio P.Junqueira,BenjaminReed, 2010."ZooKeeper: wait-free coordination for internet-scale systems",USENIXATC'10: Proceedings of the 2010 USENIX conference on USENIX annual technical conference, USENIX Association -

http://container-solutions.com/the-rise-of-apache-mesos/.

http://developer.yahoo.com/blogs/ydn/hadoop-yahoo-more-ever-54421.html

., -

JordàPolo,YolandaBecerra, 2010.Performance-driven task coscheduling for MapReduce environments”,In 12th IEEE/IFIP Network Operations and Management Symposium. ACM -

AnQin,and ChangGao, 2009."XConveryer: Guarantee Hadoop throughput via lightweight OS-level virtualization, ",In Proceedings of 8th International Conference on Grid and Cooperative Computing 299 -304

Hadoop, ., -

Hadoop, r0.:,FairScheduler: -

ThomasSandholm, 2010.Dynamic proportional share scheduling in Hadoop”,Proceedings of the 15th international conference on Job scheduling strategies for parallel processing 110 -131

KhaledElmeleegy, 2009.”Job Scheduling for Multi-User MapReduce Clusters”,Technical Report No. UCB/EECS-2009-55, April 30 -