Scheduling heterogeneous MapReduce jobs for efficiency improvement in enterprise clusters

The MapReduce paradigm and its open source implementation Hadoop are emerging as an important standard for large-scale data-intensive processing in both industry and academia. A MapReduce cluster is typically shared among multiple users with different types of workloads. When a flock of jobs are concurrently submitted to a MapReduce cluster, they compete for the shared resources and the overall system performance might be seriously degraded. Therefore, one challenging issue is to efficiently schedule all the jobs in such a shared MapReduce environment. However, we find that prior scheduling algorithms supported by Hadoop cannot guarantee good performance under different workloads. In this paper, we propose a new Hadoop scheduler, which leverages the knowledge of workload patterns to improve the system performance by dynamically tuning the resource shares among users and the scheduling algorithms for each user. Experimental results from Amazon EC2 cluster show that our scheduler reduces the average MapReduce job response times under a variety of workloads compared to the existing FIFO and Fair schedulers.