Admission control in YARN clusters based on dynamic resource reservation

Yi Yao; Jason Lin; Jiayin Wang; Ningfang Mi; Bo Sheng

doi:10.1109/INM.2015.7140389

Admission control in YARN clusters based on dynamic resource reservation

Yi Yao
, Jason Lin
, Jiayin Wang
, Ningfang Mi
, Bo Sheng

Computer Science

College of Engineering
University of Massachusetts Boston

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

Original language	English
Title of host publication	Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
Editors	Remi Badonnel, Jin Xiao, Shingo Ata, Filip De Turck, Voicu Groza, Carlos Raniery P. dos Santos
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	838-841
Number of pages	4
ISBN (Electronic)	9783901882760
DOIs	https://doi.org/10.1109/INM.2015.7140389
State	Published - Jun 29 2015
Event	14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 - Ottawa, Canada Duration: May 11 2015 → May 15 2015

Publication series

Name	Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

Conference

Conference	14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
Country/Territory	Canada
City	Ottawa
Period	5/11/15 → 5/15/15

ASJC Scopus Subject Areas

Software
Computer Science Applications
Computer Networks and Communications

Access to Document

10.1109/INM.2015.7140389

Cite this

Yao, Y., Lin, J., Wang, J., Mi, N., & Sheng, B. (2015). Admission control in YARN clusters based on dynamic resource reservation. In R. Badonnel, J. Xiao, S. Ata, F. De Turck, V. Groza, & C. R. P. dos Santos (Eds.), Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 (pp. 838-841). Article 7140389 (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INM.2015.7140389

Admission control in YARN clusters based on dynamic resource reservation. / Yao, Yi; Lin, Jason; Wang, Jiayin et al.
Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. ed. / Remi Badonnel; Jin Xiao; Shingo Ata; Filip De Turck; Voicu Groza; Carlos Raniery P. dos Santos. Institute of Electrical and Electronics Engineers Inc., 2015. p. 838-841 7140389 (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yao, Y, Lin, J, Wang, J, Mi, N & Sheng, B 2015, Admission control in YARN clusters based on dynamic resource reservation. in R Badonnel, J Xiao, S Ata, F De Turck, V Groza & CRP dos Santos (eds), Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015., 7140389, Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015, Institute of Electrical and Electronics Engineers Inc., pp. 838-841, 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015, Ottawa, Canada, 5/11/15. https://doi.org/10.1109/INM.2015.7140389

Yao Y, Lin J, Wang J, Mi N, Sheng B. Admission control in YARN clusters based on dynamic resource reservation. In Badonnel R, Xiao J, Ata S, De Turck F, Groza V, dos Santos CRP, editors, Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 838-841. 7140389. (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015). doi: 10.1109/INM.2015.7140389

Yao, Yi ; Lin, Jason ; Wang, Jiayin et al. / Admission control in YARN clusters based on dynamic resource reservation. Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. editor / Remi Badonnel ; Jin Xiao ; Shingo Ata ; Filip De Turck ; Voicu Groza ; Carlos Raniery P. dos Santos. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 838-841 (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015).

@inproceedings{8be8d21b00104a548319d7e978babb0b,

title = "Admission control in YARN clusters based on dynamic resource reservation",

abstract = "Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.",

author = "Yi Yao and Jason Lin and Jiayin Wang and Ningfang Mi and Bo Sheng",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 ; Conference date: 11-05-2015 Through 15-05-2015",

year = "2015",

month = jun,

day = "29",

doi = "10.1109/INM.2015.7140389",

language = "English",

series = "Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "838--841",

editor = "Remi Badonnel and Jin Xiao and Shingo Ata and \{De Turck\}, Filip and Voicu Groza and \{dos Santos\}, \{Carlos Raniery P.\}",

booktitle = "Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015",

}

TY - GEN

T1 - Admission control in YARN clusters based on dynamic resource reservation

AU - Yao, Yi

AU - Lin, Jason

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

PY - 2015/6/29

Y1 - 2015/6/29

N2 - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

AB - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

UR - https://www.scopus.com/pages/publications/84942579093

UR - https://www.scopus.com/pages/publications/84942579093#tab=citedBy

U2 - 10.1109/INM.2015.7140389

DO - 10.1109/INM.2015.7140389

M3 - Conference contribution

AN - SCOPUS:84942579093

T3 - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

SP - 838

EP - 841

BT - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

A2 - Badonnel, Remi

A2 - Xiao, Jin

A2 - Ata, Shingo

A2 - De Turck, Filip

A2 - Groza, Voicu

A2 - dos Santos, Carlos Raniery P.

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

Y2 - 11 May 2015 through 15 May 2015

ER -

Admission control in YARN clusters based on dynamic resource reservation

Abstract

Publication series

Conference

ASJC Scopus Subject Areas

Access to Document

Other files and links

Fingerprint

Cite this