TY - GEN
T1 - ATuMm
T2 - 38th IEEE International Performance Computing and Communications Conference, IPCCC 2019
AU - Jia, Danlin
AU - Bhimani, Janki
AU - Nguyen, Son Nam
AU - Sheng, Bo
AU - Mi, Ningfang
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/10
Y1 - 2019/10
N2 - Apache Spark is an in-memory analytic framework that has been adopted in the industry and research fields. Two memory managers, Static and Unified, are available in Spark to allocate memory for caching Resilient Distributed Datasets (RDDs) and executing tasks. However, we found that the static memory manager (SMM) lacks flexibility, while the unified memory manager (UMM) puts heavy pressure on the garbage collection of JVM on which Spark resides. To address these issues, we design an auto-tuning memory manager (ATuMm) to support dynamic memory allocation with the consideration of both memory demands and latency introduced by garbage collection. We implement our new memory manager in Spark 2.2.0 and evaluate it by conducting experiments in a real Spark cluster. Our experimental results show that our auto-tuning memory manager can reduce the total garbage collection time and thus further improve the performance (i.e., reduced latency) of Spark applications, compared to the existing Spark memory management solutions.
AB - Apache Spark is an in-memory analytic framework that has been adopted in the industry and research fields. Two memory managers, Static and Unified, are available in Spark to allocate memory for caching Resilient Distributed Datasets (RDDs) and executing tasks. However, we found that the static memory manager (SMM) lacks flexibility, while the unified memory manager (UMM) puts heavy pressure on the garbage collection of JVM on which Spark resides. To address these issues, we design an auto-tuning memory manager (ATuMm) to support dynamic memory allocation with the consideration of both memory demands and latency introduced by garbage collection. We implement our new memory manager in Spark 2.2.0 and evaluate it by conducting experiments in a real Spark cluster. Our experimental results show that our auto-tuning memory manager can reduce the total garbage collection time and thus further improve the performance (i.e., reduced latency) of Spark applications, compared to the existing Spark memory management solutions.
UR - https://www.scopus.com/pages/publications/85079102105
UR - https://www.scopus.com/pages/publications/85079102105#tab=citedBy
U2 - 10.1109/IPCCC47392.2019.8958724
DO - 10.1109/IPCCC47392.2019.8958724
M3 - Conference contribution
AN - SCOPUS:85079102105
T3 - 2019 IEEE 38th International Performance Computing and Communications Conference, IPCCC 2019
BT - 2019 IEEE 38th International Performance Computing and Communications Conference, IPCCC 2019
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 29 October 2019 through 31 October 2019
ER -