Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • apa.csl
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Spark-Tuner: An elastic auto-tuner for apache spark streaming
University of Sidney, AUS.
Karlstad University, Faculty of Health, Science and Technology (starting 2013), Department of Mathematics and Computer Science (from 2013).ORCID iD: 0000-0001-9194-010X
University of Sidney, AUS.
RMIT University, AUS.
2020 (English)In: IEEE International Conference on Cloud Computing, CLOUD, IEEE Computer Society, 2020, p. 544-548Conference paper, Published paper (Refereed)
Abstract [en]

Spark has emerged as one of the most widely and successfully used data analytical engine for large-scale enterprise, mainly due to its unique characteristics that facilitate computations to be scaled out in a distributed environment. This paper deals with the performance degradation due to resource contention among collocated analytical applications with different priority and dissimilar intrinsic characteristics in a shared Spark platform. We propose an auto-tuning strategy of computing resources in a distributed Spark platform for handling scenarios in which submitted analytical applications have different quality of service (QoS) requirements (e.g., latency constraints), while the interference among computing resources is considered as a key performance-limiting parameter. We compared Spark-Tuner to two widely used resource allocation heuristics in a large scale Spark cluster through extensive experimental settings across several traffic patterns with uncertain rate and application types. Experimental results show that with Spark-Tuner, the Spark engine can decrease the $p$-99 latency of high priority applications by 43% during the high-rate traffic periods, while maintaining the same level of CPU throughput across a cluster.

Place, publisher, year, edition, pages
IEEE Computer Society, 2020. p. 544-548
Keywords [en]
Apache Spark Streaming Platform, Computer System Modeling and Profiling, Data Stream Processing Engine, Elastic Auto-Tuning
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kau:diva-83158DOI: 10.1109/CLOUD49709.2020.00082ISI: 000668554300074Scopus ID: 2-s2.0-85099334478ISBN: 9781728187808 (print)OAI: oai:DiVA.org:kau-83158DiVA, id: diva2:1530097
Conference
13th IEEE International Conference on Cloud Computing, CLOUD 2020, 18 October 2020 through 24 October 2020
Available from: 2021-02-21 Created: 2021-02-21 Last updated: 2021-09-07Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Taheri, Javid

Search in DiVA

By author/editor
Taheri, Javid
By organisation
Department of Mathematics and Computer Science (from 2013)
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 127 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • apa.csl
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf