Graceful Performance Degradation in Apache Storm
2021 (English)In: Parallel and Distributed Computing, Applications and Technologies / [ed] Yong Zhang ; Yicheng Xu ; Hui Tian, Springer Science+Business Media B.V., 2021, p. 389-400Conference paper, Published paper (Refereed)
Abstract [en]
The concept of stream data processing is becoming challenging in most business sectors where try to improve their operational efficiency by deriving valuable information from unstructured, yet, contentiously generated high volume raw data in an expected time spans. A modern streamlined data processing platform is required to execute analytical pipelines over a continues flow of data-items that might arrive in a high rate. In most cases, the platform is also expected to dynamically adapt to dynamic characteristics of the incoming traffic rates and the ever-changing condition of underlying computational resources while fulfill the tight latency constraints imposed by the end-users. Apache Storm has emerged as an important open source technology for performing stream processing with very tight latency constraints over a cluster of computing nodes. To increase the overall resource utilization, however, the service provider might be tempted to use a consolidation strategy to pack as many applications as possible in a (cloud-centric) cluster with limited number of working nodes. However, collocated applications can negatively compete with each other, for obtaining the resource capacity in a shared platform that, in turn, the result may lead to a severe performance degradation among all running applications. The main objective of this work is to develop an elastic solution in a modern stream processing ecosystem, for addressing the shared resource contention problem among collocated applications. We propose a mechanism, based on design principles of Model Predictive Control theory, for coping with the extreme conditions in which the collocated analytical applications have different quality of service (QoS) levels while the shared-resource interference is considered as a key performance limiting parameter. Experimental results confirm that the proposed controller can successfully enhance the p -99 latency of high priority applications by 67%, compared to the default round robin resource allocation strategy in Storm, during the high traffic load, while maintaining the requested quality of service levels.
Place, publisher, year, edition, pages
Springer Science+Business Media B.V., 2021. p. 389-400
Series
Lecture Notes in Computer Science ; 12606
Keywords [en]
Apache storm streaming processing platform, Elastic resource controller, Performance modeling of computer system, Quality of Services (QoS), Cluster computing, Computation theory, Data streams, Model predictive control, Open systems, Pipeline processing systems, Storms, Analytical applications, Computational resources, Dynamic characteristics, Graceful performance degradations, Operational efficiencies, Performance degradation, Resource allocation strategies, Shared resource contentions, Quality of service
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kau:diva-84476DOI: 10.1007/978-3-030-69244-5_35Scopus ID: 2-s2.0-85104421760ISBN: 9783030692438 (print)OAI: oai:DiVA.org:kau-84476DiVA, id: diva2:1565478
Conference
International Conference on Parallel and Distributed Computing: Applications and Technologies PDCAT 2020 28 - 30 December 2020
2021-06-142021-06-142021-11-03Bibliographically approved