Queueing theory based modeling and optimal scheduling in Map-Reduce-like Frameworks
FORK-JOIN QUEUE MODELING AND OPTIMAL SCHEDULING IN PARALLEL PROGRAMMING FRAMEWORKS
ABSTRACTMapReduce framework is widely used to parallelize batch jobs since it exploits a high degree of multi-tasking to process them. However, it has been observed that when the number of servers increases, the map phase can take much longer than expected. This thesis analytically shows that the stochastic behavior of the servers has a negative effect on the completion time of a MapReduce job, and continuously increasing the number of servers without accurate scheduling can degrade the overall performance. We analytically model the map phase in terms of hardware, system, and application parameters to capture the effects of stragglers on the performance. Mean sojourn time (MST), the time needed to sync the completed tasks at a reducer, is introduced as a performance metric and mathematically formulated. Following that, we stochastically investigate the optimal task scheduling which leads to an equilibrium property in a datacenter with different types of servers. Our experimental results show the performance of the different types of schedulers targeting MapReduce applications. We also show that, in the case of mixed deterministic and stochastic schedulers, there is an optimal scheduler that can always achieve the lowest MST.
KEYWORDS
ᐧ
ABSTRACTMapReduce framework is widely used to parallelize batch jobs since it exploits a high degree of multi-tasking to process them. However, it has been observed that when the number of servers increases, the map phase can take much longer than expected. This thesis analytically shows that the stochastic behavior of the servers has a negative effect on the completion time of a MapReduce job, and continuously increasing the number of servers without accurate scheduling can degrade the overall performance. We analytically model the map phase in terms of hardware, system, and application parameters to capture the effects of stragglers on the performance. Mean sojourn time (MST), the time needed to sync the completed tasks at a reducer, is introduced as a performance metric and mathematically formulated. Following that, we stochastically investigate the optimal task scheduling which leads to an equilibrium property in a datacenter with different types of servers. Our experimental results show the performance of the different types of schedulers targeting MapReduce applications. We also show that, in the case of mixed deterministic and stochastic schedulers, there is an optimal scheduler that can always achieve the lowest MST.
KEYWORDS
Cloud computing, Computational modeling, Delays, Map-Reduce, Mapper, MapReduce, Mean sojourn time, Optimal scheduling, Optimization, Performance Evaluation, Queuing Theory, Reducer, Servers, Stochastic Modeling, Stochastic processes, Stragglers, Synchronizatio
Farshid Farhat 's Site @ PSU
Farshid Farhat 's Site @ PSU
Publications
- Detecting Dominant Vanishing Points in Natural Scenes with Application to Composition-Sensitive Image Retrieval, Z Zhou, F Farhat, JZ Wang, IEEE Transactions on Multimedia, 2017. [code][dataset]
- Shape matching using skeleton context for automated bow echo detection, MM Kamani, F Farhat, S Wistar, JZ Wang, Big Data (Big Data), 2016 IEEE International Conference on, 901-908, 2016.
- Detecting Vanishing Points in Natural Scenes with Application in Photo Composition Analysis, Z Zhou, F Farhat, JZ Wang, arXiv preprint arXiv:1608.04267, 2016.
- Optimal Placement of Cores, Caches and Memory Controllers in Network On-Chip, DZ Tootaghaj, F Farhat, arXiv preprint arXiv:1607.04298, 2016.
- Towards Stochastically Optimizing Data Computing Flows, F Farhat, DZ Tootaghaj, M Arjomand, arXiv preprint arXiv:1607.04334, 2016.
- Stochastic Modeling and Optimization of Stragglers, F Farhat, D Zad Tootaghaj, Y He, et al. IEEE Transaction on Cloud Computing, 14, 6, 2016.
- Stochastic modeling and optimization of stragglers in mapreduce framework, F Farhat, The Pennsylvania State University, 2015.
- Evaluating the Combined Impact of Node Architecture and Cloud Workload Characteristics on Network Traffic and Performance/Cost, DZ Tootaghaj, F Farhat, M Arjomand, P Faraboschi et al. Workload characterization (IISWC), 2015 IEEE international symposium on, 203-212, 2015.
- Towards blind detection of low-rate spatial embedding in image steganalysis, F Farhat, S Ghaemmaghami, IET Image Processing 9 (1), 31-42, 2015.
- Modeling and optimization of straggling mappers, F Farhat, DZ Tootaghaj, A Sivasubramaniam, MT Kandemir, CR Das, Technical report, Technical Report CSE-14-006, Pennsylvania State University, 2014.
- Security Weaknesses in PGP Protocol, F Farhat, M Heydari, MR Aref, 2013.
- Eigenvalues-based LSB steganalysis, F Farhat, A Diyanat, S Ghaemmaghami, MR Aref, The ISC International Journal of Information Security 4 (2), 97-106, 2012.
- Image steganalysis based on SVD and noise estimation: Improve sensitivity to spatial LSB embedding families, A Diyanat, F Farhat, S Ghaemmaghami, TENCON 2011-2011 IEEE Region 10 Conference, 1266-1270, 2011.
- Multi-dimensional correlation steganalysis, F Farhat, A Diyanat, S Ghaemmaghami, MR Aref, Multimedia Signal Processing (MMSP), 2011 IEEE 13th International Workshop, 2011.
- Game-theoretic approach to mitigate packet dropping in wireless Ad-hoc networks, DZ Tootaghaj, F Farhat, MR Pakravan, MR Aref, Consumer Communications and Networking Conference (CCNC), 2011 IEEE, 163-165, 2011.
- Risk of attack coefficient effect on availability of Ad-hoc networks, DZ Tootaghaj, F Farhat, MR Pakravan, MR Aref, Consumer Communications and Networking Conference (CCNC), 2011 IEEE, 166-168, 2011.
- Private Identification, Authentication and Key Agreement Protocol with Security Mode Setup, F Farhat, S Salimi, A Salahi, IACR Cryptology ePrint Archive, 2011.
- Locally Multipath Adaptive Routing Protocol Resilient to Selfishness and Wormholes, F Farhat, MR Pakravan, M Salmasizadeh, MR Aref, International Conference on Information Security Practice and Experience, 2010.
- Extended Authentication and Key Agreement Protocol of UMTS, F Farhat, S Salimi, A Salahi, International Conference on Information Security Practice and Experience, 2009.
MAX AMINI @ Penn State
ISA is proud to have the popular and super comedian Max Amini here at Penn State again on March 18th. Get your tickets at Max’s website at “www.maxamini.com” before they are sold out!
Diman Zad Tootaghaj 's publications
- 1.D. Z. Tootaghaj, N. Bartolini, H. Khamfroush and T. La Porta “Controlling Cascading Failures in Interdependent Networks under Uncertain Knowledge of Damages”. Submitted to SRDS 2017.
-
3.D. Z. Tootaghaj, H. Khamfroush, N. Bartolini, S. Ciavarella, S. Hayes and T. La Porta, “Network Recovery under Uncertainty”, in INSR industry day 2017. [file]
-
9.Diman Zad Tootaghaj, Farshid Farhat, Mohammad Arjomand, Paolo Faraboschi, Mahmut Taylan Kandemir, Anand Sivasubramaniam, Chita R. Das, Evaluating the Combined Impact of Datacenter Architecture and Cloud Workload Characteristics on Performance, Network Traffic and Cost, IEEE International Symposium on Workload Characterization (IISWC) 2015. [link] [pdf]
-
13. Mohammad Amin Kashiha, Diman Zad Tootaghaj, Dolat Djamshidi, Partial Discharge Source Classification and De-noising in Rotating Machines Using Discrete Wavelet Transform and Directional Coupling Capacitor, Journal of Electromagnetic Analysis and Applications and submitted and accepted to Asia-Pacific Power and Engineering Conference (APPEEC), sponsored by IEEE Power & Energy Society (PES), March 2009. [link] [pdf]
Electrical Engineering PhD, System Communications, Sharif University of Technology
- شماره پایان نامه:45945
- کد دانشکده:05
- پديدآور:فرحت، فرشيد
- عنوان:تحليل پنهان سازي تصوير با نرخ درج پايين در حوزه مكان.
- نام دانشگاه/پژوهشگاه:صنعتي شريف
- سال اخذ مدرك:1392.
- نام دانشکده:مهندسي برق
- مقطع:دكتري
- گرایش:مخابرات سيستم
- توصیف ظاهری:89ص.: جدول، نمودار، كتابنامه؛ چكيده به فارسي و انگليسي
- توصیفگر:نهان نگاري تصويرImage Watermarking
- توصیفگر:تحليل نهان نگاريSteganalysis
- توصیفگر:تجزيه مقادير ويژهEigenvalue Decomposition
- توصیفگر:جاسازي كم ارزش ترين بيتLeast Significant Bit (LSB)Embedding
- توصیفگر:خودهمبستگي نسبيRelative Autocorrelation
- توصیفگر:مجموع كمترين جزء هاParts Min Sum
- توصیفگر:تخمين نرخ جاسازيEmbedding Rate Estimation
- توصیفگر:مجموع درگاشت هاي محليLocal Entropies Sum
- استاد راهنما.قائم مقامي، شاهرخ
- استاد مشاور.عارف، محمد رضا
ارتقاء امنيت پروتكل هاي مسيريابي در شبكه هاي اقتضايي (فرشيد فرحت)
- شماره پایان نامه:37533
- کد دانشکده:05
- پديدآور:فرحت، فرشيد.
- عنوان:ارتقاء امنيت پروتكل هاي مسيريابي در شبكه هاي اقتضايي.
- نام دانشگاه/پژوهشگاه:صنعتي شريف
- سال اخذ مدرك:1386
- نام دانشکده:مهندسي برق
- گرایش:كارشناسي ارشد (مخابرات رمز)
- یادداشت:100ص0: جدول، نمودار، كتابنامه؛ چكيده به فارسي و انگليسي
- توصیفگر:رمزنگاريCRYPTOGRAPHY
- توصیفگر:امنيتSecurity
- توصیفگر:شبكه خودجوشAd Hoc Network
- توصیفگر:قرارداد مسيريابيRouting Protocol
- توصیفگر:الگوريتم رمز قالبي كليد متقارنSymmetric-Key Block Cipher
- توصیفگر:حمله لانه كرميWormhole Attack
- توصیفگر:شبيه ساز MANETMANET Simulator
- استاد راهنما.عارف، محمد رضا،
- استاد راهنما.سلماسي زاده، محمود،
Subscribe to:
Posts (Atom)
Your Facebook fundraiser refund receipt
Hi xxx, We wanted to let you know that your donation of $xx.xx to the fundraiser Raise to support the victims of earthquake in Iran was ...
-
62540818_BS | RUHANI .ARSALAN | RUHAANI@CE.SHARIF.AC.IR <mailto:RUHAANI@CE.SHARIF.AC.IR> 65265109_BS | SANOORI .FARIDEH | SANOORIE@CE...
-
Arash - Tike Tike Kardi (Translation) Farsi -> English Arash - Tike Tike Kardi (=you do slice by slice=you cut into slices) آرش - تیکه ت...
-
Rapidshare Tips & Tricks (rapidshare.de) | Megaupload Account This article has tips on how to beat Rapidshare time limit, get a free...
