Delay asymptotics for heavy-tailed MapReduce jobs

Abstract

A MapReduce job consists of two phases that are processed in a map queue and a redeuce queue, respectively. The map queue is characterized by the processor sharing discpline, and the reduce queue by a multi-server station. A reduce task is composed of two sequential steps: the copy/shuffle step and the reduce function step. A synchronization barrier between the map and reduce phases complicates the process: the copy/shuffle step can overlap with the map phase, but its finish point and the start of the reduce function step have to be strictly after the completion of the map phase of the same job. This dependency can result in an interesting criticality phenomenon for the job delay distribution in MapReduce scheduling. We refine the logarithmic asymptotics that has been established for heavy-tailed MapReduce jobs by studying the exact asymptotics. The analysis reveals that the MapReduce framework combines the features of both processor sharing and first in first out disciplines. © 2012 IEEE.