An effective algorithm for parallelizing hash joins in the presence of data skew

Abstract

A parallel hash join algorithm based on the concept of hierarchical hashing is proposed to address the problem of data skew. The proposed algorithm adds an extra scheduling phase to the usual hash and join phases. During the scheduling phase, a heuristic optimization algorithm, using the output of the hash phase, attempts to balance the load across the multiple processors in the subsequent join phase. The algorithm naturally identifies the hash partitions with the largest skew elements, splits them up, and assigns each of them to an optimal number of processors. Assuming a Zipf-like distribution of the elements in the join column, the algorithm is shown to achieve good load balancing for the join phase in a CPU-bound environment, and it is shown to be fairly robust relative to the degree of data skew and the total number of processors. The overall speedup due to this algorithm is compared with conventional parallel hash join methods and the considerable advantage of the proposed method is demonstrated even when the additional scheduling phase is taken into account.