A gossip-based reliable multicast for large-scale high-throughput applications

Abstract

Group-based reliable multicast is an important building block for distributed applications. For large systems, however, traditional approaches do not scale well due to centralized recovery mechanisms and excessive message overhead. In this paper, we present a reliable probabilistic multicast, rpbcast, that is a hybrid of the centralized and gossip-based approaches. In particular, rpbcast extends previous work by supporting high packet rates and many active senders. Rpbcast uses gossip as the primary re-transmission mechanism and only contacts loggers if gossips fail. Large groups of active senders are supported using negative gossip that specifies those messages a receiver is missing instead of those messages it received. Moreover, we show that negative gossip allows pull based recovery and converges faster than push based recovery. Rpbcast also applies hashing techniques to reduce message overhead and approximate group membership for garbage collection. We describe the key features of rpbcast and present simulation results.