A stratified approach for supporting high throughput event processing applications

Abstract

The quantity of events that a single application needs to process is constantly increasing. RFID related events have doubled within the past year and reached 4 trillion events per day, financial applications in large banks are processing 400 million events per day, and Massively Multiplayer Online (MMO) games are monitoring millions of events per second during peak periods. It is evident that scalability in event throughput is a major requirement for such applications. While the first generation of event processing systems is centralized, we see various solutions that attempt to use both scale-up and scale-out techniques. Alas, partitioning of the processing manually is difficult due to the semantic dependencies among event processing agents. It is also difficult to manually tune up the partition dynamically. This paper proposes a horizontal partition that is automatically created by analyzing the semantic dependencies among agents using a stratification principle. Each stratum contains a collection of independent agents, and events are routed to subsequent strata. We also implement a profiling-based technique for assigning agents to nodes in each stratum with the goal of maximizing throughput. A complementary step is to distribute load among different execution nodes dynamically based on their performance characteristics and the event traffic model. Experimental results show significant improvement in the ability to process high throughput of events relative to both centralized solutions as well as vertical partitions. We find this to be a promising approach to achieve high scalability particularly when the traffic model and network topology change frequently.