Streamsight - A visualization tool for large-scale streaming applications

Abstract

Stream processing is becoming a new and important computing paradigm. Innovative streaming applications are being developed in areas ranging from scientific applications (e.g., environment monitoring), to business intelligence (e.g., fraud detection and trend analysis), to financial markets (e.g., algorithmic trading strategies). Developing, understanding, debugging, and optimizing streaming applications is non-trivial because of the adaptive and dynamic nature of these applications. The sheer complexity and the distributed character of a large number of cooperating components hosted on a distributed environment further complicate matters. In this paper we describe Streamsight, a new visualization tool built to examine, monitor, and help understand the dynamic behavior of streaming applications. Previously developed stream processing visualization tools focus solely on composition of dataflow graphs. Streamsight's novelty hinges on a wide range of capabilities, including the ability to manage the dynamics of large and evolving topologies comprising multiple streaming applications with thousands of nodes and interconnections. From rendering live performance counters using different perspectives to allowing recordings and replays of the execution process, Streamsight provides the mechanisms that permit a better understanding of the evolving and adaptive behavior of streaming applications. These capabilities are used for debugging purposes, for performance optimization, and management of resources, including capacity planning. More than 50 developers, both inside and outside IBM, have been using Streamsight. © 2008 ACM.