Analyzing the cost of a cache miss using pipeline spectroscopy

Abstract

We describe a new technique called Pipeline Spectroscopy that allows us to precisely measure the cost of each cache miss. The cost of a miss is displayed (graphed) as a histogram, which represents a precise readout showing a detailed visualization of the cost of each cache miss throughout all levels of the memory hierarchy. We call the graphs 'spectrograms' because they reveal certain signature characteristics of the processor's memory hierarchy, the pipeline, and the miss pattern itself. We show that in a memory hierarchy with N cache levels (L1, L2, ..., LN, and memory) and a miss cluster of size C, there are (C N+C) possible miss penalties. This represent all possible sums from all possible combinations of the miss latencies with and without overlap from each level of the memory hierarchy (L2, L3, ... Memory) for a given cluster size. Additionally, a theory is presented that describes the shape of a spectrogram, and we use this theory to predict the shape of spectrograms for larger miss clusters. Next we provide to examples using spectroscopy to optimize the processor's hardware or application's software. The first example uses a miss spectrogram to improve the software design of an application. The second example uses a miss spectrogram to analyze bus queuing. Our experiments show that performance gains of up to 8% are possible. Detailed analysis of a spectrograph leads to much greater hsight in pipeline dynamics, including effects due to prefetching, and miss queuing delays.