Predicting the performance of passage retrieval for question answering

Abstract

We present a novel approach to predicting the performance of passage retrieval for question answering. That is, estimating the effectiveness, for answer extraction, of a list of passages retrieved in response to a question when relevance judgments are not available. Our prediction model integrates two types of estimates. The first estimates the probability that the information need expressed by the question is satisfied by the passages. This estimate is devised by adapting query-performance predictors developed for the document retrieval task. The second type estimates the probability that the passages contain the answers. This estimate relies on the occurrences of named entities that are likely to answer the question. Empirical evaluation demonstrates the merits of our prediction approach. For example, the prediction quality is much better than that of the only previous prediction method devised for the task at hand. © 2012 ACM.