Extracting semantics from audiovisual content: The final frontier in multimedia retrieval

Abstract

Multimedia understanding is a fast emerging interdisciplinary research area. There is tremendous potential for effective use of multimedia content through intelligent analysis. Diverse application areas are increasingly relying on multimedia understanding systems. Advances in multimedia understanding are related directly to advances in signal processing, computer vision, pattern recognition, multimedia databases, and smart sensors. We review the state-of-the-art techniques in multimedia retrieval. In particular we discuss how multimedia retrieval can be viewed as a pattern recognition problem. We discuss, how reliance on powerful pattern recognition and machine learning techniques is increasing in the field of multimedia retrieval. We review state-of-the-art multimedia understanding systems with particular emphasis on a system for semantic video indexing centered around multijects and multinets. We discuss how semantic retrieval is centered around concepts and context and also discuss various mechanisms for modeling concepts and context.