Automated Scene Understanding

Automatic extraction and representation of visual concepts and semantic information in scenes is a desired capability in surveillance operations. In this effort we will advance the foundations of data representation and fusion at various levels of abstraction. We target the problem of complex event recognition in network information environment, where lack of effective visual processing tools and incomplete domain knowledge frequently cause uncertainty in the datasets and consequently, in the visual primitives extracted from it. We employ Markov Logic Network (MLN) to address the task of reasoning under uncertainty. In Phase I, we demonstrated use of MLN as a domain knowledge representation language that can be used for inferring complex events in real world. In Phase II, our emphasis will be on developing algorithms to fuse data from multiple sources, perform reasoning in the presence of incomplete data, and transfer learning for domain adaptation. At the visual processing level, transfer learning will enable zero-shot recognition of unknown classes. At the decision level, transfer learning is applied to MLN to automatically infer rules for related but unseen domains. Technical claims made during the study will be justified using rigorous testing and comparison with other state-of-the-art methods on publically available datasets.