NOTE: The following materials are presented for timely
dissemination of academic and technical work. Copyright and all other rights
therein are reserved by authors and/or other copyright holders. Persoanl
use of the following materials is permitted and, however, people using
the materials or information are expected to adhere to the terms and
constraints invoked by the related copyright.
Multi-Label Zero-Shot Human Action Recognition via Joint Latent Ranking Embedding
ABSTRACT
Human action recognition is one of the most challenging tasks in computer vision. Most of the existing works in human action
recognition are limited to single-label classification. A real-world video stream, however, often contains multiple human actions.
Such a video stream is usually annotated collectively with a set of relevant human action labels, which leads to a multi-label
learning problem. Furthermore, there are a great number of meaningful human actions in reality but it would be extremely difficult,
if not impossible, to collect/annotate sufficient video clips regarding all these human actions for training a supervised learning
model. In this paper, we formulate a real-world human action recognition task as a multi-label zero-shot learning problem. To
address this problem, a joint latent ranking embedding framework is proposed. Our framework holistically tackles the issue of
unknown temporal boundaries between different actions within a video clip for multi-label learning and exploits the side information
regarding the semantic relationship between different human actions for zero-shot learning. Specifically, our framework consists of
two component neural networks for visual and semantic embedding respectively. Thus, multi-label zero-shot recognition is done
by measuring relatedness scores of concerned action labels to a test video clip in the joint latent visual and semantic embedding
spaces. We evaluate our framework in different settings, including a novel data split scheme designed especially for evaluating
multi-label zero-shot learning. The experimental results on two weakly annotated multi-label human action datasets (i.e. Breakfast
and Charades) demonstrate the effectiveness of our framework.
Click
nn2019.pdf
for full text