Holistic Representation Learning for Multitask Trajectory Anomaly Detection

Video anomaly detection deals with the recognition of abnormal events in videos. Apart from the visual signal, video anomaly detection has also been addressed with skeleton sequences. We propose a holistic representation of skeleton trajectories to learn expected motions across segments at different times. Our approach uses multitask learning to reconstruct any continuous unobserved temporal segment of the trajectory allowing the extrapolation of past and future segments and the interpolation of in-between segments. We use an end-to-end attention-based encoder-decoder to encode temporally occluded trajectories, jointly learn latent representations of the occluded trajectory segments, and reconstruct trajectories of expected motions across different temporal segments. Extensive experiments over three trajectory-based video anomaly detection datasets show the advantages and effectiveness of our method with state-of-the-art results on the detection of anomalies in skeleton trajectories