On complementing end-to-end human behavior predictors with planning

High capacity end-to-end approaches for human motion (behavior) prediction have the ability to represent subtle nuances in human behavior; but struggle with robustness to out of distribution inputs and tail events. Planning-based prediction; on the other hand; can reliably output decent-but-not-great predictions: it is much more stable in the face of distribution shift (as we verify in this work); but it has high inductive bias; missing important aspects that drive human decisions; and ignoring cognitive biases that make human behavior suboptimal. In this work; we analyze one family of approaches that strive to get the best of both worlds: use the end-to-end predictor on common cases; but do not rely on it for tail events / out-of-distribution inputs — switch to the planning-based predictor there. We contribute an analysis of different approaches for detecting when to make this switch; using an autonomous driving domain. We find that promising approaches based on ensembling or generative modeling of the training distribution might not be reliable; but that there very simple methods which can perform surprisingly well — including training a classifier to pick up on tell-tale issues in predicted trajectories.