Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation

Control of nonlinear systems with unknown dynamics is a major challenge on the road to fully autonomous agents. Current trends in reinforcement learning (RL) focus on complex representations of dynamics and policies. Such approaches have yielded impressive results in solving a variety of hard control tasks. However, this new sophistication has come with the cost of an overall reduction in our ability to interpret the resulting policies from a classical perspective, and the need for extremely over-parameterized controllers. In this paper, we take inspiration from the control community and apply the principles of hybrid switching systems, in order to break down complex representations into simpler components. We exploit the rich representational power of probabilistic graphical models and derive a new expectation-maximization (EM) algorithm for learning a generative model and automatically decomposing nonlinear dynamics into stochastic switching linear dynamical systems. Moreover, we show how this framework of probabilistic switching models enables extracting hierarchies of Markovian and auto-regressive locally linear controllers from nonlinear experts in an imitation learning scenario.