Contrastive Inductive Bias Controlling Networks for Reinforcement Learning

Effective learning in an visual-based environment is essential for reinforcement learning (RL) agent, while it has been empirically observed that learning from high dimensional observations such as raw pixels is sample-inefficient. For common practice, RL algorithms for image input often use encoders composed of CNNs to extract useful features from high dimensional observations. Recent studies have shown that CNNs have strong inductive bias towards image styles rather than content (i.e. agent shapes), while content is the information that RL algorithms should focus on. Inspired by this, we suggest reducing the intrinsic style bias of CNNs by proposing Contrastive Inductive Bias Controlling Networks for RL. It can help RL algorithms effectively focus on truly noteworthy information like agents’ own characteristics. Our approach incorporates two transfer networks and feature encoder with contrastive learning methods, guiding RL algorithms to learn more efficiently with sampling. Extensive experiments show that the extended framework greatly enhances the performance of existing model-free methods (i.e. SAC), enabling it to reach state-of-the-art performance on the DeepMind control suite benchmark.