← Back to papers
2017 ICML ICML 2017

Learning in POMDPs with Monte Carlo Tree Search

Abstract

The POMDP is a powerful framework for reasoning under outcome and information uncertainty, but constructing an accurate POMDP model is difficult. Bayes-Adaptive Partially Observable Markov Decision Processes (BA-POMDPs) extend POMDPs to allow the model to be learned during execution. BA-POMDPs are a Bayesian RL approach that, in principle, allows for an optimal trade-off between exploitation and exploration. Unfortunately, BA-POMDPs are currently impractical to solve for any non-trivial domain. In this paper, we extend the Monte-Carlo Tree Search method POMCP to BA-POMDPs and show that the resulting method, which we call BA-POMCP, is able to tackle problems that previous solution methods have been unable to solve. Additionally, we introduce several techniques that exploit the BA-POMDP structure to improve the efficiency of BA-POMCP along with proof of their convergence.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Machine Learning and Reinforcement Learning
🧭 Keyword Pioneer — model learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics
🐣 Hot Topic Early Bird — partially observable markov decision process

Authors

Sammie Katt , Frans A. Oliehoek , Christopher Amato

Topics

Artificial Intelligence > Core AI > Agent Systems Machine Learning > Optimization & Theory > Bayesian Inference Reinforcement Learning > Methods > Deep RL Reinforcement Learning > Methods > Policy Learning Artificial Intelligence > Core AI > Reasoning Machine Learning > Learning Paradigms > Multi-Task Learning

Keywords

reinforcement learning bayesian learning bayesian reinforcement learning partially observable markov decision process monte carlo tree search model learning bayes-adaptive pomdp optimal exploration
Download PDF

Related papers

Bottleneck Conditional Density Estimation 2017
Constrained Policy Optimization 2017
Near-Optimal Design of Experiments via Regret Minimization 2017
Input Convex Neural Networks 2017
An Efficient, Sparsity-Preserving, Online Algorithm for Low-Rank Approximation 2017