← Back to papers
2023 ICML ICML 2023

An Instrumental Variable Approach to Confounded Off-Policy Evaluation

Abstract

Off-policy evaluation (OPE) aims to estimate the return of a target policy using some pre-collected observational data generated by a potentially different behavior policy. In many cases, there exist unmeasured variables that confound the action-reward or action-next-state relationships, rendering many existing OPE approaches ineffective. This paper develops an instrumental variable (IV)-based method for consistent OPE in confounded sequential decision making. Similar to single-stage decision making, we show that IV enables us to correctly identify the target policy’s value in infinite horizon settings as well. Furthermore, we propose a number of policy value estimators and illustrate their effectiveness through extensive simulations and real data analysis from a world-leading short-video platform.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Machine Learning and Reinforcement Learning
🧭 Keyword Pioneer — confounded sequential decision
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Yang Xu , Jin Zhu , Chengchun Shi , Shikai Luo , Rui Song

Topics

Artificial Intelligence > Core AI > Causal Inference Reinforcement Learning > Methods > Offline RL Machine Learning > Learning Types > Reinforcement Learning Machine Learning > Learning Types > Offline RL

Keywords

causal inference off-policy evaluation sequential decision making behavior policy instrumental variable confounded sequential decision policy value estimation confounded evaluation
Download PDF

Related papers

Orthogonality-Enforced Latent Space in Autoencoders: An Approach to Learning Disentangled Representations 2023
On the Convergence Rate of Gaussianization with Random Rotations 2023
Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models 2023
Generative Adversarial Symmetry Discovery 2023
Function-Space Regularization in Neural Networks: A Probabilistic Perspective 2023