Jointly learning rewards and policies: an iterative Inverse Reinforcement Learning framework with…

Jointly learning rewards and policies: an iterative Inverse Reinforcement Learning framework with ranked synthetic trajectories

A novel tractable and interpretable algorithm to learn from expert demonstrations

Introduction

Imitation Learning has recently gained increasing attention in the Machine Learning community, as it enables the transfer of expert knowledge to autonomous agents through observed behaviors. A first category of algorithm is Behavioral Cloning (BC), which aims to directly replicate expert demonstrations, treating the imitation process as a supervised learning task where the agent attempts to match the expert’s actions in given states. While straightforward and computationally efficient, BC often suffers from overfitting and poor generalization.

In contrast, Inverse Reinforcement Learning (IRL) targets the underlying intent of expert behavior by inferring a reward function that could explain the expert’s actions as optimal within the considered environment. Yet, an important caveat of IRL is the inherent ill-posed nature of the problem — i.e. that multiple (if not possibly an infinite number of) reward functions can make the expert trajectories as optimal. A widely adopted class of methods to tackle this ambiguity includes Maximum Entropy IRL algorithms, which introduce an entropy maximization term to encourage stochasticity and robustness in the inferred policy.

In this article, we choose a different route and introduce a novel iterative IRL algorithm that jointly learns both the reward function and the optimal policy from expert demonstrations alone. By iteratively synthesizing trajectories and guaranteeing their increasing quality, our approach departs from traditional IRL models to provide a fully tractable, interpretable and efficient solution.

The organization of the article is as follows: section 1 introduces some basic concepts in IRL. Section 2 gives an overview of recent advances in the IRL literature, which our model builds on. We derive a sufficient condition for the our model to converge in section 3. This theoretical result is general and can apply to a large class of algorithms. In section 4, we formally introduce the full model, before concluding on key differences with existing literature and further research directions in section 5.

1. Background definitions:

First let’s define a few concepts, starting with the general Inverse Reinforcement Learning problem (note: we assume the same notations as (2004), Stanford Artificial Intelligence Laboratory

[2] J. Bohg, M. Pavone, D. Sadigh, (2019), Proceedings of Machine Learning Research

[4] D. S. Brown, W. Goo, P. Nagarajan, S. Niekum, (2019), arXiv

[6] A. Mandyam, D. Li, D. Cai, A. Jones, B. E. Engelhardt, (2022), arXiv

[8] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, Ch. Fu, C. Ma, J. Jiao, S. Levine, K. Hausman, was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.