Probabilistic Prediction via Hierarchical Inverse Reinforcement Learning

Contents:

• Introduction
• Researchers
• Publications

Introduction

Motivation

Autonomous vehicles (AVs) are interacting with human on the road. To this end, it needs to predict and reason about possible future behavior of human, and plan its own trajectories accordingly. Wrong prediction can cause either too conservative motions such as unnecessary stops/yielding, or dangerous situations like emergency brakes and unavoidable collisions. Hence, accurate prediction is of crucial importance to enable a safe and efficient autonomous car.

Starting from as simple as assuming that the other drivers would maintain their current velocities within the planning horizon to as complicated as modelling the probability distributions over all possible trajectories/actions, many prediction approaches have been proposed. Most of them, however, formulate prediction in isolation, i.e., the influence of other vehicles’ actions to the human driver is ignored.

In fact, human drivers will actively anticipate and reason about surrounding vehicles’ behavior when they decide the next-step actions. This means that besides historical and current scene states, the distribution over all possible trajectories is also influenced by their beliefs about other vehicles’ plan. For example, given the same historical trajectories, a lane-keeping driver might be more probable to decelerate rather than maintaining current speed if the driver believes that another vehicle is about to merge into his lane. The difference between Prediction in Isolation and Interaction-Aware Prediction are illustrated in Fig. 1.

Our insight is that in highly interactive driving scenarios, an accurate prediction should consider not only the time-domain dependency but also the interaction among vehicles. Namely, an autonomous car should anticipate the conditional probability over all possible trajectories of the human driver given not only historical and current states, but also its own future plans.

Approach

To obtain such a probabilistic and interactive prediction of human drivers’ behavior, we approximate the ``internal incentive’’ of human via Inverse Reinforcement Learning. Moreover, noting that human’s planning procedure is naturally hierarchical with both discrete and continuous driving decisions, we propose a hierarchical inverse reinforcement learning (IRL) framework. Given the autonomous vehicle’s future plan, the probability distribution over all future trajectories of the predicted human driver is modelled as a mixture of distributions, partitioned by the discrete driving decisions. The example of the hierarchical structure is given in Fig. 2.

Case Study

We collect 134 human driving trajectories in a freeway ramp-merging scenario from the Next Generation SIMulation (NGSIM) dataset. 80 of them are used for training, and the remaining 54 are for test. Figure 3 shows an examplar group of trajectories on the road map. There are four cars in scene, one merging vehicle (red), one lane-keeping vehicle (blue) and two surrounding vehicles (black), with one ahead of the blue car and the other behind. Our interest focuses on the interactive driving behavior of both the merging vehicle and the lane-keeping vehicle.

Training Performance

Figure 4 gives the training curves regarding to both the continuous and discrete driving decisions.

Test Performance

Prediction Performance among three approaches is compared: the proposed hierarchical IRL method, a neural-network (NN) based method and a hidden markov models (HMM) based method.

A fatality-aware Brier score is adopted as an evaluation metric for the prediction of the discrete driving decisions. For the continuous driving trajectories, we use mean Euclidean distance as an evaluation metric. The results are shown in Table II and III.

Also, some examplar predicted trajectories as well as the corresponding ground-truth trajectories are given in Fig. 5.

Researchers

Publications

L. Sun, W. Zhan, and M. Tomizuka, “Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning”, to appear on the 21st IEEE International Conference on Intelligent Transportation Systems, November 4-7, 2018, Maui, Hawaii, USA.