Autonomous Driving

Introduction

Our lab is focused on proposing novel methodologies and building corresponding facilities to solve the most challenging practical problems for full stack autonomous driving. We combine conventional control, planning and state estimation methods (optimal/robust control, optimization, graph search, Bayesian filtering, etc.) with state-of-the-art machine learning approaches (reinforcement learning, deep neural network, probabilistic graphical models, etc.) to exploit their advantages, and answer questions including but not limited to:

How to combine conventional and learning-based methodologies for computationally efficient planning and control to achieve theoretical guarantee on stability, feasibility and safety, as well as flexibility to learn from trials and data?
How to enable decision-making and planning under uncertainty with social interactions to achieve non-conservatively defensive driving strategies and persuade others to behave as we expected?
What are good probabilistic predictions for reactions of other road users given our future decision, and how to provide accurate probabilistic and reactive predictions to enable safe decision-making and planning with desirable driving quality in a variety of highly interactive driving scenarios?
How to fuse the information from multiple sensors to enable robust 3D object detection and tracking (with LIDAR and camera) and accurate large-scale mapping and localization (with LIDAR, GPS, IMU and camera)?

Research Topics

Motion Planning and Control Generic Environmental Representation and Planning Planning with Constrained Iterative LQR Constrained Policy Net for Safe and Feasible Planning Planning and Control via Imitation Learning Zero-Shot Deep Reinforcement Learning Policy Transfer based on Robust Control
Decision-Making and Behavior Planning Non-Conservatively Defensive Strategy with Integrated Decision-Making and Planning Robustly-Safe Automated Driving (ROAD) System Deep Hierarchical Reinforcement Learning for Decision-Making and Planning Interaction-and-Uncertainty-Aware Joint Decision-Making and Planning Baysian Persuasive Driving Inverse Reinforcement Learning for Social Interactions
Prediction and Behavior Modeling Predicting the probabilistic behavior of others in highly interactive scenarios is one of the most challenging problems for autonomous driving. We propose novel methodologies based on deep neural networks, probabilistic graphical models and inverse reinforcement learning for interactive and probabilistic prediction problems. We also address the fundamental but under-explored aspects in the area of prediction, namely, problem formulation, representation simplification and evaluation metrics. Then we propose to construct a unified framework and fatality-aware benchmark to evaluate a variety of probabilistic reaction prediction algorithms. Fatality-Aware Benchmark for Probabilistic Reaction Prediction Inverse Reinforcement Learning for Social Interactions Hierarchical Inverse Reinforcement Learning for Probabilistic Prediction Deep Neural Network for Semantic Intention and Motion Situation Prediction based on Layered Hidden Markov Model Generative Adversarial Network for Probabilistic Prediction
Detection, Tracking, Localization and Mapping We are focused on LIDAR-Camera fusion for 3D object detection and tracking based on deep convolutional neural networks (CNN) for perception with onboard sensors. One of the models we proposed, RoarNet, was ranked #2 on KITTI benchmark for 3D car detection when submitted. When the object is partially or even fully occluded and the detection is noisy, we introduce learning-based behavioral prediction models combined with Bayesian filtering to achieve accurate tracking. We also investigate framework and methods to enable large-scale mapping and localization based on LIDAR, GPS, IMU and camera. Sparse Non-Homogeneous Pooling for CNN-based LIDAR-Camera Fusion RoarNet: RegiOn Approximation Refinement for CNN-based LIDAR-Camera Fusion Occluded Object Tracking based on Modiﬁed Mixture Particle Filter Mapping and Localization based on Multi-Sensor Fusion

Recent Publications

C. Tang*, Z. Xu*, and M. Tomizuka, “Disturbance observer based tracking controller for neural network driving policy transfer,” submitted to IEEE Transaction on Intelligent Transportation System, 2019.

C. Tang, J. Chen, and M. Tomizuka, “Adaptive probabilistic vehicle trajectory prediction through physically feasible Bayesian recurrent neural network,” in 2019 International Conference on Robotics and Automation (ICRA) pp. 3846–3852, IEEE, 2019.

J. Li, W. Zhan, Y. Hu and M. Tomizuka, “Generic Tracking and Prediction Framework and Its Application in Autonomous Driving”, IEEE Transactions on Intelligent Transportation Systems, early access, 2019.

J. Li, H. Ma and M. Tomizuka, “Conditional Generative Neural System for Probabilistic Trajectory Prediction”, 2019 International Conference on Robotics and Automation (IROS), 2019.

J. Li, H. Ma, W. Zhan and M. Tomizuka, “Coordination and Trajectory Prediction for Vehicle Interactions via Bayesian Generative Modeling”, 2019 IEEE Intelligent Vehicles Sumposium (IV), 2019.

H. Ma, J. Li, W. Zhan and M. Tomizuka, “Wasserstein Generative Learning with Kinematic Constraints for Probabilistic Interactive Driving Behavior Prediction”, 2019 IEEE Intelligent Vehicles Sumposium (IV), 2019.

J. Chen, W. Zhan, and M. Tomizuka, “Autonomous Driving Motion Planning with Constrained Iterative LQR”, submitted to IEEE Transactions on Intelligent Vehicles.

J. Li, H. Ma and Masayoshi Tomizuka, “Interaction-aware Multi-agent Tracking and Probabilistic Behavior Prediction via Adversarial Learning”, 2019 International Conference on Robotics and Automation (ICRA), 2019.

K. Shin, Y. P. Kwon and M.Tomizuka, “RoarNet: A Robust 3D Object Detection based on RegiOn Approximation Refinement”, submitted on ICRA 2019.

C. Peng and M. Tomizuka, “Bayesian Persuasive Driving”, submitted to American Control Conference (ACC), 2019.

W. Zhan, L. Sun, Y. Hu, J. Li, and M. Tomizuka, “Towards a Fatality-Aware Benchmark of Probabilistic Reaction Prediction in Highly Interactive Driving Scenarios”, in 2018 IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.

L. Sun, W. Zhan, and M. Tomizuka, “Probabilistic Prediction of Interactive Driving Behavior via Hierarchical Inverse Reinforcement Learning’’, in 2018 IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.

Y. Hu, W. Zhan, and M. Tomizuka, “A Framework for Probabilistic Generic Traffic Scene Prediction”, in 2018 IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.

J. Li, H. Ma, W. Zhan, and M. Tomizuka, “Generic Probabilistic Interactive Situation Recognition and Prediction: From Virtual to Real”, in 2018 IEEE 21st International Conference on Intelligent Transportation Systems (ITSC), 2018.

Z. Xu, C. Tang, and M. Tomizuka, “Zero-shot Deep Reinforcement Learning Driving Policy Transfer for Autonomous Vehicles based on Robust Control”, in IEEE International Conference on Intelligent Transportation Systems (ITSC), 2018. (Best Student Paper Award)

L. Sun, W. Zhan, M. Tomizuka, and A. Dragan, “Courteous Autonomous Cars”, in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.

L. Sun, C. Peng, W. Zhan, and M. Tomizuka, “A Fast Integrated Planning and Control Framework for Autonomous Driving”, in ASME 2018 Dynamic System and Control Conference (DSCC), 2018.

W. Zhan, A. de La Fortelle, Y.-T. Chen, C.-Y. Chan, and M. Tomizuka, “Probabilistic Prediction from Planning Perspective: Problem Formulation, Representation Simplification and Evaluation Metric”, in 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 1150-1156.

Y. Hu, W. Zhan, and M. Tomizuka, “Probabilistic Prediction of Vehicle Semantic Intention and Motion”, in 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 307-313. (Best Student Paper Award)

J. Li, W. Zhan, and M. Tomizuka, “Generic Vehicle Tracking Framework Capable of Handling Occlusions Based on Modified Mixture Particle Filter”, in 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 936-942.

Z. Wang, W. Zhan, and M. Tomizuka, “Fusing Bird-Eye View LIDAR Point Cloud and Front View Camera Image for 3D Object Detection”, in 2018 IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 834-839.

J. Chen, C. Tang, L. Xin, S. Li and M. Tomizuka, “Continuous Decision Making for On-road Autonomous Driving under Uncertain and Interactive Environments”, in 2018 IEEE Intelligent Vehicles Symposium (IV).

J. Chen, Z. Wang and M. Tomizuka, “Deep Hierarchical Reinforcement Learning for Autonomous Driving with Distinct Behaviors”, in 2018 IEEE Intelligent Vehicles Symposium (IV).

C. Peng and M. Tomizuka, “Cooperative Driving Based on Negotiation with Persuasion and Concession”, in IEEE Intelligent Vehicles Symposium (IV), 2018.

J. Chen, W. Zhan, and M. Tomizuka, “Constrained Iterative LQR for On-Road Autonomous Driving Motion Planning”, in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), 2017, pp. 2232-2238.

W. Zhan, J. Li, Y. Hu, and M. Tomizuka, “Safe and Feasible Motion Generation for Autonomous Driving via Constrained Policy Net”, in Annual Conference on IEEE Industrial Electronics Society (IECON), 2017.

C. Liu, W. Zhan, and M. Tomizuka, “Speed profile planning in dynamic environments via temporal optimization”, in IEEE Intelligent Vehicles Symposium (IV), 2017.

W. Zhan, J. Chen, C.-Y. Chan, C. Liu and M. Tomizuka, “Spatially-Partitioned Environmental Representation and Planning Architecture for On-Road Autonomous Driving”, in IEEE Intelligent Vehicles Symposium (IV), 2017.

C. Liu, J. Chen, T. Nguyen, and M. Tomizuka, “The robustly-safe automated driving system for enhanced active safety”, in SAE World Congress, 2017.

C. Liu, and M. Tomizuka, “Enabling Safe Freeway Driving for Automated Vehicles”, in American Control Conference, 2016.

W. Zhan, C. Liu, Ching-Yao Chan, and M. Tomizuka, “A Non-Conservatively Defensive Strategy for Urban Autonomous Driving”, in International IEEE Conference on Intelligent Transportation Systems (ITSC), 2016.