The core focus of our research is to understand the interactive human (driving) behavior and the scene with proper representations, and to predict and emulate the behavior to enable safe and high-quality autonomy for various mobility. In order to sufficiently support the focus, our research scope also expands to the fundamental aspects such as validation/evaluation and dataset construction (INTERACTION dataset for behavior/prediction and UrbanLoco dataset for localization/mapping). We also dive into perception from different perspectives and control for various platforms, and incorporate our behavior-related research insights into these aspects. Our viewpoint is to exploit the synergies of model-based planning/control methods and state-of-the-art machine learning techniques to combine our prior and what we can learn from data. Several recent publications on prediction, decision-making, planning and control for autonomous driving by the group members have received Best Student Paper or Best Paper Finalist in flagship conferences on robotics and intelligent vehicle/transportation, such as IEEE/RSJ IROS, IEEE ITSC and IEEE IV.

Facilities and Datasets


Data-collection vehicle: with a high-end navigation system, and a LiDAR calibrated and synchronized with 6 cameras. Data collection and tests in simulation: Two sets of driving simulator interfaces composed with parts from real vehicles. Autonomous vehicles: Our algorithms were tested on autonomous vehicles in test fields supported by our sponsors.


In addition to the hardware and software infrastructures, data is another key asset for research spanning from perception and localization to prediction and planning. We published part of our internal datasets on driving behavior and localization in collaboration with our partners in Europe and Asia to facilitate the research community and accelerate industrial implementations. Please directly request the data via the websites for research purposes, or contact us if you are interested in commercial use.

INTERACTION Dataset: highly interactive driving behavior with semantic map

  • Densely interactive beahvior and critical situations in complex scenarios: negotiations, inexplicit right-of-way, irrational behavior, near-collision situations, violation of traffic rules.
  • Highly accurate trajectories and complete information of surroundings.
  • HD maps with full semantics required by prediction and planning algorithms.
  • International locations (US, Germany, China) with diversified scenarios (roundabouts, signalized/unsignalized intersections, highway/urban merging and lane change).
  • Dataset website, and papers (arxiv, IROS19).
  • Prediction challenge website

UrbanLoco Dataset: localization and mapping in dense urban scenes


INTERPRET: INTERACTION-Dataset-based PREdicTion Challenge

In the field of autonomous driving, it is a consensus in both academia and industry that behavior prediction (e.g., trajectories, actions, intentions) is one of the most challenging problems blocking the realization of full autonomy. Unfortunately, there are yet no benchmark to fairly compare the performance of different prediction models/algorithms, particularly when the influence of prediction performance in a closed-loop format (integrated with different planners) is considered. To expedite research and inspire discussions on the evaluation of prediction models/algorithms, we present the INTERACTION-dataset-based PREdicTion (INTERPRET) Challenge. This is a step towards the construction of effective and valuable predictors for the development of autonomous driving.

Workshops & competitions in flagship conferences

Machine learning community
Robotics community
Intelligent transportation/vehicle community

Research Topics

1. Prediction, representation and emulation of interactive behavior

Application: prediction, decision-making and behavior planning for autonomous vehicles, comprehension and modeling of interactive, social behavior, (scene/motion) representation learning and construction, imitation and generation of human (driving) behavior.

1.1 Inverse reinforcement learning (IRL, also inverse optimal control) and game theory for human-like behavior generation

  • Modeling, learning and imitating social behaviors from humans, from planning behavior to perception behavior: IROS18, IV19, TechXplore
  • A hierarchical IRL algorithm to capture the hierarchical structure of human decision-making process: ITSC18
  • Exploring and expressing the irrationality of human behaviors via a risk-sensitivity IRL algorithm: ITSC19
  • A better prediction model by online combination of deep learning models with IRL models: ITSC19
  • Learning generic and suitable cost functions for driving: ICRA20.
  • Efficient sampling-based Maximum Entropy IRL: RAL.
  • Modeling the multi-agent interaction based on Game Theory: papers to appear.

1.2 Decision and planning under uncertainty with interaction

1.3 Interactive prediction with dynamics, semantic representation and reasoning

1.4 Scene/motion representation learning and construction

  • Comprehensive scene and motion representation construction: IV17, arxiv.
  • Representation learning: arxiv, and papers to appear.

2. Perception, mapping and localization for humans, vehicles and cities

Application: detection/tracking/mapping/localization with onboard sensors, traffic cameras/drones, connectivity, as well as occlusion inference via behavior of others.

2.1 3D detection and segmentation with LiDARs and cameras

  • 3D point cloud segmentation: ECCV20.
  • 3D detection with LiDAR-camera fusion: IV18, IV19, arxiv.
  • Uncertainty-aware 3D detection: ITSC20
  • Detection from bird’s eye view: arxiv.
  • Visual transformers: arxiv

2.2 Mapping, localization and 3D reconstruction

  • 3D point cloud reconstruction and registration: IROS19, IFAC20.
  • Automatic HD map generation with semantics.
  • City-scale mapping and localization.
  • Object removal for better mapping and localization.

2.3 Occluded object detection and tracking via social/cooperative perception

  • Occluded object tracking by inferring from motions of its surrounding objects: Trans-ITS, IV18.
  • Social perception to infer existence of occluded objects from behavior of others: IV19, TechXplore.
  • Cooperative perception to achieve better detection and tracking of occluded object via connectivity.

3. Validation/test and simulation for autonomy and environments

Application: test and evaluation of prediction/planning/perception/localization modules by reconstructing the real world and utilizing high-quality data.

3.1 Prediction benchmark and challenge

3.2 Uncertainty estimation and evaluation for perception and localization

  • Uncertainty-aware evaluation and labeling analysis for 3D detection: IROS20.
  • Uncertainty estimation in LiDAR-base localization: IET EL Letter.

3.3 Testing prediction/decision/planning in simulation

  • papers to appear

4. Planning, dynamics and control for autonomous systems

Application: Trajectory planning, dynamics modeling and controller design for passenger/delivery/racing vehicles (with connectivity).

4.1 Trajectory planning

4.2 Policy transfer, safe learning and control

  • Policy transfer with robust control: Trans-ITS, ITSC18 (Best Student Paper).
  • Safe learning and control.

4.3 Vehicle dynamics, state estimation and control

  • System identification: IV20.
  • Adaptive and robust control: papers to appear.
  • Estimation and control with connectivity: Sensors.
  • Selective vehicle dynamics and control research in MSC Lab (in collaboration with California PATH) over a decade ago:

Joining Our Group

Please send an email to Professor Masayoshi Tomizuka ( and Dr. Wei Zhan ( if you are interested in our Research Topics and joining our group.

  • We are welcoming Berkeley students to directly work with us, or students out of Berkeley to visit us. We also accept virtual visit to work with us remotely for those with difficulties to conduct a physical visit. Please note that an experience will not be recognized without a formal interview and approval by the faculty and postdocs.
  • For prospective Ph.D. students, please apply to the Mechanical Engineering Department of UC Berkeley by December 1st and send an email to address your strength and interest.

Please make sure that the following aspects are well covered in your application email.

  • Indicate in the email on your 1) primary goal of the research experience and particular interests; 2) start and end dates for working with us; 3) uniqueness and strength on research experiences/publications and/or skills and knowledge; 4) long-term/career goals.
  • Attach a CV including your 1) home university, major, GPA and ranking; 2) research/working experiences; 3) publications/patents (if any); 4) skill set on coding/software/hardware and corresponding proficiency; 5) knowledge set on methods/algorithms.
  • Attach a brief introduction (within 5 pages of slides) showing the core methods/algorithms and main results and demos of your previous research or working experiences. Links to cloud storage are welcome for large files.
  • Attach all publications (including submitted paper) or well-formatted project final reports if any. Links to cloud storage or online publications are welcome.

Members and Contact

The group lead is Dr. Wei Zhan. The research activities are coordinated by 2 Postdocs (Dr. Wei Zhan and Dr. Liting Sun) and conducted by 14 Ph.D. students (following headshots) with over 10 visiting researchers.

Please contact Dr. Wei Zhan ( and Professor Masayoshi Tomizuka ( for more information if you are interested in the topics above by

  • working with us (join/visit the group or collaborate) from academia, or
  • research collaboration from industry.