A comprehensive, open-source benchmark unifying 2D/3D occupancy labels, per-voxel flow annotations, and multi-agent support across multiple real-world and synthetic datasets.

Our UniOcc Framework

enables three representative tasks: occupancy prediction, occupancy forecasting with optional flow, and cooperative occupancy prediction and forecasting with optional flow

by unifying:

• ⬇ Data Format and Features
  We define Semantic Occupancy Label, Camera Images, Camera Field-of-View (FOV) Mask, Camera Intrinsics and Extrinsics, Ego-to-World Transformation, Forward Occupancy Flow, Backward Occupancy Flow, and Object Annotations.
• ⬇ Datasets
  We build our unified datasets from: nuScenes, Waymo, CARLA, and OpenCOOD.
• ⬇ Occupancy Processing Toolkit
  We build tools for Object Identification, Object Tracking, and Object Alignment.
• ⬇ Evaluation Metrics
  We incorporate both Voxel-Based Evaluation and Ground-Truth-Free Evaluation.

Abstract

We introduce UniOcc, a comprehensive, unified benchmark for occupancy forecasting (i.e., predicting future occupancies based on historical information) and current-frame occupancy prediction from camera images. UniOcc unifies data from multiple real-world datasets (i.e., nuScenes, Waymo) and high-fidelity driving simulators (i.e., CARLA, OpenCOOD), which provides 2D/3D occupancy labels with per-voxel flow annotations and support for cooperative autonomous driving. In terms of evaluation, unlike existing studies that rely on suboptimal pseudo labels for evaluation, UniOcc incorporates novel metrics that do not depend on ground-truth occupancy, enabling robust assessment of additional aspects of occupancy quality. Through extensive experiments on state-of-the-art models, we demonstrate that large-scale, diverse training data and explicit flow information significantly enhance occupancy prediction and forecasting performance.

Key Ideas and Contributions

1) First-of-its-kind unified 2D/3D occupancy forecasting and prediction benchmark: including flow information for conventional and cooperative driving by unifying real data from nuScenes and Waymo and synthetic data from CARLA & OpenCOOD.
2) A user-friendly platform for current-frame occupancy prediction and multi-frame occupancy forecasting: enabling easy setup, cross-dataset augmentation, and comprehensive occupancy evaluation with or without reference to ground-truth labels
3) State-of-the-art performance of our pipeline and evaluation metrics on leading occupancy forecasting/prediction models: showing that (1) incorporating flow information yields performance gains in occupancy forecasting and (2) existing methods face challenges in cross-domain generalization, highlighting avenues for future research.