SMART: Scalable Multi-agent Real-time Motion Generation via Next-token Prediction

Wei Wu1,2 , Xiaoxin Feng1, Ziyan Gao1, Yuheng Kan1
1SenseTime Research, 2Tsinghua University
Paper arXiv Code

Abstract

Data-driven autonomous driving motion generation tasks are frequently impacted by the limitations of dataset size and the domain gap between datasets, which precludes their extensive application in real-world scenarios. To address this issue, we introduce SMART, a novel autonomous driving motion generation paradigm that models vectorized map and agent trajectory data into discrete sequence tokens. These tokens are then processed through a decoder-only transformer architecture to train for the next token prediction task across spatial-temporal series. This GPT-style method allows the model to learn the motion distribution in real driving scenarios. SMART achieves state-of-the-art performance across most of the metrics on the generative Sim Agents challenge, ranking 1st on the leaderboards of Waymo Open Motion Dataset (WOMD), demonstrating remarkable inference speed. Moreover, SMART represents the generative model in the autonomous driving motion domain, exhibiting zero-shot generalization capabilities: Using only the NuPlan dataset for training and WOMD for validation, SMART achieved a competitive score of 0.71 on the Sim Agents challenge. Lastly, we have collected over 1 billion motion tokens from multiple datasets, validating the model’s scalability. These results suggest that SMART has initially emulated two important properties: scalability and zero-shot generalization, and preliminarily meets the needs of large-scale real-time simulation applications. We have released all the code to promote the exploration of models for motion generation in the autonomous driving field.

Video

News

Zero-shot Generalization

In driving motion generation, we have focused on the model’s zero-shot generalizability across datasets. SMART, trained solely on the NuPlan dataset, performed well on the WOMD test dataset, despite the non-overlapping map areas. The following videos show SMART's performance, with scenarios nearly absent in NuPlan, highlighting its ability to generalize and perform in unseen conditions. This demonstrates SMART's robustness in generating realistic and diverse motion patterns in novel situations.

Scenario Generalization

As a motion generation model, SMART is capable of generalizing log scenarios to produce a multitude of simulation scenarios. The following videos demonstrate several examples of scenario generalization, showcasing the versatility and robustness of SMART in creating diverse and realistic driving environments. These examples illustrate how SMART can effectively simulate various conditions and interactions, enhancing the scope and reliability of autonomous driving simulations.

BibTeX

@article{wu2024smart,
  title     = {SMART: Scalable Multi-agent Real-time Simulation via Next-token Prediction},
  author    = {Wu, Wei and Feng, Xiaoxin and Gao, Ziyan and Kan, Yuheng},
  journal   = {arXiv preprint arXiv:2405.15677},
  year      = {2024},
}