Jianbiao Mei^1,2,*, Yukai Ma^1,2,*, Xuemeng Yang², Licheng Wen², Xinyu Cai², Xin Li^2,3, Daocheng Fu², Bo Zhang², Pinlong Cai², Min Dou², Botian Shi^2,†, Liang He³, Yong Liu^1,†, Yu Qiao^2
1 Zhejiang University ² Shanghai Artificial Intelligence Laboratory ³ East China Normal University
^* Equal Contribution ^† Corresponding Authors

• [2024-5-27] The paper can be accessed at arxiv

• [2024-5-22] We released our project website here

We introduce LeapAD, a novel paradigm for autonomous driving inspired by the human cognitive process. Specifically, LeapAD emulates human attention by selecting critical objects relevant to driving decisions, simplifying environmental interpretation, and mitigating decision-making complexities. Additionally, LeapAD incorporates an innovative dual-process decision-making module, which consists of an Analytic Process (System-II) for thorough analysis and reasoning, along with a Heuristic Process (System-I) for swift and empirical processing.

The scene understanding module analyzes surrounding images and provides descriptions of critical objects that may influence driving decisions. These scenario descriptions are then fed into the dual-process decision module for reasoning and decision-making. The generated decisions are then transmitted to action executor, where they are converted into control signals for interaction with the simulator. The Analytic Process then uses an LLM to accumulate driving analysis and decision-making experience and conduct reflection on accidents. The experience is stored in the memory bank and transferred to a lightweight language model, forming our Heuristic Process for quick responses and continuous learning.

When Heuristic Process encounters traffic accidents, the Analytic Process intervenes, analyzing historical frames to pinpoint errors and provide corrected samples. These corrected samples are then integrated into the memory bank to facilitate continuous learning.

We conduct closed-loop tests in CARLA. It can be seen that LeapAD can make informed decisions using the Heuristic Process with only 1.8B parameters while driving.

This project is released under the Apache 2.0 license.