融合鸟瞰图的端到端强化学习决策规划模型

doi:10.19562/j.chinasae.qcgc.2025.09.004

摘要/Abstract

摘要：

端到端自动驾驶决策规划模型是行业的热点研究方向，传感器信号与动作输出的空间、时序不统一以及端到端模型的收敛问题，极大制约了模型的实际应用效果。为此，本文提出一种融合鸟瞰图预测的端到端强化学习模型FB-Roach，通过鸟瞰图预测模型建立环境信息表征，设计了以静态查询表为核心的前向投影模块，以及融合时序信息、深度嵌入和语义嵌入的多任务后向投影模块，保证输入信号与输出动作的一致性；进一步结合注意力机制创新性地提出一种无循环的深度网络架构，有效融合鸟瞰图和车辆状态信息，并使用强化学习PPO算法优化模型的动作输出，实现自动驾驶车辆的智能决策控制。基于CARLA模拟器，构建了不同基准下多样性的量化评估指标。实验结果表明，所提出算法在模型收敛速度和驾驶决策安全性方面均优于目前主流算法。

关键词: 端到端自动驾驶, 鸟瞰图, 强化学习, 决策规划

Abstract:

End-to-end autonomous driving decision-making and planning models are a hot research direction in the industry. The spatial and temporal inconsistency between sensor signals and action outputs， as well as the convergence issues of end-to-end models， greatly limit the practical application effectiveness of these models. Therefore， in this paper an end-to-end reinforcement learning model called FB-Roach is proposed that integrates bird's-eye view prediction. Environmental information representation is established through a bird's-eye view prediction model. A forward projection module centered on a static Look-Up table， as well as a multi-task backward projection module that integrates temporal information， depth embedding， and semantic embedding， is designed to ensure the consistency between input signals and output actions. Furthermore， by innovatively incorporating the attention mechanism， the non-recurrent deep network architecture is proposed that effectively fuses bird's-eye view and vehicle state information. The model's action output is optimized using the PPO reinforcement learning algorithm to achieve intelligent decision-making and control for autonomous vehicles. Based on the CARLA simulator， a variety of quantitative evaluation indicators are constructed under different benchmarks. The experiments results show that the proposed algorithm outperforms current mainstream algorithms in terms of model convergence speed and driving decision safety.

Key words: end-to-end autonomous vehicles, BEV, reinforcement learning, decision-making

汤白雪,蔡英凤,陈龙,王海,饶中钰,刘泽. 融合鸟瞰图的端到端强化学习决策规划模型[J]. 汽车工程, 2025, 47(9): 1674-1685.

Baixue Tang,Yingfeng Cai,Long Chen,Hai Wang,Zhongyu Rao,Ze Liu. End-to-End Decision-Making Model Based on Reinforcement Learning Incorporating Bird's Eye View Representation[J]. Automotive Engineering, 2025, 47(9): 1674-1685.

图/表 17

图1

图2

图3

图4

表1

奖励函数设置"

奖励	触发事件	奖励价值
$r a c t i o n$	$Δ$ steering≥0.01	-1- $v$
$r t e r m i n a l$	闯红灯	-1- $v$
	违反停车标识	-1- $v$
	碰撞	-1- $v$
	偏离路线	-1
	拥堵	-1

表1

图5

表2

表3

表4

图6

表5

图7

图8

图9

图10

表6

表7

参考文献 19

[1]	CHEN L， WU P， CHITTA K， et al. End-to-end autonomous driving： challenges and frontiers［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2024， 46（12）： 10164-10183.
[2]	李升波，刘畅，殷玉明. 汽车端到端自动驾驶系统的关键技术与发展趋势［J］. 人工智能， 2023， 5： 1-16.
	LI S B， LIU C， YIN Y M. Key technologies and development trend of automotive end-to-end autonomous driving system［J］. Artificial Intelligence， 2023， 5： 1-16.
[3]	CHEN D， ZHOU B， KOLTUN V， et al. Learning by cheating［C］.Conference on Robot Learning. PMLR， 2020： 66-75.
[4]	TOROMANOFF M， WIRBEL E， MOUTARDE F. End-to-end model-free reinforcement learning for urban driving using implicit affordances［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 7153-7162.
[5]	ZHANG Z， LINIGER A， DAI D， et al. End-to-end urban driving by imitating a reinforcement learning coach［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15222-15232.
[6]	CHEN J， YUAN B， TOMIZUKA M. Model-free deep reinforcement learning for urban autonomous driving［C］. 2019 IEEE Intelligent Transportation Systems Conference （ITSC）. IEEE， 2019： 2765-2771.
[7]	DOSOVITSKIY A， ROS G， CODEVILLA F， et al. CARLA： an open urban driving simulator［C］. Conference on Robot Learning. PMLR， 2017： 1-16.
[8]	陈妍妍，王海，蔡英凤，等. 基于检测的高效自动驾驶实例分割方法［J］. 汽车工程， 2023， 45（4）： 541-550.
	CHEN Y Y，WANG H，CAI Y F， et al. Efficient automatic driving instance segmentation method based on detection［J］. Automotive Engineering， 2023， 45（4）： 541-550.
[9]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：， 2017.
[10]	RAO Z， CAI Y， WANG H， et al. Enhancing autonomous driving： a low-cost monocular end-to-end framework with multi-task integration and temporal fusion［J/OL］. IEEE Transactions on Intelligent Vehicles， 2024： 1-14.
[11]	XIE E， YU Z， ZHOU D， et al. M²BEV： multi-camera joint 3D detection and segmentation with unified birds-eye view representation［J］. arXiv preprint arXiv：， 2022.
[12]	HUANG B， LI Y， XIE E， et al. Fast-BEV： towards real-time on-vehicle bird’s-eye view perception ［J］. arXiv preprint arXiv：， 2023.
[13]	LI Z， WANG W， LI H， et al. Bevformer： learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers［C］. European Conference on Computer Vision. Cham： Springer Nature Switzerland， 2022： 1-18.
[14]	CODEVILLA F， SANTANA E， LÓPEZ A M， et al. Exploring the limitations of behavior cloning for autonomous driving［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 9329- 9338.
[15]	CHEN D， KOLTUN V， KRAHENBUHL P. Learning to drive from a world on rails［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15590-15599.
[16]	CHEN D， KRÄHENBÜHL P. Learning from all vehicles［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2022： 17222-17231.
[17]	CHITTA K， PRAKASH A， JAEGER B， et al. Transfuser： imitation with transformer-based sensor fusion for autonomous driving［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 45（11）： 12878-12895.
[18]	PRAKASH A， BEHL A， OHN-BAR E. Exploring data aggregation in policy learning for vision-based urban autonomous driving［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 11763-11773.
[19]	ZHAO Y， WU K， XU Z， et al.CADRE： a cascade deep reinforcement learning framework for vision-based autonomous urban driving［J］. arXiv：.

基准	地图	车辆数量	行人数量
NoCrash-dense	Town 01	100	250
NoCrash-dense	Town 02	70	150
Town05- Long	Town 01	120	120
	Town 02	70	70
	Town 04	150	80
	Town 05	120	120
	Town 06	120	80

方法	训练方法	DS↑	RC↑	IS↑
CILRS^［14］	IL	7.8	10.3	0.75
LBC^［3］	IL	24.1	73.4	0.31
Rails^［15］	RL	31.4	57.6	0.56
LAV^［16］	IL	46.5	69.8	0.73
Transfuser^［17］	IL	55.0	89.7	0.63
Roach^［5］	RL+BC	60.1	85.8	0.69
FB-Roach	RL	62.8	87.6	0.72

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn
CILRS^［14］	45±6	39±4	23±1	26±2
DARB^［18］	71±4	72±3	41±2	43±2
CADRE^［19］	82±0	76±0	61±0	52±0
LBC^［3］	86±3	83±6	60±3	59±8
Rails^［15］	96±0	74±0	84±0	66±0
Roach^［5］	91±4	90±7	83±3	83±3
FB-Roach	93±3	92±1	85±1	83±4

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn
CADRE	87±3	83±1	78±3	74±2
Roach	95±2	95±3	91±3	90±2
FB-Roach	96±1	96±2	91±1	91±1

静态查询表	融合时序信息	深度信息嵌入	语义信息嵌入	DS↑	RC↑	IS↑
				55.4	85.1	0.65
√				56.3	85.3	0.66
√	√			58.7	86.5	0.68
√	√	√		61.5	87.2	0.71
√	√	√	√	62.8	87.6	0.72