融合鸟瞰图的端到端强化学习决策规划模型

doi:10.19562/j.chinasae.qcgc.2025.09.004

Abstract

Abstract:

End-to-end autonomous driving decision-making and planning models are a hot research direction in the industry. The spatial and temporal inconsistency between sensor signals and action outputs， as well as the convergence issues of end-to-end models， greatly limit the practical application effectiveness of these models. Therefore， in this paper an end-to-end reinforcement learning model called FB-Roach is proposed that integrates bird's-eye view prediction. Environmental information representation is established through a bird's-eye view prediction model. A forward projection module centered on a static Look-Up table， as well as a multi-task backward projection module that integrates temporal information， depth embedding， and semantic embedding， is designed to ensure the consistency between input signals and output actions. Furthermore， by innovatively incorporating the attention mechanism， the non-recurrent deep network architecture is proposed that effectively fuses bird's-eye view and vehicle state information. The model's action output is optimized using the PPO reinforcement learning algorithm to achieve intelligent decision-making and control for autonomous vehicles. Based on the CARLA simulator， a variety of quantitative evaluation indicators are constructed under different benchmarks. The experiments results show that the proposed algorithm outperforms current mainstream algorithms in terms of model convergence speed and driving decision safety.

Key words: end-to-end autonomous vehicles, BEV, reinforcement learning, decision-making

Baixue Tang,Yingfeng Cai,Long Chen,Hai Wang,Zhongyu Rao,Ze Liu. End-to-End Decision-Making Model Based on Reinforcement Learning Incorporating Bird's Eye View Representation[J].Automotive Engineering, 2025, 47(9): 1674-1685.

Figures/Tables 17

奖励	触发事件	奖励价值
$r a c t i o n$	$Δ$ steering≥0.01	-1- $v$
$r t e r m i n a l$	闯红灯	-1- $v$
	违反停车标识	-1- $v$
	碰撞	-1- $v$
	偏离路线	-1
	拥堵	-1

References 19

[1]	CHEN L， WU P， CHITTA K， et al. End-to-end autonomous driving： challenges and frontiers［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2024， 46（12）： 10164-10183.
[2]	李升波，刘畅，殷玉明. 汽车端到端自动驾驶系统的关键技术与发展趋势［J］. 人工智能， 2023， 5： 1-16.
	LI S B， LIU C， YIN Y M. Key technologies and development trend of automotive end-to-end autonomous driving system［J］. Artificial Intelligence， 2023， 5： 1-16.
[3]	CHEN D， ZHOU B， KOLTUN V， et al. Learning by cheating［C］.Conference on Robot Learning. PMLR， 2020： 66-75.
[4]	TOROMANOFF M， WIRBEL E， MOUTARDE F. End-to-end model-free reinforcement learning for urban driving using implicit affordances［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 7153-7162.
[5]	ZHANG Z， LINIGER A， DAI D， et al. End-to-end urban driving by imitating a reinforcement learning coach［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15222-15232.
[6]	CHEN J， YUAN B， TOMIZUKA M. Model-free deep reinforcement learning for urban autonomous driving［C］. 2019 IEEE Intelligent Transportation Systems Conference （ITSC）. IEEE， 2019： 2765-2771.
[7]	DOSOVITSKIY A， ROS G， CODEVILLA F， et al. CARLA： an open urban driving simulator［C］. Conference on Robot Learning. PMLR， 2017： 1-16.
[8]	陈妍妍，王海，蔡英凤，等. 基于检测的高效自动驾驶实例分割方法［J］. 汽车工程， 2023， 45（4）： 541-550.
	CHEN Y Y，WANG H，CAI Y F， et al. Efficient automatic driving instance segmentation method based on detection［J］. Automotive Engineering， 2023， 45（4）： 541-550.
[9]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：， 2017.
[10]	RAO Z， CAI Y， WANG H， et al. Enhancing autonomous driving： a low-cost monocular end-to-end framework with multi-task integration and temporal fusion［J/OL］. IEEE Transactions on Intelligent Vehicles， 2024： 1-14.
[11]	XIE E， YU Z， ZHOU D， et al. M²BEV： multi-camera joint 3D detection and segmentation with unified birds-eye view representation［J］. arXiv preprint arXiv：， 2022.
[12]	HUANG B， LI Y， XIE E， et al. Fast-BEV： towards real-time on-vehicle bird’s-eye view perception ［J］. arXiv preprint arXiv：， 2023.
[13]	LI Z， WANG W， LI H， et al. Bevformer： learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers［C］. European Conference on Computer Vision. Cham： Springer Nature Switzerland， 2022： 1-18.
[14]	CODEVILLA F， SANTANA E， LÓPEZ A M， et al. Exploring the limitations of behavior cloning for autonomous driving［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 9329- 9338.
[15]	CHEN D， KOLTUN V， KRAHENBUHL P. Learning to drive from a world on rails［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15590-15599.
[16]	CHEN D， KRÄHENBÜHL P. Learning from all vehicles［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2022： 17222-17231.
[17]	CHITTA K， PRAKASH A， JAEGER B， et al. Transfuser： imitation with transformer-based sensor fusion for autonomous driving［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 45（11）： 12878-12895.
[18]	PRAKASH A， BEHL A， OHN-BAR E. Exploring data aggregation in policy learning for vision-based urban autonomous driving［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 11763-11773.
[19]	ZHAO Y， WU K， XU Z， et al.CADRE： a cascade deep reinforcement learning framework for vision-based autonomous urban driving［J］. arXiv：.

基准	地图	车辆数量	行人数量
NoCrash-dense	Town 01	100	250
NoCrash-dense	Town 02	70	150
Town05- Long	Town 01	120	120
	Town 02	70	70
	Town 04	150	80
	Town 05	120	120
	Town 06	120	80

方法	训练方法	DS↑	RC↑	IS↑
CILRS^［14］	IL	7.8	10.3	0.75
LBC^［3］	IL	24.1	73.4	0.31
Rails^［15］	RL	31.4	57.6	0.56
LAV^［16］	IL	46.5	69.8	0.73
Transfuser^［17］	IL	55.0	89.7	0.63
Roach^［5］	RL+BC	60.1	85.8	0.69
FB-Roach	RL	62.8	87.6	0.72

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn
CILRS^［14］	45±6	39±4	23±1	26±2
DARB^［18］	71±4	72±3	41±2	43±2
CADRE^［19］	82±0	76±0	61±0	52±0
LBC^［3］	86±3	83±6	60±3	59±8
Rails^［15］	96±0	74±0	84±0	66±0
Roach^［5］	91±4	90±7	83±3	83±3
FB-Roach	93±3	92±1	85±1	83±4

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn
CADRE	87±3	83±1	78±3	74±2
Roach	95±2	95±3	91±3	90±2
FB-Roach	96±1	96±2	91±1	91±1

静态查询表	融合时序信息	深度信息嵌入	语义信息嵌入	DS↑	RC↑	IS↑
				55.4	85.1	0.65
√				56.3	85.3	0.66
√	√			58.7	86.5	0.68
√	√	√		61.5	87.2	0.71
√	√	√	√	62.8	87.6	0.72

End-to-End Decision-Making Model Based on Reinforcement Learning Incorporating Bird's Eye View Representation

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 19

Related Articles 15

Metrics

Comments

Recommended 0

自注意力机制	交叉注意力机制	DS↑	RC↑	IS↑
		59.8	86.5	0.68
√		60.5	87.1	0.69
	√	61.2	87.3	0.71
√	√	62.8	87.6	0.72

[1]	Junjie Zhao,Yinuo Wang,Jiang Wu,Sichao Wu,Changdi Zou,Hongda Wang,ShengboEben Li,Fei Ma,Jingliang Duan. Research on Diffusion Reinforcement Learning Method for Vehicle Trajectory Tracking and Collision Avoidance of Autonomous Vehicles [J]. Automotive Engineering, 2025, 47(8): 1490-1500.
[2]	Jian Wu,Yukang Shi,Bing Zhu,Jian Zhao,Zhicheng Chen. Intelligent Vehicle Decision for Roundabouts Based on Subjective Prior Reinforcement Learning [J]. Automotive Engineering, 2025, 47(8): 1513-1521.
[3]	Jiayi Guan,Bin Li,Ao Zhou,Zhiguo Zhao,Qiao Lin,Guang Chen. Study on Safe Parking Path Planning Algorithm for Narrow Environment [J]. Automotive Engineering, 2025, 47(5): 797-808.
[4]	Yingjiu Pan,Yi Xi,Yansen Liu,Wenpeng Fang,Wenshan Zhang. An Energy Consumption Prediction-Based Optimization Strategy for Eco-driving of Connected Electric Buses [J]. Automotive Engineering, 2025, 47(5): 839-850.
[5]	Peicheng Shi,Runshuai Ge,Chakir Chadia,Xinlong Dong,Taonian Liang,Aixi Yang. PolarDet: An End-to-End 3D Object Detection Algorithm in Polar Coordinates Based on Position and Semantic Information Weighting [J]. Automotive Engineering, 2025, 47(3): 430-439.
[6]	Bing Zhu,Shizheng Jia,Jian Zhao,Jiayi Han,Peixing Zhang,Dongjian Song,Zhicheng Chen. Human-Like Decision-Making Based on Sequential Games for Automated Vehicles Considering Subjective Cognition [J]. Automotive Engineering, 2025, 47(1): 13-22.
[7]	Jiahui Liu,Yuan Zou,Wei Sun,Yihao Meng,Xiaoran Lu,Yuanyuan Li. ICV Task Offloading and Resource Allocation Based on Hybrid Deep Reinforcement Learning [J]. Automotive Engineering, 2025, 47(1): 35-43.
[8]	Yang Luo,Ping Liu,Mingjie Liu,Changhao Piao,Peng Yuan,Kailin Wan. Decision System Design of Personalized Thermal Comfortable Intelligent Air Conditioning for Automobile Passenger Cabin [J]. Automotive Engineering, 2024, 46(6): 1114-1124.
[9]	Fuxing Yao,Chao Sun,Yungang Lan,Bing Lu,Bo Wang,Haiyang Yu. A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model [J]. Automotive Engineering, 2024, 46(5): 882-892.
[10]	Guodong Du,Yuan Zou,Xudong Zhang,Wenjing Sun,Wei Sun. Research on Automatic Driving Motion Control Based on Double Estimator Reinforcement Learning Combined with Forward Predictive Control [J]. Automotive Engineering, 2024, 46(4): 564-576.
[11]	Xinke Fu,Yingfeng Cai,Long Chen,Hai Wang,Qingchao Liu. Decision-Making for Autonomous Driving in Uncertain Environment [J]. Automotive Engineering, 2024, 46(2): 211-221.
[12]	Zhongjie Liu,Zhiguo Zhao,Qin Yu. Vehicle Assisted Driving Behavior Decision-Making Based on Dynamic Risk Assessment [J]. Automotive Engineering, 2024, 46(11): 2005-2016.
[13]	Xiaohu Geng,Yao Fu,Jie Wang,Yulong Lei,Weidong Liu,Yuhai Wang,Ke Liu. Predictive Cruise Control for Commercial Vehicles Considering Different Time Domains [J]. Automotive Engineering, 2024, 46(11): 2046-2058.
[14]	Shengbo Eben Li,Guojian Zhan,Yuxuan Jiang,Zhiqian Lan,Yuhang Zhang,Wenjun Zou,Chen Chen,Bo Cheng,Keqiang Li. Key Technologies of Brain-Inspired Decision and Control Intelligence for Autonomous Driving Systems [J]. Automotive Engineering, 2023, 45(9): 1499-1515.
[15]	Zuliang Jin,Hanbing Wei,Liu Zheng,Lu Lou,Guofeng Zheng. Lightweight Semantic Segmentation Method Based on Local Window Cross Attention [J]. Automotive Engineering, 2023, 45(9): 1617-1625.