基于复杂网络理论和安全经验回放机制的强化学习自动驾驶方法研究

doi:10.19562/j.chinasae.qcgc.2026.03.003

摘要/Abstract

摘要：

驾驶安全一直是自动驾驶领域的首要任务。近年来智能汽车面临的驾驶环境日益复杂，为了提高智能汽车面对复杂环境的认知能力以及驾驶策略的安全性，本文提出了一种知识数据融合驱动的强化学习算法。首先，将动态驾驶环境抽象为复杂网络风险认知域模型，实现了车辆节点间交互关系的有效刻画。其次，提出了一种安全经验回放机制，充分地挖掘数据中的信息。最后，提出了一种基于安全经验回放机制的强化学习算法，在Actor-Critic算法框架下增加了一个安全性评估模块，并将风险认知域形成的驾驶建议融入强化学习算法的训练过程。实验结果表明，在Carla Leaderboard基准测试中，本文算法的驾驶分数和成功率分别提升至87%和81%，有效提升了自动驾驶系统的安全性。

关键词: 自动驾驶, 深度强化学习, 复杂网络, 安全经验回放

Abstract:

Driving safety has always been the primary concern in the field of autonomous driving. In recent years， intelligent vehicles have faced increasingly complex driving environment. To enhance the cognitive capabilities in such scenarios and improve the safety of driving strategies， a knowledge-data fusion-driven reinforcement learning algorithm is proposed in this paper. Firstly， the dynamic driving environment is abstracted into a complex network-based risk cognition domain model， effectively capturing the interactive relationship among vehicle nodes. Secondly， a safety-enhanced experience replay mechanism is introduced to fully exploit the information within the data. Finally， a reinforcement learning algorithm based on the safety-aware experience replay mechanism is proposed. Within the Actor-Critic framework， a safety evaluation module is incorporated， and driving recommendation derived from the risk cognition domain is integrated into the reinforcement learning training process. The experimental results show the proposed method achieves an 87% driving score and 81% success rate on the CARLA Leaderboard， improving autonomous driving safety.

Key words: autonomous driving, deep reinforcement learning, complex network, safe experience replay

闫辉,蔡英凤,孙晓强,王海,陈龙,张晓东. 基于复杂网络理论和安全经验回放机制的强化学习自动驾驶方法研究[J]. 汽车工程, 2026, 48(3): 518-528.

Hui Yan,Yingfeng Cai,Xiaoqiang Sun,Hai Wang,Long Chen,Xiaodong Zhang. Reinforcement Learning-Based Autonomous Driving Method Using Complex Network Theory and Safe Experience Replay Mechanism[J]. Automotive Engineering, 2026, 48(3): 518-528.

图/表 15

图1

图2

图3

图4

图5

图6

表1

表2

表3

超参数设置"

超参数	数值
学习率	0.000 01
总步数	6 144
小批次容量	128
轮次	20
$λ p p o$	0.5
$λ e x p$	0.01
$λ r i s k$	0.05
PPO截断系数	0.2

表3

表4

表5

图7

表6

图8

图9

参考文献 35

[1]	LI Y， QI H， ZHU F， et al. Interpretable autonomous driving model based on cognitive reinforcement learning［C］. 2024 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2024： 515-520.
[2]	HE X， WU J， HUANG Z， et al. Fear-neuro-inspired reinforcement learning for safe autonomous driving［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 46（1）： 267-279.
[3]	COELHO D， OLIVEIRA M. A review of end-to-end autonomous driving in urban environments［J］. IEEE Access， 2022， 10： 75296-75311.
[4]	CHU Duanfeng， WANG Rukang， WANG Jingyi， et al. End-to-end autonomous driving： advancements and challenges［J］. China Journal of Highway and Transport， 2024， 37（10）： 209-232.
[5]	BÖTTCHER L， PORTER M A. Complex networks with complex weights［J］. Physical Review E， 2024， 109（2）： 024314.
[6]	ZHENG Y， WU Y， JIANG G P. Exploring synchronizability of complex dynamical networks from edges perspective［J］. Physica A： Statistical Mechanics and Its Applications， 2024， 638： 129660.
[7]	UNICOMB S， IÑIGUEZ G， GLEESON J P， et al. Dynamics of cascades on burstiness-controlled temporal networks［J］. Nature Communications， 2021， 12（1）： 133.
[8]	LIU Y Y， SLOTINE J J， BARABÁSI A L. Controllability of complex networks［J］. Nature， 2011， 473（7346）： 167-173.
[9]	BAGGIO G， BASSETT D S， PASQUALETTI F. Data-driven control of complex networks［J］. Nature Communications， 2021， 12（1）： 1429.
[10]	ZHANG T， LIU H， WANG W， et al. Virtual tools for testing autonomous driving： a survey and benchmark of simulators， datasets， and competitions［J］. Electronics， 2024， 13（17）： 3486.
[11]	DOLGOV D， THRUN S， MONTEMERLO M， et al. Practical search techniques in path planning for autonomous driving［C］. Search in Artificial Intelligence and Robotics.American Association for Artificial Intelligence （AAAI）， 2009：32-37.
[12]	URMSON C， ANHALT J， BAGNELL D， et al. Autonomous driving in urban environments： boss and the urban challenge［J］. Journal of Field Robotics， 2008， 25（8）： 425-466.
[13]	TOROMANOFF M， WIRBEL E， MOUTARDE F. End-to-end model-free reinforcement learning for urban driving using implicit affordances［C］. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition， 2020： 7153-7162.
[14]	CHEN J， LI S E， TOMIZUKA M. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 23（6）： 5068-5078.
[15]	WANG H， XU Y， WANG Z， et al. Centernet-auto： a multi-object visual detection algorithm for autonomous driving scenes based on improved centernet［J］. IEEE Transactions on Emerging Topics in Computational Intelligence， 2023， 7（3）： 742-752.
[16]	LI Y， FENG F， CAI Y， et al. Localization for intelligent vehicles in underground car parks based on semantic information［J］. IEEE Transactions on Intelligent Transportation Systems， 2023， 25（2）： 1317-1332.
[17]	CULTRERA L， BECATTINI F， SEIDENARI L， et al. Addressing limitations of state-aware imitation learning for autonomous driving［J］. IEEE Transactions on Intelligent Vehicles， 2023， 9（1）： 2946-2955.
[18]	YANG X， JI Z， WU J， et al. Recent advances of deep robotic affordance learning： a reinforcement learning perspective［J］. IEEE Transactions on Cognitive and Developmental Systems， 2023， 15（3）： 1139-1149.
[19]	ARULKUMARAN K， DEISENROTH M P， BRUNDAGE M， et al. Deep reinforcement learning： a brief survey［J］. IEEE Signal Processing Magazine， 2017， 34（6）： 26-38.
[20]	SHARIF A， MARIJAN D. Evaluating the robustness of deep reinforcement learning for autonomous policies in a multi-agent urban driving environment［C］. 2022 IEEE 22nd International Conference on Software Quality， Reliability and Security （QRS）. IEEE， 2022： 785-796.
[21]	GARCIA J， FERNÁNDEZ F. A comprehensive survey on safe reinforcement learning［J］. Journal of Machine Learning Research， 2015， 16（1）： 1437-1480.
[22]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：， 2017.
[23]	CODEVILLA F， SANTANA E， LÓPEZ A M， et al. Exploring the limitations of behavior cloning for autonomous driving［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 9329-9338.
[24]	WANG Jianqiang， WU Jian， LI Yang. The driving safety field based on driver-vehicle-road interactions ［J］. IEEE Transactions on Intelligent Transportation Systems， 2015， 16（4）： 2203-2214.
[25]	BANSAL M， KRIZHEVSKY A， OGALE A. Chauffeurnet： learning to drive by imitating the best and synthesizing the worst［J］. arXiv preprint arXiv：， 2018.
[26]	HOU C， ZHANG W. End-to-end urban autonomous driving with safety constraints［J］. IEEE Access， 2024， 12， 132198-132209.
[27]	HART P E， NILSSON N J， RAPHAEL B. A formal basis for the heuristic determination of minimum cost paths［J］. IEEE Transactions on Systems Science and Cybernetics， 1968， 4（2）： 100-107.
[28]	RHINEHART N， MCALLISTER R， LEVINE S. Deep imitative models for flexible inference， planning， and control［J］. arXiv preprint arXiv：， 2018.
[29]	HUANG X， CHENG Y， YU Q， et al. Deep reinforcement learning for autonomous driving based on safety experience replay［J］. IEEE Transactions on Cognitive and Developmental Systems， 2024，16（6）： 2070-2084.
[30]	SCHULMAN J， MORITZ P， LEVINE S， et al. High-dimensional continuous control using generalized advantage estimation［J］. arXiv preprint arXiv：， 2015.
[31]	CAI Y， LU S， WANG H， et al. Complex network cognition-based federated reinforcement learning for end-to-end urban autonomous driving［J］. IEEE Transactions on Transportation Electrification， 2023， 10（3）： 7513-7525.
[32]	DOSOVITSKIY A， ROS G， CODEVILLA F， et al. CARLA： an open urban driving simulator［C］. Conference on Robot Learning. PMLR， 2017： 1-16.
[33]	CHEN D， ZHOU B， KOLTUN V， et al. Learning by cheating［C］. Conference on Robot Learning. PMLR， 2020： 66-75.
[34]	CHEN D， KOLTUN V， KRÄHENBÜHL P. Learning to drive from a world on rails［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15590-15599.
[35]	ZHANG Z， LINIGER A， DAI D， et al. End-to-end urban driving by imitating a reinforcement learning coach［C］. Proceedings of the IEEE/CVF International Conference on Computer Vision， 2021： 15222-15232.

条件	数值
闯红灯	-1-s
无视停车标志	-1-s
和其他物体碰撞	-1-s
被堵塞	-1
其他	0

测试基准	地图	汽车数量	行人数量
Nocrash-dense	Town01	100	250
Nocrash-dense	Town02	70	150
LeaderBoard	Town01	120	120
	Town02	70	70
	Town03	70	70
	Town04	150	80
	Town05	120	120
	Town06	120	80
天气组别	天气
组1	晴天正午，晴天日落
组2	小雨日落，大雨日落

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn	LB-all
Roach	95±2	95±3	91±3	90±2	85±3
AP	83±1	87±2	71±3	72±2	78±2
SCPPO	97±2	94±2	93±3	91±1	87±3

方法	NCd-tt	NCd-tn	NCd-nt	NCd-nn	LB-all
LBC	86±3	83±6	60±3	59±8	N/A
Rails	96±0	74±0	84±0	66±0	N/A
Roach	91±4	90±7	83±3	83±3	72±6
AP	95±2	94±3	80±3	80±7	73±4
SCPPO	95±1	95±4	84±2	84±5	81±4

SER	RC	SR	CP	CV
		66±2	0.384±1.495	2.697±1.783
√		79±3	0.163±0.627	1.429±0.451
	√	80±5	0.104±0.528	0.994±0.337
√	√	84±2	0.037±0.025	0.119±0.045