自动驾驶车辆轨迹跟踪避撞的扩散强化学习方法研究

doi:10.19562/j.chinasae.qcgc.2025.08.006

Abstract

Abstract:

The intelligence of autonomous vehicles is key to upgrading of the automotive industry， where trajectory tracking and collision avoidance technologies are crucial for ensuring vehicle safety. In this paper， for the problem of insufficient exploration of existing reinforcement learning control methods， a diffusion reinforcement learning algorithm is proposed. By combining diffusion models with reinforcement learning frameworks and replacing traditional policy networks with diffusion generative policy networks， introducing the multimodal distribution matching capability of diffusion models into reinforcement learning， and combining it with the distributional soft actor-critic algorithm， a diffusion distributional actor-critic algorithm （DDAC） is proposed. Simulation and real-vehicle experiments demonstrate that the proposed algorithm exhibits high exploration efficiency， with real vehicle lateral tracking error less than 0.03 m and velocity tracking error less than 0.05 m/s， verifying the superiority of the algorithm.

Key words: trajectory tracking, active collision avoidance, distributional reinforcement learning, diffusion model

Junjie Zhao,Yinuo Wang,Jiang Wu,Sichao Wu,Changdi Zou,Hongda Wang,ShengboEben Li,Fei Ma,Jingliang Duan. Research on Diffusion Reinforcement Learning Method for Vehicle Trajectory Tracking and Collision Avoidance of Autonomous Vehicles[J].Automotive Engineering, 2025, 47(8): 1490-1500.

Figures/Tables 19

Network：扩散型策略网络
Input：状态 $s t$ ，总降噪步数 $N$
初始化降噪网络： $? ? a t i, s t, i$
初始化超参数： $α 、 β$
Repeat
	$a i - 1 a i = a i α i - β i α i 1 - α ˉ i ? ? a i, s, i$
until $i = 1$
Output： $a t 0$

Algorithm：DDAC算法
初始化扩散策略网络参数 $? 、值分布网络参数 θ 1 、 θ 2$
初始化值分布网络学习率 $β Z$ 、策略网络学习率 $β π$
初始化目标网络参数 $θ ˉ 1$ ← $θ 1$ ， $θ ˉ 2$ ← $θ 2$ ， $? ˉ$ ← $?$
初始化同步速率 $τ$
初始化迭代步数 $k = 0$
repeat
	repeat
		从 $π ? (a t s t)$ 中计算 $a t$
		与系统交互得到奖励 $r$ 和下一个状态 $s t + 1$
		更新 $T' = s, a, r, s'$ 至经验池 $B$
	until 采样完成
	repeat
		从 $B$ 中采样数据
		从目标网络 $π ? ˉ (a t + 1 s t + 1)$ 采样 $a t + 1 0$
		更新值分布网络梯度： $θ ← θ - β Z ? θ J Z (θ)$
		更新策略网络梯度： $? ← ? + β π ? ? J π (?)$
		更新目标网络： $θ ˉ ← τ θ + 1 - τ θ ˉ$
		更新目标网络： $? ˉ ← τ ? + 1 - τ ? ˉ$
	until 收敛
until 迭代完成

参数	数值
策略网络学习率	$2 e$ -3
值网络学习率	$1 e$ -3
目标网络学习率	$2 e$ -3
折扣因子	$0.99$
优化方法经验池大小	$1 e 6$
单次迭代样本数	256
优化方法	Adam
反向降噪链系数 $β$	$0 ~ 1$
反向步数	$5$
策略分布标准差 $σ π$ 边界	$[- 20,0.5]$

算法	累计回报
DDAC （ $N$ =20）	-269±57
DDAC （ $N$ =5）	-307±24
DSAC	-272±24
DDPG	-401±51
TD3	-306±47
SAC	-356±33
PPO	-354±92
TRPO	-389±101

算法名称	平均耗时/ms	最大耗时/ms
DDAC （ $N$ =5）	2.11	3.00
DDAC （ $N$ =20）	8.46	10.00

References 30

[1]	王建，许叁征，甘浩，等. 智能汽车纵深防御关键技术及挑战［C］. 2018 中国汽车工程学会年会论文集，2018：287⁃291.
	WANG J，XU S Z，GAN H，et al. Key technologies and challenges of intelligent vehicle in-depth defense［C］. 2018 SAE-China Annual Conference Proceedings，2018：287-291.
[2]	肖礼明，张发旺，陈良发，等.依托多风格强化学习的车辆轨迹跟踪避撞控制［J］.汽车工程，2024，46（6）：945-955.
	XIAO L M，ZHANG F W，CHEN L F，et al. Vehicle trajectory tracking and collision avoidance control based on multi-style reinforcement learning［J］. Automotive Engineering，2024，46（6）， 945-955.
[3]	DUAN J， REN Y， ZHANG F， et al. Encoding distributional soft actor-critic for autonomous driving in multi-lane scenarios ［J］. IEEE Computational Intelligence Magazine， 2024， 19（2）： 96-112.
[4]	MNIH V， KAVUKCUOGLU K， SILVER D， et al. Playing atari with deep reinforcement learning［J］. arXiv preprint arXiv：， 2013.
[5]	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［J］. arXiv preprint arXiv：， 2015.
[6]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：， 2017.
[7]	MNIH V， BADIA A P， MIRZA M， et al. Asynchronous methods for deep reinforcement learning［C］. International Conference on Machine Learning. PMLR， 2016： 1928-1937.
[8]	HAARNOJA T， ZHOU A， ABBEEL P， et al. Soft actor-critic： off-policy maximum entropy deep reinforcement learning with a stochastic actor［C］.International Conference on Machine Learning. PMLR， 2018： 1861-1870.
[9]	DUAN J， GUAN Y， LI S E， et al. Distributional soft actor-critic： off-policy reinforcement learning for addressing value estimation errors［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021， 33（11）： 6584-6598.
[10]	DUAN J， WANG W， XIAO L， et al. DSAC-T： distributional soft actor-critic with three refinements［J］. arXiv preprint arXiv：， 2023.
[11]	YANG L， HUANG Z， LEI F， et al. Policy representation via diffusion probability model for reinforcement learning［J］. arXiv preprint arXiv：， 2023.
[12]	KANG B， MA X， DU C， et al. Efficient diffusion policies for offline reinforcement learning［J］. Advances in Neural Information Processing Systems， 2024， 36.
[13]	ARENZ O， NEUMANN G， ZHONG M. Efficient gradient-free variational inference using policy search［C］.International Conference on Machine Learning. PMLR， 2018： 234-243.
[14]	TANG Y， AGRAWAL S. Boosting trust region policy optimization by normalizing flows policy［J］. arXiv preprint arXiv：， 2018.
[15]	HAARNOJA T， TANG H， ABBEEL P， et al. Reinforcement learning with deep energy-based policies［C］. International Conference on Machine Learning. PMLR， 2017： 1352-1361.
[16]	CROITORU F A， HONDRU V， IONESCU R T， et al. Diffusion models in vision： a survey［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023.
[17]	SONG Y， DURKAN C， MURRAY I， et al. Maximum likelihood training of score-based diffusion models［J］. Advances in Neural Information Processing Systems， 2021， 34： 1415-1428.
[18]	DHARIWAL P， NICHOL A. Diffusion models beat gans on image synthesis［J］. Advances in Neural Information Processing Systems， 2021， 34： 8780-8794.
[19]	SOHL-DICKSTEIN J， WEISS E， MAHESWARANATHAN N， et al. Deep unsupervised learning using nonequilibrium thermodynamics［C］. International Conference on Machine Learning. PMLR， 2015： 2256-2265.
[20]	HO J， JAIN A， ABBEEL P. Denoising diffusion probabilistic models［J］. Advances in Neural Information Processing Systems， 2020， 33： 6840-6851.
[21]	WANG Z， HUNT J J， ZHOU M. Diffusion policies as an expressive policy class for offline reinforcement learning［J］. arXiv preprint arXiv：， 2022.
[22]	AJAY A， DU Y， GUPTA A， et al. Is conditional generative modeling all you need for decision-making？［J］. arXiv preprint arXiv：， 2022.
[23]	CHEN Y， LI H， ZHAO D. Boosting continuous control with consistency policy［J］. arXiv preprint arXiv：， 2023.
[24]	CHI C， FENG S， DU Y， et al. Diffusion policy： visuomotor policy learning via action diffusion［J］. arXiv preprint arXiv：， 2023.
[25]	CODEVILLA F， SANTANA E， LÓPEZ A M， et al. Exploring the limitations of behavior cloning for autonomous driving［C］.Proceedings of the IEEE/CVF International Conference on Computer Vision， 2019： 9329-9338.
[26]	LY A O， AKHLOUFI M. Learning to drive by imitation： an overview of deep behavior cloning methods［J］. IEEE Transactions on Intelligent Vehicles， 2020， 6（2）： 195-209.
[27]	PSENKA M， ESCONTRELA A， ABBEEL P， et al. Learning a diffusion model policy from rewards via Q-score matching［J］. arXiv preprint arXiv：， 2023.
[28]	XIAO Z， KREIS K， VAHDAT A. Tackling the generative learning trilemma with denoising diffusion gans［J］. arXiv preprint arXiv：， 2021.
[29]	WANG W， ZHANG Y， GAO J， et al. GOPS： a general optimal control problem solver for autonomous driving and industrial control applications［J］. Communications in Transportation Research， 2023， 3： 100096.
[30]	SCHULMAN J， WOLSKI F， DHARIWAL P， et al. Proximal policy optimization algorithms［J］. arXiv preprint arXiv：， 2017.

参数	数值
自身质量	68 kg
外形大小	760 mm×520 mm×210 mm
负载能力	250 kg
旋转半径	387 mm
轮胎半径	150 mm
车轮间距	423 mm

Research on Diffusion Reinforcement Learning Method for Vehicle Trajectory Tracking and Collision Avoidance of Autonomous Vehicles

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 19

References 30

Related Articles 15

Metrics

Comments

Recommended 0

[1]	Mingchen Wang,Hai Wang,Yingfeng Cai,Long Chen,Yicheng Li. MSF-Diffuser: A Multi-sensor Adaptive Fusion Autonomous Driving Method Based on Diffusion Model Under BEV [J]. Automotive Engineering, 2025, 47(6): 1122-1132.
[2]	Yusheng Dai,Yuan Chang,Zeyu Yang,Bowei Zhang,Manjiang Hu,Jin Huang. Trajectory Tracking Constraint-Following Control for Dual-Vehicle Cooperative Transportation Systems [J]. Automotive Engineering, 2025, 47(4): 658-668.
[3]	Kegang Zhao,Weilin Ou,Zheng Zhang,Zhihao Liang. A Tuner of Trajectory Control Parameters and the Construction Method of its Training Set [J]. Automotive Engineering, 2025, 47(2): 248-258.
[4]	Zhengcai Yang,Huiquan Zhang,Linhe Ge,Tianjun Sun. Trajectory Tracking Control Method for Autonomous Vehicles Considering Time-Varying Reference and Steering Delay [J]. Automotive Engineering, 2025, 47(1): 44-54.
[5]	Peng Chen,Yingfeng Cai,Haibo Yuan,Long Chen,Xiaoqiang Sun. Research on Semi-Trailer Trajectory Tracking Based on Type-2 Fuzzy Logic Control [J]. Automotive Engineering, 2025, 47(1): 55-66.
[6]	Liming Xiao,Fawang Zhang,Liangfa Chen,Haoqi Yan,Fei Ma,Shengbo Eben Li,Jingliang Duan. Vehicle Trajectory Tracking and Collision Avoidance Control Based on Multi-style Reinforcement Learning [J]. Automotive Engineering, 2024, 46(6): 945-955.
[7]	Shuen Zhao,Sheng Wang,Yao Leng. Multi-objective Explicit Model Predictive Control for Intelligent Vehicle Trajectory Tracking [J]. Automotive Engineering, 2024, 46(5): 784-794.
[8]	Shaobo Lu,Lingfeng Dai,Chenhui Wang,Bingjun Liu,Zhigang Chu,Wenke Xie. Trajectory Planning and Control of Autonomous Vehicle Under Extreme Conditions Based on Autonomous Drift [J]. Automotive Engineering, 2024, 46(10): 1780-1789.
[9]	Zhiqun Yuan,Yanqiang Chen,Yuxuan Chang,Diansheng Huo,Li Lin. Model Predictive Control with Adaptive Horizon for Vehicle Trajectory Tracking Considering Crosswind Stability [J]. Automotive Engineering, 2024, 46(10): 1829-1841.
[10]	Yunfei Zha,Lü Xiaolong,Huiqin Chen,Yingchun Yi,Yanyan Wang. Vehicle Trajectory Tracking Control Based on Road Adhesion Coefficient Estimation [J]. Automotive Engineering, 2023, 45(6): 1010-1021.
[11]	Yinghong Yu,Li Huang,Yinong Li,Ling Zheng,Jia Zhou,Yixiao Liang. Research on Emergency Trajectory Tracking Control Based on Dynamics Decoupling [J]. Automotive Engineering, 2023, 45(6): 997-1009.
[12]	Yu Zhang,Mingfan Xu,Guangyu Bai,Mingming Dong,Li Gao,Yechen Qin. Intelligent Vehicle Switching Control Considering Dynamic Stability Constraints [J]. Automotive Engineering, 2023, 45(5): 709-718.
[13]	Bin Zhang,Yuan Zou,Xudong Zhang,Fengchun Sun,Zhe Wu,Yihao Meng. Research on Trajectory Tracking Control of Hybrid Tracked Unmanned Platform [J]. Automotive Engineering, 2023, 45(4): 579-587.
[14]	Qin Li,Jianming Tang,Boyuan Zhang,Yong Chen,Yong Wang. Research on Fault-Tolerant Control of Multi-Actuator for Distributed Drive Electric Vehicles [J]. Automotive Engineering, 2023, 45(12): 2251-2259.
[15]	Daofei Li,Anfei Zha,Biao Xu,Jiajie Zhang. Trajectory Tracking Control Algorithm of Emergency Collision Avoidance for Tractor Semi-trailer Combination [J]. Automotive Engineering, 2022, 44(7): 1098-1106.