依托多风格强化学习的车辆轨迹跟踪避撞控制

doi:10.19562/j.chinasae.qcgc.2024.06.001

Abstract

Abstract:

Trajectory tracking and collision avoidance are key functions of vehicle intelligence. For the singular control style limitation of existing control methods in the same scene， a novel multi-style reinforcement learning （RL） method is proposed in this paper. To achieve diversity in control styles， style indicators are innovatively incorporated into value and policy networks to establish a multi-style tracking and collision avoidance policy network. Alongside this， a multi-style policy iteration framework is developed combining the distributional RL theory. Based on the framework， a multi-style distributional soft actor-critic algorithm （M-DSAC） is put forward. Through simulation and real vehicle tests， it is validated that the proposed method is capable of executing trajectory tracking and collision avoidance tasks across various driving styles， such as aggressive， neutral， and conservative， with the real vehicle’s steady-state trajectory tracking error less than 5 cm， with high control accuracy. The average single-step decision-making time for the real vehicle is merely 6.07 ms， meeting real-time requirements.

Key words: multi-style, DSAC, trajectory tracking, active collision avoidance

Liming Xiao,Fawang Zhang,Liangfa Chen,Haoqi Yan,Fei Ma,Shengbo Eben Li,Jingliang Duan. Vehicle Trajectory Tracking and Collision Avoidance Control Based on Multi-style Reinforcement Learning[J].Automotive Engineering, 2024, 46(6): 945-955.

Figures/Tables 17

Algorithm： Multi-style DSAC算法
初始化值分布网络参数θ、策略网络参数ω和策略熵系数 $α$
初始化目标网络参数θ'←θ和ω'←ω
选择合适的学习率 $β Z$ 、 $β ω$ 、 $β α$ 和 $τ$
repeat
	随机化 $λ ∈ [- m, m]$
	根据策略选择动作 $a ～ π ω (a s, λ)$ 和系统交互得到s'以及 $r$
	将数据元组 $(s, a, r, s')$ 存入经验池 $B$
	从 $B$ 中随机选择 $N$ 组数据
	利用式（15）更新值分布网络
	利用式（16）更新策略网络
	利用式（17）更新策略熵系数α
	利用式（18）更新目标网络
until 收敛

算法超参数	数值
优化方法	Adam
Batch Size	256
策略网络学习率	$2 × 10 - 3$
值网络学习率	$1 × 10 - 3$
折扣因子 $γ$	$0.99$
目标网络学习率 $τ$	$1 × 10 - 3$
策略熵 $α$ 学习率	$1 × 10 - 3$
策略熵目标值 $H ˉ$	$- 2$
风格指标系数 $λ$ 边界 $m$	100

References 36

1	李升波，关阳，侯廉，等. 深度神经网络的关键技术及其在自动驾驶领域的应用［J］. 汽车安全与节能学报， 2019， 10（2）： 119-145.
	LI S E， GUAN Y， HOU L， et al. Key technology of deep neural network and its application in the field of autonomous driving［J］. Journal of Automotive Safety and Energy Conservation， 2019， 10（2）： 119-145.
2	ZHANG P， ZHU B， ZHAO J， et al. Performance evaluation method for automated driving system in logical scenario［J］. Automotive Innovation， 2022， 5（3）： 299-310.
3	李道飞，查安飞，徐彪，等. 半挂汽车列车紧急避撞轨迹跟踪控制算法［J］. 汽车工程， 2022， 44 （7）： 1098-1106.
	LI D F， CHA A F， XU B， et al. Trajectory tracking control algorithm for emergency collision avoidance of semi-trailer automobile train ［J］. Automotive Engineering， 2022， 44 （7）： 1098-1106.
4	李升波，占国建，蒋宇轩，等. 类脑学习型自动驾驶决控系统的关键技术［J］. 汽车工程， 2023，45（9）： 1499-1515.
	LI S E， ZHAN G J， JIANG Y X， et al. Key technologies of brain-inspired decision and control intelligence for autonomous driving systems［J］. Automotive Engineering， 2023，45（9）： 1499-1515.
5	GUAN Y， TANG L， LI C， et al. Integrated decision and control for high-level automated vehicles by mixed policy gradient and its experiment verification［J］. arXiv preprint arXiv：， 2022.
6	LI G， ZHOU W， LIN S， et al. On-ramp merging for highway autonomous driving： an application of a new safety indicator in deep reinforcement learning［J］. Automotive Innovation， 2023， 6（3）： 453-465.
7	王建，许叁征，甘浩，等. 智能汽车纵深防御关键技术及挑战［C］. 2018 中国汽车工程学会年会论文集， 2018： 287291.
	WANG J， XU S Z， GAN H， et al. Key technologies and challenges of intelligent vehicle in-depth defense ［C］. 2018 SAE-China Annual Conference Proceedings， 2018： 287-291.
8	LIU Z， ZHANG W， ZHAO F. Impact， challenges and prospect of software-defined vehicles［J］. Automotive Innovation， 2022， 5（2）： 180-194.
9	WANG Y， CAO X， HU Y. A trajectory planning method of automatic lane change based on dynamic safety domain［J］. Automotive Innovation， 2023， 6（3）： 466-480.
10	LIANG Y， LI Y， YU Y， et al. Path-following control of autonomous vehicles considering coupling effects and multi-source system uncertainties［J］. Automotive Innovation， 2021， 4（3）： 284-300.
11	GUO N， ZHANG X， ZOU Y. Real-time predictive control of path following to stabilize autonomous electric vehicles under extreme drive conditions［J］. Automotive Innovation， 2022， 5（4）： 453-470.
12	GE Q， SARTORETTI G， DUAN J， et al. Distributed model predictive control of connected multi-vehicle systems at unsignalized intersections［C］. 2022 IEEE International Conference on Unmanned Systems （ICUS）. IEEE， 2022： 1466-1472.
13	王宏伟，刘晨宇，李磊，等. 基于高效NMPC算法的无人车轨迹跟踪控制研究［J］. 汽车工程， 2022， 44（10）： 1494-1502.
	WANG H W， LIU C Y， LI L， et al. Research on unmanned vehicle trajectory tracking control based on efficient NMPC algorithm ［J］. Automotive Engineering， 2022， 44（10）： 1494-1502.
14	LI G， ZHANG X， GUO H， et al. Real-time optimal trajectory planning for autonomous driving with collision avoidance using convex optimization［J］. Automotive Innovation， 2023： 1-11.
15	LIU Z， DUAN J， WANG W， et al. Recurrent model predictive control： learning an explicit recurrent controller for nonlinear systems［J］. IEEE Transactions on Industrial Electronics， 2022， 69（10）： 10437-10446.
16	DUAN J， LI J， GE Q， et al. Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems［J］. IEEE Transactions on Intelligent Vehicles， 2023，8（5）： 3299-3311.
17	YIN Y， LI S E， TANG K， et al. Approximate optimal filter design for vehicle system through actor-critic reinforcement learning［J］. Automotive Innovation， 2022， 5（4）： 415-426.
18	HE X， LV C. Towards safe autonomous driving： decision making with observation-robust reinforcement learning［J］. Automotive Innovation， 2023： 1-12.
19	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［C］. The 4th International Conference on Learning Representations （ICLR）. San Juan， Puerto Rico： ICLR， 2016.
20	HAARNOJA T， ZHOU A， ABBEEL P， et al. Soft actorcritic： offpolicy maximum entropy deep reinforcement learning with a stochastic actor［C］. Proceedings of the 35^th International Conference on Machine Learning （ICML）. Stockholmsmässan， Sweden： PMLR， 2018： 18611870.
21	DUAN J， GUAN Y， LI S E， et al. Distributional soft actor-critic： off-policy reinforcement learning for addressing value estimation errors［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021， 33（11）： 6584-6598.
22	PENG B， DUAN J， CHEN J， et al. Model-based chance-constrained reinforcement learning via separated proportional-integral lagrangian［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022.
23	KENDALL A， HAWKE J， JANZ D， et al. Learning to drive in a day［C］. 2019 International Conference on Robotics and Automation （ICRA）. IEEE， 2019： 8248-8254.
24	QUANTE L， ZHANG M， PREUK K， et al. Human performance in critical scenarios as a benchmark for highly automated vehicles［J］. Automotive Innovation， 2021， 4（3）： 274-283.
25	孙冬鸣，周培栋，宋雪松，等. 引入驾驶风格系数的跟车控制优化策略［J］. 汽车与新动力， 2023， 6（3）： 7-11.
	SUN D M， ZHOU P D， SONG X S， et al. Optimization strategy of vehicle control introducing driving style coefficient［J］. Automotive & New Power， 2023， 6（3）： 7-11.
26	汪选要，魏星，谢东，等. 基于权值惩罚法自适应人机协同避撞策略［J］. 科学技术与工程， 2022， 22（13）： 5463-5471.
	WANG X Y， WEI X， XIE D， et al. Adaptive human-machine cooperative collision avoidance strategy based on weight penalty method［J］. Science Technology and Engineering， 2022， 22（13）： 5463-5471.
27	LU H， LU C， YU Y， et al. Autonomous overtaking for intelligent vehicles considering social preference based on hierarchical reinforcement learning［J］. Automotive Innovation， 2022， 5（2）： 195-208.
28	GAO B， CAI K， QU T， et al. Personalized adaptive cruise control based on online driving style recognition technology and model predictive control［J］. IEEE Transactions on Vehicular Technology， 2020， 69（11）： 12482-12496.
29	LI Z， WU C， TAO P， et al. DP and DS-LCD： a new lane change decision model coupling driver’s psychology and driving style［J］. IEEE Access， 2020， 8： 132614-132624.
30	REN Y， DUAN J， LI S E， et al. Improving generalization of reinforcement learning with minimax distributional soft actor-critic［C］. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems （ITSC）. IEEE， 2020： 1-6.
31	WURMAN P R， BARRETT S， KAWAMOTO K， et al. Outracing champion Gran Turismo drivers with deep reinforcement learning［J］. Nature， 2022， 602（7896）： 223-228.
32	DABNEY W， ROWLAND M， BELLEMARE M， et al. Distributional reinforcement learning with quantile regression［C］. Proceedings of the AAAI Conference on Artificial Intelligence， 2018， 32（1）.
33	YANG Q， SIMÃO T D， TINDEMANS S H， et al. Safety-constrained reinforcement learning with a distributional safety critic［J］. Machine Learning， 2023， 112（3）： 859-887.
34	LI S E. Reinforcement learning for sequential decision and optimal control［M］. Springer， 2023.
35	WANG W， ZHANG Y， GAO J， et al. GOPS： a general optimal control problem solver for autonomous driving and industrial control applications［J］. Communications in Transportation Research， 2023， 3： 100096.
36	FUJIMOTO S， VAN HOOF H， MEGER D. Addressing function approximation error in actorcritic methods［C］. Proceedings of the 35th International Conference on Machine Learning （ICML）. Stockholmsmässan， Sweden： PMLR， 2018： 1587-1596.

参数	数值
外形尺寸/mm	760×520×210
旋转直径/mm	774
质量/kg	68
最大负载/kg	250
轮间距/mm	423
轮径/mm	150

应用场景	均值	标准差	最大值	最小值
仿真	0.52	0.12	1.02	0.37
实车	6.07	0.22	18.98	0.84

[1]	Shuen Zhao,Sheng Wang,Yao Leng. Multi-objective Explicit Model Predictive Control for Intelligent Vehicle Trajectory Tracking [J]. Automotive Engineering, 2024, 46(5): 784-794.
[2]	Yunfei Zha,Lü Xiaolong,Huiqin Chen,Yingchun Yi,Yanyan Wang. Vehicle Trajectory Tracking Control Based on Road Adhesion Coefficient Estimation [J]. Automotive Engineering, 2023, 45(6): 1010-1021.
[3]	Yinghong Yu,Li Huang,Yinong Li,Ling Zheng,Jia Zhou,Yixiao Liang. Research on Emergency Trajectory Tracking Control Based on Dynamics Decoupling [J]. Automotive Engineering, 2023, 45(6): 997-1009.
[4]	Yu Zhang,Mingfan Xu,Guangyu Bai,Mingming Dong,Li Gao,Yechen Qin. Intelligent Vehicle Switching Control Considering Dynamic Stability Constraints [J]. Automotive Engineering, 2023, 45(5): 709-718.
[5]	Bin Zhang,Yuan Zou,Xudong Zhang,Fengchun Sun,Zhe Wu,Yihao Meng. Research on Trajectory Tracking Control of Hybrid Tracked Unmanned Platform [J]. Automotive Engineering, 2023, 45(4): 579-587.
[6]	Qin Li,Jianming Tang,Boyuan Zhang,Yong Chen,Yong Wang. Research on Fault-Tolerant Control of Multi-Actuator for Distributed Drive Electric Vehicles [J]. Automotive Engineering, 2023, 45(12): 2251-2259.
[7]	Daofei Li,Anfei Zha,Biao Xu,Jiajie Zhang. Trajectory Tracking Control Algorithm of Emergency Collision Avoidance for Tractor Semi-trailer Combination [J]. Automotive Engineering, 2022, 44(7): 1098-1106.
[8]	Peijun Fang,Yingfeng Cai,Long Chen,Yubo Lian,Hai Wang,Yilin Zhong,Xiaoqiang Sun. Trajectory Tracking Control Method Based on Vehicle Dynamics Hybrid Model for Intelligent Vehicle [J]. Automotive Engineering, 2022, 44(10): 1469-1483.
[9]	Shaobo Lu,Feifei Xie,Bohan Zhang,Jiafeng Lu,Caixia Li. Human-Vehicle Cooperative Game Collision Avoidance Based on Asymmetric Potential Fields [J]. Automotive Engineering, 2022, 44(10): 1484-1493.
[10]	Hongwei Wang,Chenyu Liu,Lei Li,Haotian Zhang. Research on Trajectory Tracking Control of Unmanned Vehicle Based on Efficient NMPC Algorithm [J]. Automotive Engineering, 2022, 44(10): 1494-1502.
[11]	Anjie Wang,Ling Zheng,Yinong Li,Kan Wang. Motion Planning for Active Collision Avoidance of Intelligent Vehicles Based on Predictive Risk Field [J]. Automotive Engineering, 2021, 43(7): 1096-1104.
[12]	Guochen Niu,Wenshuai Li,Hongxu Wei. Intelligent Vehicle Lane Changing Trajectory Planning Based on Double Quintic Polynomials [J]. Automotive Engineering, 2021, 43(7): 978-986.
[13]	Zhiyong Zhang,Kai Long,Ronghua Du,Caixia Huang. Trajectory Tracking Coordinated Control for Autonomous Vehicle in High⁃speed Overtaking [J]. Automotive Engineering, 2021, 43(7): 995-1004.
[14]	Xingjia Li,Jianfen Li,Min Zhu,Nengling Peng,Shuai Zuo. Research on Positioning Fusion and Verification Algorithm Based on UKF [J]. Automotive Engineering, 2021, 43(6): 825-832.
[15]	Long Chen,Kai Zou,Yingfeng Cai,Chenglong Teng,Xiaoqiang Sun,Hai Wang. Longitudinal and Lateral Comprehensive Trajectory Tracking Control of Intelligent Vehicles Based on NMPC [J]. Automotive Engineering, 2021, 43(2): 153-161.

Vehicle Trajectory Tracking and Collision Avoidance Control Based on Multi-style Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 36

Related Articles 15

Metrics

Comments

Recommended 10