基于双估计强化学习结合前向预测控制的自动驾驶运动控制研究

doi:10.19562/j.chinasae.qcgc.2024.04.002

Abstract

Abstract:

Motion control research is an important part to achieve the goal of autonomous driving. To solve the problem of suboptimal control sequence due to the limitation of single-step decision in traditional reinforcement learning algorithm， a motion control framework based on the combination of double estimator reinforcement learning algorithm and forward predictive control method （DEQL-FPC） is proposed. In this framework， double estimators are introduced to solve the problem of action overestimation of traditional reinforcement learning methods and improve the speed of optimization. The forward predictive multi-step decision making method is designed to replace the single step decision making of traditional reinforcement learning so as to effectively improve the performance of global control strategies. Through virtual driving environment simulation， the superiority of the control framework applied in path tracking and safe obstacle avoidance of autonomous vehicles is proved， and the accuracy， safety， rapidity and comfort of motion control are guaranteed.

Key words: autonomous vehicle, motion control optimization, double estimator reinforcement learning algorithm, forward predictive control method

Guodong Du,Yuan Zou,Xudong Zhang,Wenjing Sun,Wei Sun. Research on Automatic Driving Motion Control Based on Double Estimator Reinforcement Learning Combined with Forward Predictive Control[J].Automotive Engineering, 2024, 46(4): 564-576.

Figures/Tables 19

References 20

1	HAYDARI A， YILMAZ Y. Deep reinforcement learning for intelligent transportation systems： a survey［J］. IEEE Transactions on Intelligent Transportation Systems， 2020， 23（1）： 11-32.
2	李克强，戴一凡，李升波，等. 智能网联汽车（ ICV ）技术的发展现状及趋势［J］. 汽车安全与节能学报， 2017， 8（1）： 1-14.
	LI K Q， DAI Y F， LI S B， et al. State-of-the-art and technical trends of intelligent and connected vehicles［J］. Journal of Automotive Safety and Energy， 2017， 8（1）： 1-14.
3	金立生，韩广德，谢宪毅，等. 基于强化学习的自动驾驶决策研究综述［J］. 汽车工程， 2023， 45（4）： 527-540.
	JIN L S， HAN G D， XIE X Y， et al. Review of autonomous driving decision-making research based on reinforcement learning［J］. Automotive Engineering， 2023， 45（4）： 527-540.
4	HAN G， FU W， WANG W， et al. The lateral tracking control for the intelligent vehicle based on adaptive PID neural network［J］. Sensors， 2017， 17（6）： 1244.
5	潘世举，李永乐，李子先，等. 基于改进纯跟踪的智能车路径跟随方法［J］. 汽车工程， 2023， 45（1）： 1-8，19.
	PAN S J， LI Y L， LI Z X， et al. Path following method of intelligent vehicles based on improved pure tracking［J］. Automotive Engineering， 2023， 45（1）： 1-8，19.
6	胡杰，钟鑫凯，陈瑞楠，等. 基于模糊LQR的智能汽车路径跟踪控制［J］. 汽车工程， 2022， 44（1）： 17-25，43.
	HU J， ZHONG X K， CHEN R N， et al. Path tracking control of intelligent vehicles based on fuzzy LQR［J］. Automotive Engineering， 2022， 44（1）： 17-25，43.
7	TANG L， YAN F， ZOU B， et al. An improved kinematic model predictive control for high-speed path tracking of autonomous vehicles［J］. IEEE Access， 2020， 8： 51400-51413.
8	杨博，张缓缓，江忠顺. 基于模型预测控制的车辆避障路径跟踪控制仿真研究［J］. 智能计算机与应用， 2020， 10（12）： 99-103.
	YANG B， ZHANG H H， JIANG Z S. Simulation of vehicle obstacle avoidance path tracking control based on model predictive control［J］. Intelligent Computer and Applications， 2020， 10（12）： 99-103.
9	李骏，万文星，郝三强，等. 复杂路况下无人驾驶路径跟踪模型预测控制研究［J］. 汽车工程， 2022， 44（5）： 664-674.
	LI J， WAN W X， HAO S Q， et al. Research on model predictive control of autonomous vehicle path tracking under complex road condition［J］. Automotive Engineering， 2022， 44（5）： 664-674.
10	GAO F， HAN Y， DANG D. Balancing accuracy and efficiency： fast motion planning based on nonlinear model predictive control［C］. Proceedings of the 2021 5th CAA International Conference on Vehicular Control and Intelligence， 2021： 3074-3089.
11	石贞洪，江洪，于文浩，等. 适用于路径跟踪控制的自适应MPC算法研究［J］. 计算机工程与应用， 2020， 56（21）： 266-271.
	SHI Z H， JIANG H， YU W H， et al. Research on adaptive MPC algorithm for path tracking control［J］. Computer Engineering and Applications， 2020， 56（21）： 266-271.
12	SHAN Y， ZHENG B， CHEN L， et al. A reinforcement learning-based adaptive path tracking approach for autonomous driving［J］. IEEE Transactions on Vehicular Technology， 2020， 69（10）： 10581-10595.
13	梁旺，秦兆博，陈亮，等. 基于改进BP神经网络的智能车纵向控制方法［J］.汽车工程， 2022， 44（8）： 1162-1172.
	LIANG W， QIN Z B， CHEN L， et al. Longitudinal control method of intelligent vehicles based on the improved BP neural network［J］. Automotive Engineering， 2022， 44（8）： 1162-1172.
14	DING L， LI S， GAO H， et al. Adaptive partial reinforcement learning neural network-based tracking control for wheeled mobile robotic systems［J］. IEEE Transactions on Systems， Man， and Cybernetics： Systems， 2020， 50（7）： 2512-2523.
15	ZHANG L， ZHANG R， WU T， et al. Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles［J］. IEEE Transactions on Neural Networks and Learning Systems， 2021，（32）12： 5435-5444.
16	WANG X， KRASOWSKI H， ALTHOFF M. CommonRoad-RL： a configurable reinforcement learning environment for motion planning of autonomous vehicles［C］. 2021 IEEE Intelligent Transportation Systems Conference， 2021： 466-472.
17	兰凤崇，刘迎节，陈吉清，等. 基于动态博弈算法的切入场景下自动驾驶车辆运动规划研究［J］. 汽车工程， 2023， 45（1）： 9-19.
	LAN F C， LIU Y J， CHEN J Q， et al. Study on motion planning of autonomous vehicles in cut-in scenes based on dynamic game algorithm［J］. Automotive Engineering， 2023， 45（1）： 9-19.
18	余嘉星， Arab Aliasghar，裴晓飞，等. 考虑路径平滑性和避撞稳定性的智能汽车弯道轨迹规划研究［J］. 汽车工程， 2022， 44（5）： 656-663.
	YU J X， ARAB A， PEI X F， et al. Research on cornering trajectory planning for intelligent vehicle considering trajectory smoothness and stability for collision avoidance［J］. Automotive Engineering， 2022， 44（5）： 656-663.
19	DU G， ZOU Y， ZHANG X， et al. Hierarchical motion planning and tracking for autonomous vehicles using global heuristic based potential field and reinforcement learning based predictive control［J］. IEEE Transactions on Intelligent Transportation Systems， 2023， 24（8）： 8304-8323.
20	HASSELT H. Double Q-learning［J］. Advances in Neural Information Processing Systems， 2010， 23.

参数	数值
学习率 α	0.955
折扣因子 γ	0.23
经验回放抽样数 n	30
优先级系数次方 μ	0.70
除法保护常数 σ	0.000 01
回放缓冲区容量 N	6 000
初始探索因子 ε	0.65
探索衰减率 υ	0.98
预测时域长度 h_p	6
初始预测衰减系数 λ	0.4

方法	平均跟踪误差/m	平均行驶速度/（m·s^-1）	训练时间/ 计算时间/s
PP-LP	0.97	2.00	- / 0.29
Q-learning	0.74	3.35	839.86 / 0.11
DEQL	0.37	4.17	516.37 / 0.11
DEQL-FPC	0.26	6.61	508.65 / 0.15

方法	额外避障时间/s	避障速度振荡/ （m·s^-1）	计算时间/s
PP-LP	61.34		0.43
Q-learning	50.27	1.02	0.12
DEQL	32.50	0.73	0.11
DEQL-FPC	21.36	0.56	0.15

方法	平均跟踪误差/m	平均行驶速度/（m·s^-1）	训练时间/ 计算时间/s
PP-LP	0.86	2.00	- / 0.27
Q-learning	0.71	3.12	525.93 / 0.11
DEQL	0.39	4.04	369.18 / 0.11
DEQL-FPC	0.30	5.35	366.41 / 0.14

方法	额外避障时间/s	避障速度振荡/ （m·s^-1）	计算时间/s
PP-LP	57.18		0.49
Q-learning	46.97	1.33	0.11
DEQL	30.61	0.92	0.11
DEQL-FPC	21.82	0.70	0.15

Research on Automatic Driving Motion Control Based on Double Estimator Reinforcement Learning Combined with Forward Predictive Control

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 19

References 20

Related Articles 15

Metrics

Comments

Recommended 0

[1]	Xinke Fu,Yingfeng Cai,Long Chen,Hai Wang,Qingchao Liu. Decision-Making for Autonomous Driving in Uncertain Environment [J]. Automotive Engineering, 2024, 46(2): 211-221.
[2]	Cheng Lin, Bowen Wang, Lü Peiyuan, Xinle Gong, Xiao Yu. Research on Motion Planning and Cooperative Control for Autonomous Vehicles with Lane Change Gaming Maneuvers Under the Curved Road [J]. Automotive Engineering, 2023, 45(7): 1099-1111.
[3]	Yanyan Chen,Hai Wang,Yingfeng Cai,Long Chen,Yicheng Li. Efficient Automatic Driving Instance Segmentation Method Based on Detection [J]. Automotive Engineering, 2023, 45(4): 541-550.
[4]	Ziwei Zhang,Ling Zheng,Yinong Li,Xuqiang Qiao,Hao Zheng,Kan Wang. A Multi-objective Adaptive Cruise Control Strategy for Autonomous Vehicle Considering Uncertain Movements of Preceding Vehicle [J]. Automotive Engineering, 2023, 45(3): 361-371.
[5]	Xinrong Zhang,Yuhang Tan,Yifan Jia,Jin Huang,Quanning Xu. Robust Control of Path Tracking for Four-Wheel Independent Drive Electric Vehicles [J]. Automotive Engineering, 2023, 45(2): 253-262.
[6]	Lü Ying,Xu Qi,Qiuzheng Liu,Xinyu Wang,Guoying Chen. Path Tracking Control Method with Steering Lag for Autonomous Vehicles [J]. Automotive Engineering, 2023, 45(12): 2234-2241.
[7]	Zhengfa Liu,Ya Wu,Peigen Liu,Rongqi Gu,Guang Chen. Cross-Domain Object Detection for Intelligent Driving Based on Joint Distribution Matching of Features and Labels [J]. Automotive Engineering, 2023, 45(11): 2082-2091.
[8]	Long Chen,Chen Yang,Yingfeng Cai,Hai Wang,Yicheng Li. Pedestrian Crossing Intention Prediction Method Based on Multimodal Feature Fusion [J]. Automotive Engineering, 2023, 45(10): 1779-1790.
[9]	Fengchong Lan,Yingjie Liu,Jiqing Chen,Zhaolin Liu. Study on Motion Planning of Autonomous Vehicles in Cut-in Scenes Based on Dynamic Game Algorithm [J]. Automotive Engineering, 2023, 45(1): 9-19.
[10]	Jun Li,Wenxing Wan,Sanqiang Hao,Wu Qin,Feifei Liu. Research on Model Predictive Control of Autonomous Vehicle Path Tracking Under Complex Road Condition [J]. Automotive Engineering, 2022, 44(5): 664-674.
[11]	Yingfeng Cai,Ziheng Lu,Yicheng Li,Long Chen,Hai Wang. Tightly Coupled SLAM System Based on Multi-Sensor Fusion [J]. Automotive Engineering, 2022, 44(3): 350-361.
[12]	Jingwei Zhang,Tiejun Liu,Rengang Li,Dan Liu,Jinglin Zhan,Hongwei Kan. A Temporal Calibration Method for Multi-Sensor Fusion of Autonomous Vehicles [J]. Automotive Engineering, 2022, 44(2): 215-224.
[13]	Chaoyang Jiang,Tianran Lan,Xiaoni Zheng,Jiulong Gao,Xuetong Ye. Distributed Multi-vehicle Collaborative Visual SLAM System [J]. Automotive Engineering, 2022, 44(12): 1809-1817.
[14]	Yuande Jiang,Bing Zhu,Xiangmo Zhao,Jian Zhao,Bingbing Zheng. Modeling of Traffic Vehicle Interaction for Autonomous Vehicle Testing [J]. Automotive Engineering, 2022, 44(12): 1825-1833.
[15]	Bing Zhu,Yuhang Sun,Jian Zhao,Peixing Zhang,Tianxin Fan,Dongjian Song. Automatic Extraction Method for Autonomous Vehicle Test Scene Primitives [J]. Automotive Engineering, 2022, 44(11): 1647-1655.