仿驾驶员DDPG汽车纵向自动驾驶决策方法

doi:10.19562/j.chinasae.qcgc.2021.12.001

Abstract

Abstract:

The decision-making layer of vehicle longitudinal autonomous driving decides the ideal action instruction according to the current motion state of the vehicle and environmental information. At present， how to consider the behavior of human drivers in autonomous driving decision-making strategies has become a hotspot. In longitudinal autonomous driving decision-making strategies， traditional rule-based decision-making strategies are difficult to be applied to complex scenarios. Current decision-making methods use reinforcement learning and deep reinforcement learning to construct reward functions designed with safety， comfort， and economy formulas. The obtained decision-making strategy still has a big gap compared with that of the human driver. To solve the above problems， this paper uses driver data to design a reward function by BP neural network， and uses DDPG algorithm to establish a driver-like longitudinal autonomous driving decision-making method. Finally， the effectiveness of the method and the consistency with the driver's behavior are verified by simulation tests.

Key words: autonomous driving, decision-making algorithm, deep reinforcement learning, deep deterministic policy gradient（DDPG）

Zhenhai Gao,Xiangtong Yan,Fei Gao,Tianjun Sun. A Driver-like Decision-making Method for Longitudinal Autonomous Driving Based on DDPG[J].Automotive Engineering, 2021, 43(12): 1737-1744.

Figures/Tables 16

参数	数值	参数	数值
折扣因子 $γ$	0.99	更新系数 $τ$	0.001
Actor网络学习率	0.000 1	Critic网络学习率	0.001
Mini-batch大小	64	最大训练回合数	1 000

References 21

1	朱冰，蒋渊德，赵健，等.基于深度强化学习的车辆跟驰控制［J］.中国公路学报，2019，32（6）：53-60.
	ZHU B， JIANG Y D， ZHAO J， et al. A car-following control algorithm based on deep reinforcement learning［J］. China Journal of Highway and Transport， 2019， 32（6）： 53-60.
2	罗禹贡，陈涛，周磊，等.奔腾智能混合动力电动轿车自适应巡航控制系统［J］. 机械工程学报，2010，46（6）：2-7.
	LUO Y G， CHEN T， ZHOU L， et al. Adaptive cruise control system of besturn intelligent hybrid electric vehicle［J］. Journal of Mechanical Engineering，2010， 46（6）：2-7.
3	郝克宇. 电动汽车自适应巡航控制系统研究［D］.长春：长春工业大学，2018.
	HAO K Y.On the adaptive cruise controlsystem for electric vehicle［D］.Changchun：Changchun University of Technology，2018.
4	张进国. 汽车主动巡航智能化控制与仿真［D］.锦州：辽宁工业大学，2018.
	ZHANG J G.Intelligent control and simulation of automobile active cruise system［D］.Jinzhou： Liaoning University of Technology， 2018.
5	高振海，孙天骏，何磊.汽车纵向自动驾驶的因果推理型决策［J］.吉林大学学报（工学版），2019，49（5）：1392-1404.
	GAO Z H， SUN T J， HE L.Causal reasoning decision⁃making for vehicle longitudinalautonomous driving［J］. Journal of Jilin University（Engineering and Technology Edition）， 2019， 49（5）： 1392-1404.
6	SUN M， ZHAO W Q， SONG G H， et al. DDPG-based decision-making strategy of adaptive cruising for heavy vehicles considering stability［J］.IEEE Access，2020，8： 59225-59246.
7	CHARLES D， BRAHIM C.Cooperative adaptive cruise control： a reinforcement learning approach［J］.IEEE Transactions on Intelligent Transportation Systems， 2011，12（4）： 1248-1260.
8	CHAE H，KANG C M， KIM B D，et al.Autonomous braking systemvia deep reinforcement learning［C］. 2017 IEEE 20th International Conference on Intelligent Transportation Systems （ITSC）. IEEE， 2017： 1-6.
9	YUAN H， NI J， HU J.A centralised training algorithm with D3QN for scalable regular unmanned ground vehicle formation maintenance［J］.IET Intelligent Transport Systems，2021，15（4）： 562-572.
10	GUO Q， OHAY A， LIU Z J， et al.Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles alongsignalized corridors［J］.Transportation Research Part C： Emerging Technologies，2021，124： 102980.
11	LIN Y， JOHN M， NASSER L A.Comparison of deep reinforcement learning and model predictive control for adaptive cruise control［J］.IEEE Transactions on Intelligent Vehicles，2020，6（2）： 221-231.
12	YANG J R， LIU X L， LIU S D， et al.Longitudinal tracking control of vehicle platooning using DDPG-based PID［C］.2020 4th CAA International Conference on Vehicular Control and Intelligence （CVCI），2020.
13	YE Y J， ZHANG X H， SUN J.Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment［J］.Transportation Research Part C： Emerging Techndogies，2019，107： 155-170.
14	ZHU M X， WANG Y H， PU Z Y.Safe， efficient， and comfortablevelocity control based on reinforcement learning for autonomous driving［J］.Transportation Research Part C： Emerging Technologies，2020，117：102662.
15	郭景华，李文昌，罗禹贡，等. 基于深度强化学习的驾驶员跟车模型研究［J］.汽车工程，2021，43（4）： 571-579.
	GUO J H， LI W C， LUO Y G， et al.Driver car⁃following model based on deep reinforcement learning［J］. Automotive Engineering， 2021， 43（4）： 571-579.
16	ZHU M X， WANG X S， WANG Y H， et al.Human-like autonomous car-following model with deep reinforcement learning［J］.Transportation Research Part C： Emerging Techndogies，2018，97： 348-368.
17	SUTTON R S， BARTO A G. Reinforcement learning： an introduction［J］.IEEE Transactions on Neural Networks，1998，9（5）： 1054.
18	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［J］. arXiv preprint arXiv：， 2015.
19	SILVER D， LEVER G， HEESS N， et al. Deterministic policy gradientalgorithms［C］.ICML. 2014.
20	VAN H， GUEZ A， SILVER D. Deep reinforcement learning with double Q-learning［J］. arXiv preprint arXiv：， 2015.
21	MNIH V， KAVUKCUOGLU K， SILVER D， et al. Human-level control through deep reinforcement learning［J］. Nature， 2015， 518（7540）： 529.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	ZONG Yi-Qi, GU Zheng-Qi, LUO Ze-Min, JIANG Cai-Mao, ZHANG Qi-Dong, YANG Zhen-Dong. [J]. , 2016, 38(12): 1427 -1433 .
[3]	ZHANG Ying-Chao, ZHAN Da-Peng, ZHAO Jing, ZHANG Zhe, LI Jie. [J]. , 2016, 38(12): 1434 -1439 .
[4]	TANG You-Ming, YAN Ling-Bo, LUO Qian, CAO Li-Bo. [J]. , 2016, 38(12): 1452 -1458 .
[5]	ZHANG Lei, LU Jian-Wei, JIANG Jun-Zhao, YAN Pei-Lei, LI Lei. [J]. , 2016, 38(12): 1477 -1482 .
[6]	ZHAO Liang, ZHANG Qiang, GUO Kong-Hui. [J]. , 2017, 39(1): 86 -91 .
[7]	SHEN Deng-Feng, WANG Chen, YU Hai-Sheng, ZHANG Tong, YI Xian-Ke. [J]. , 2017, 39(1): 15 -22 .
[8]	ZOU Tie-Fang, HU Lin, LI Ping-Fan, YI Liang. [J]. , 2017, 39(1): 41 -46 .
[9]	ZHOU Kai, ZANG Jing-Lun. [J]. , 2017, 39(2): 127 -132 .
[10]	ZUO Wen-Jie, CHEN Ji-Shun, LI Yi-Wen, LIU Nian. [J]. , 2017, 39(2): 145 -149 .

状态集			动作集
相对距离	相对速度	本车速度	纵向加速度
d_r	v_r	v_x	a_x
m	m·s^-1	m·s^-1	m·s^-2

[1]	Xiaocong Zhao,Shiyu Fang,Zirui Li,Jian Sun. Extraction and Application of Key Utility Term for Social Driving Interaction [J]. Automotive Engineering, 2024, 46(2): 230-240.
[2]	Yanli Ma, Qin Qin, Fangqi Dong, Yining Lou. Takeover Risk Assessment Model Based on Risk Field Theory Under Different Cognitive Secondary Tasks [J]. Automotive Engineering, 2024, 46(1): 9-17.
[3]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[4]	Ming Wang,Xiaolin Tang,Kai Yang,Guofa Li,Xiaosong Hu. A Motion Planning Method for Autonomous Vehicles Considering Prediction Risk [J]. Automotive Engineering, 2023, 45(8): 1362-1372.
[5]	Dongyu Zhao, Shuen Zhao. Autonomous Driving 3D Object Detection Based on Cascade YOLOv7 [J]. Automotive Engineering, 2023, 45(7): 1112-1122.
[6]	Jiahao Zhao,Zhiquan Qi,Zhifeng Qi,Hao Wang,Lei He. Calculation of Heading Angle of Parallel Large Vehicle Based on Tire Feature Points [J]. Automotive Engineering, 2023, 45(6): 1031-1039.
[7]	Lisheng Jin,Guangde Han,Xianyi Xie,Baicang Guo,Guofeng Liu,Wentao Zhu. Review of Autonomous Driving Decision-Making Research Based on Reinforcement Learning [J]. Automotive Engineering, 2023, 45(4): 527-540.
[8]	Qingyang Huang,Xiaoping Jin,Yikang Zhang. Analysis of Drivers' Driving Posture Change Rule Under the Condition of Automatic Driving Level Improvement [J]. Automotive Engineering, 2023, 45(3): 382-392.
[9]	Zhengfa Liu,Ya Wu,Peigen Liu,Rongqi Gu,Guang Chen. Cross-Domain Object Detection for Intelligent Driving Based on Joint Distribution Matching of Features and Labels [J]. Automotive Engineering, 2023, 45(11): 2082-2091.
[10]	Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios [J]. Automotive Engineering, 2023, 45(10): 1791-1802.
[11]	Pulei Xu,Yingfeng Cai,Yubo Lian,Xiaoqiang Sun,Hai Wang,Long Chen,Yilin Zhong. AFS/DYC Coordinated Control of Intelligent Vehicles Based on Improved Hierarchical Extensibility Theory [J]. Automotive Engineering, 2023, 45(1): 20-31.
[12]	Runhui Huang,Likun Hu,Mingfang Su,Daye Xu,Aoran Chen. Semantic Segmentation Method of LiDAR Point Cloud Based on 3D Conical Grid [J]. Automotive Engineering, 2022, 44(8): 1173-1182.
[13]	Zhenhai Gao,Xiangtong Yan,Fei Gao. A Decision-making Method for Longitudinal Autonomous Driving Based on Inverse Reinforcement Learning [J]. Automotive Engineering, 2022, 44(7): 969-975.
[14]	Dafang Wang,Jingdong Du,Jiang Cao,Mei Zhang,Gang Zhao. Research on Style Transfer Network for Autonomous Driving Data Generation [J]. Automotive Engineering, 2022, 44(5): 684-690.
[15]	Jie Hu,Minchao Zhang,Wencai Xu,Ruinan Chen,Xinkai Zhong,Linglei Zhu. Parallel Parking Trajectory Planning for Autonomous Vehicles [J]. Automotive Engineering, 2022, 44(3): 330-339.

A Driver-like Decision-making Method for Longitudinal Autonomous Driving Based on DDPG

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 21

Related Articles 15

Metrics

Comments

Recommended 10