基于模仿学习和强化学习的智能车辆换道行为决策

doi:10.19562/j.chinasae.qcgc.2021.01.008

Abstract

Abstract:

A lane?change behavior decision?making method of the intelligent vehicle is proposed based on imitation learning and reinforcement learning， in which the macro decision?making module constructs the extreme gradient boosting model through imitation learning， and selects the macro instructions （lane?keeping， left lane?change and right lane?change） according to the input information， so as to determine the sub?problem of lane?change behavior decision that need to be solved. Each detailed decision?making sub?module acquires its optimized strategy through the reinforcement learning of deep deterministic strategy gradient to solve the corresponding sub?problem for determining the movement target position of ego?vehicle and sending to lower?level modules for execution. Simulation results show that the strategy learning speed of the proposed method is faster than that of pure reinforcement learning， and its comprehensive performance is better than that of finite state machine， behavior clone imitation learning and pure reinforcement learning.

Key words: intelligent vehicle, behavior decision?making, reinforcement learning, imitation learning

Xiaolin Song,Xin Sheng,Haotian Cao,Mingjun Li,Binlin Huang Zhi Yi. Lane‑change Behavior Decision‑making of Intelligent Vehicle Based on Imitation Learning and Reinforcement Learning[J].Automotive Engineering, 2021, 43(1): 59-67.

Figures/Tables 14

名称	描述	取值
training episodes	训练轮次数	5 000
γ	折扣系数	0.99
$α Q$	评论家网络学习率	10^-4
$α μ$	演员网络学习率	10^-5
batch size	采样经历样本数	64
replay memory size	经验回放库容量	8 000

References 21

1	PADEN B， ČÁP M， YONG S Z， et al. A survey of motion planning and control techniques for self⁃driving urban vehicles［J］. IEEE Transactions on Intelligent Vehicles， 2016， 1（1）： 33-55.
2	冀杰，黄岩军，李云伍，等.基于有限状态机的车辆自动驾驶行为决策分析［J］.汽车技术，2018（12）：1-7.
	JI J， HUANG Y J， LI Y W，et al.Decision making analysis of autonomous driving behaviors for intelligent vehicles based on finite state machine［J］. Automobile Technology，2018（12）：1-7.
3	KURT A， ÜÖZGÜNER. Hierarchical finite state machines for autonomous mobile systems［J］. Control Engineering Practice， 2013， 21（2）： 184-194.
4	WANG M， HOOGENDOORN S P， DAAMEN W， et al. Game theoretic approach for predictive lane⁃changing and car⁃following control［J］. Transportation Research Part C： Emerging Technologies， 2015， 58： 73-92.
5	YU H， TSENG H E， LANGARI R. A human⁃like game theory-based controller for automatic lane changing［J］. Transportation Research Part C： Emerging Technologies， 2018， 88： 140-158.
6	BOJARSKI M， DDEL TESTA， DWORAKOWSKI D， et al. End to end learning for self⁃driving cars［J］. arXiv Preprint arXiv： 1604.07316， 2016.
7	CODEVILLA F， MIILLER M， LÓPEZ A， et al. End⁃to⁃end driving via conditional imitation learning［C］. 2018 IEEE International Conference on Robotics and Automation （ICRA）. IEEE， 2018： 1-9.
8	KUEFLER A， MORTON J， WHEELER T， et al. Imitating driver behavior with generative adversarial networks［C］. 2017 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2017： 204-211.
9	HESSEL M， MODAYIL J， HVAN HASSELT， et al. Rainbow： combining improvements in deep reinforcement learning［C］. Thirty⁃Second AAAI Conference on Artificial Intelligence， 2018.
10	HAARNOJA T， ZHOU A， ABBEEL P， et al. Soft actor⁃critic： offp⁃olicy maximum entropy deep reinforcement learning with a stochastic actor［C］. International Conference on Machine Learning. 2018： 1861-1870.
11	朱冰，蒋渊德，赵健，等.基于深度强化学习的车辆跟驰控制［J］.中国公路学报，2019，32（6）：53-60.
	ZHU B， JIANG Y D， ZHAO J， et al.A car⁃following control algorithm based on deep reinforcement learning［J］. China Journal of Highway and Transport， 2019，32（6）：53-60.
12	MIRCHEVSKA B， PEK C， WERLING M， et al. High⁃level decision making for safe and reasonable autonomous lane changing using reinforcement learning［C］. 2018 21st International Conference on Intelligent Transportation Systems （ITSC）. IEEE， 2018： 2156-2162.
13	WANG P， CHAN C Y， DE LA FORTELLE A. A reinforcement learning based approach for automated lane change maneuvers［C］. 2018 IEEE Intelligent Vehicles Symposium （IV）. IEEE， 2018： 1379-1384.
14	CHEN T， GUESTRIN C. Xgboost： a scalable tree boosting system［C］. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM， 2016： 785-794.
15	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［J］. arXiv Preprint arXiv： 1509.02971， 2015.
16	LIN L J. Reinforcement learning for robots using neural networks［D］. Pittsburgh： Carnegie Mellon University， 1993.
17	MNIH V， KAVUKCUOGLU K， SILVER D， et al. Playing atari with deep reinforcement learning［J］. arXiv Preprint arXiv： 1312.5602， 2013.
18	VOGEL K. A comparison of headway and time to collision as safety indicators［J］. Accident Analysis & Prevention， 2003， 35（3）： 427-433.
19	ZHANG M， LI N， GIRARD A， et al. A finite state machine based automated driving controller and its stochastic optimization［C］. ASME 2017 Dynamic Systems and Control Conference. American Society of Mechanical Engineers Digital Collection， 2017.
20	NAGESHRAO S， TSENG H E， FILEV D. Autonomous highway driving using deep reinforcement learning［C］.2019 IEEE International Conference on Systems， Man and Cybernetics （SMC）. IEEE， 2019： 2326-2331.
21	姜岩，龚建伟，熊光明，等.基于运动微分约束的无人车辆纵横向协同规划算法的研究［J］.自动化学报，2013，39（12）：2012-2020.
	JIANG Y， GONG J W， XIONG G M， et al. Research on differential Constraints⁃based planning algorithm for autonomous⁃driving vehicles［J］. Acta Automatica Sinica，2013，39（12）：2012-2020.

[1]	WANG Dan, LIU Zhong-Chang, TIAN Jing, HAN Yong-Qiang, TAN Man-Zhi. [J]. , 2016, 38(12): 1415 -1419 .
[2]	KONG Xiang-Dong, ZHANG Zhen-Dong, XIE Nai-Liu, CHENG Qiang, YIN Cong-Bo. [J]. , 2016, 38(12): 1471 -1476 .
[3]	LI Ling, MA Li, MOU Yu, XU Chao, LI Wen-Ru, SHI Shu-Ming. [J]. , 2016, 38(12): 1508 -1514 .
[4]	GAO Ji-Dong, GAO Bo-Lin, XIE Shu-Gang. [J]. , 2016, 38(12): 1515 -1520 .
[5]	QIN Xiao-Hui, WANG Jian-Qiang, XIE Bo-Yuan, HU Man-Jiang, LI Ke-Qiang. [J]. , 2017, 39(1): 73 -78 .
[6]	SU Chang, FU Li-Ming, WEI Jun, CAO Yue, HUANG Lei, LI Shuo. [J]. , 2017, 39(1): 35 -40 .
[7]	ZHANG Jian, ZHOU Dao-Kui, TONG Rui, QI Hao-Chen. [J]. , 2017, 39(1): 47 -51 .
[8]	ZHENG Wei. [J]. , 2017, 39(1): 107 -112 .
[9]	SHAO Shi-Ping, HUANG Ju-Hua, CAO Ming. [J]. , 2017, 39(2): 200 -205 .
[10]	Chen-Xin, FENG Xiao, SHEN Chuan-Liang, WANG Shuo, HU Cui-Song, LI Yan-Yang, YANG Chang-Hai-　. [J]. , 2017, 39(2): 206 -213 .

参数	描述	取值
K	基学习器数量	120
ε	学习率	0.1
max_depth	最大树深度	6
min_child_weight	最小叶子样本权重和	5
gamma	节点分裂增益阈值	0
subsample	采样比例	0.8
colsample	列采样比例	0.8

网络	层	维度	激活函数
	全连接层1	(22,400)	ReLU
评论家网络	全连接层2	(400,200)	ReLU
	全连接层3	(200,1)	ReLU
	全连接层1	(23,200)	ReLU
演员网络	全连接层2	(200,100)	ReLU
	全连接层3	（100，1）	ReLU

方法	D_FSM	D_IL	D_RL	D_IRL
碰撞次数	0	3	0	0
平均车速/（km·h^-1）	76.8	81.3	80.7	81.2
车速标准差/（km·h^-1）	10.3	5.8	6.5	5.6
平均单轮换道数	6	9	12	8

[1]	Jianping Hao,Yanzhao Su,Zhihua Zhong,Jin Huang. Service-Oriented Architecture and Service Scheduling Mechanism for Intelligent Vehicles [J]. Automotive Engineering, 2023, 45(9): 1563-1572.
[2]	Gaoshi Zhao,Long Chen,Yingfeng Cai,Yubo Lian,Hai Wang,Qingchao Liu,Chenglong Teng. Trajectory Prediction Technology Integrating Complex Network and Memory-Augmented Network [J]. Automotive Engineering, 2023, 45(9): 1608-1616.
[3]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[4]	Lin Hu,Gen Li,Fang Wang,Miao Lin,Ning Wu. Research on Test Scenarios of Passenger Cars and Two-Wheelers at Intersections Based on CIDAS Accident Data [J]. Automotive Engineering, 2023, 45(8): 1417-1427.
[5]	Qihui Hu,Yingfeng Cai,Hai Wang,Long Chen,Zhaozhi Dong,Qingchao Liu. Heterogeneous Multi-object Trajectory Prediction Method Based on Hierarchical Graph Attention [J]. Automotive Engineering, 2023, 45(8): 1448-1456.
[6]	Shiju Pan, Jianshi Li, Hua Li, Jingtao Lou, Youchun Xu. Path Following Method of Intelligent Vehicles Based on Feedback Pure Tracking Method [J]. Automotive Engineering, 2023, 45(7): 1134-1144.
[7]	Jun Li, Wei Zhou, Shuang Tang. Lane Change and Obstacle Avoidance Trajectory Planning of Intelligent Vehicle Based on Adaptive Fitting [J]. Automotive Engineering, 2023, 45(7): 1174-1183.
[8]	Zixian Li,Shiju Pan,Yuan Zhu,Binbing He,Youchun Xu. Semi-active Suspension Control for Intelligent Vehicles Based on State Feedback and Preview Feedforward [J]. Automotive Engineering, 2023, 45(5): 735-745.
[9]	Lisheng Jin,Guangde Han,Xianyi Xie,Baicang Guo,Guofeng Liu,Wentao Zhu. Review of Autonomous Driving Decision-Making Research Based on Reinforcement Learning [J]. Automotive Engineering, 2023, 45(4): 527-540.
[10]	Xiang Gao,Long Chen,Xinye Wang,Xiaoxia Xiong,Yicheng Li,Yuexia Chen. Intelligent Vehicle Driving Risk Assessment Method Based on Trajectory Prediction [J]. Automotive Engineering, 2023, 45(4): 588-597.
[11]	Jie Hu,Qi Zhu,Ruipeng Chen,Minchao Zhang,Zhihao Zhang,Haoyan Liu. Global Path Planning of Intelligent Vehicle with Must-Pass Nodes [J]. Automotive Engineering, 2023, 45(3): 350-360.
[12]	Lü Yanzhi,Chao Wei,Yuanhao He. An End-to-End Lane Change Method for Autonomous Driving Based on GCN and CIL [J]. Automotive Engineering, 2023, 45(12): 2310-2317.
[13]	Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios [J]. Automotive Engineering, 2023, 45(10): 1791-1802.
[14]	Chunyang Qi,Chuanxue Song,Shixin Song,Liqiang Jin,Da Wang,Feng Xiao. Research on Energy Management Strategy for Hybrid Electric Vehicles Based on Inverse Reinforcement Learning [J]. Automotive Engineering, 2023, 45(10): 1954-1964.
[15]	Shiju Pan,Yongle Li,Zixian Li,Binbing He,Yuan Zhu,Youchun Xu. Path Following Method of Intelligent Vehicles Based on Improved Pure Tracking [J]. Automotive Engineering, 2023, 45(1): 1-8.

Lane‑change Behavior Decision‑making of Intelligent Vehicle Based on Imitation Learning and Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 21

Related Articles 15

Metrics

Comments

Recommended 10