基于混合专家模型的智能网联汽车换道决策方法

doi:10.19562/j.chinasae.qcgc.2024.ep.001

Abstract

Abstract:

The problem of lane-changing decision-making on highways，characterized by complex scenarios，strong uncertainty，and high real-time requirements，is a research hotspot and challenge in the field of autonomous driving both domestically and internationally. Deep Reinforcement Learning （DRL） exhibits excellent real-time decision-making capabilities and adaptability to complex scenarios. However，under the constraints of limited training samples and cost，its learning effectiveness remains limited，making it difficult to ensure optimal driving efficiency and complete driving safety. In this paper， a DRL-Mixture of Expert （DRL-MOE） lane-changing decision-making method based on the improved DRL model is proposed. Firstly，the upper-level classifier dynamically determines the activation status of the lower-level DRL expert or heuristic expert based on the input state features. Then， to enhance the learning effectiveness of the DRL expert，the method utilizes Behavior Cloning （BC） for initializing the neural network parameters to make improvements on the traditional Deep Deterministic Policy Gradient （DDPG） algorithm. Finally， the Intelligent Driver Model （IDM） and the strategy of Minimizing Overall Braking Induced by Lane changes （MOBIL） are designed as heuristic experts to ensure driving safety. The simulation results show that compared to non-mixed expert DRL methods，the proposed DRL-MOE model improves driving efficiency by 15.04%，ensuring zero collisions and zero departures，demonstrating higher robustness and superior performance.

Key words: autonomous driving, high speed lane change decision-making, deep reinforcement learning, mixture of expert

Fuxing Yao,Chao Sun,Yungang Lan,Bing Lu,Bo Wang,Haiyang Yu. A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model[J].Automotive Engineering, 2024, 46(5): 882-892.

Figures/Tables 12

参数	数值
加速度参数 $δ$	4
期望时间间隔 $T$ /s	1.5
最大加速度 $a m a x$ /（m·s^-2）	6
舒适减速度 $b$ /（m·s^-2）	5
安全减速度限制 $b s a f e$ /（m·s^-2）	2
最小相对距离 $d 0$ /m	10
礼让因子 $p$	0.001
换道决策阈值 $a t h$ /（m·s^-2）	0.2

奖励系数	数值	奖励系数	数值
$k 1$	2	$k 3.3$	-1
$k 2.1$	4	$k 4.1$	-100
$k 2.2$	-5	$k 4.2$	-100
$k 2.3$	-2	$k 5.1$	-100
$k 3.1$	5	$k 5.2$	-100
$k 3.2$	1

超参数名称	数值
学习率 $α$	$3 e - 4$
折扣因子 $γ$	0.99
缓冲池大小	$1 e 6$
批次大小	128

References 25

1	LI L，OTA K，DONG M. Humanlike driving： empirical decision-making system for autonomous vehicles［J］. IEEE Transactions on Vehicular Technology，2018，67（8）：6814-6823.
2	IOANNOU P， CHIEN C. A rule-based approach to lane change maneuvers on highways［J］. IEEE Transactions on Vehicular Technology，1993，42（4）：657-672.
3	CAO Y，YU X， SHEN W. A fuzzy logic-based approach for autonomous vehicle lane change decision-making［J］. IEEE Transactions on Intelligent Transportation Systems，2003，4（2）：89-102.
4	KIRAN B R， SOBH I， TALPAERT V，et al. Deep reinforcement learning for autonomous driving： a survey［J］. IEEE Transactions on Intelligent Transportation Systems，2021.
5	LIU X，LI H，WANG J，et al. Personalized automatic driving system based on reinforcement learning technology［C］. 2019 4th International Conference on Mechanical，Control and Computer Engineering （ICMCCE）. IEEE，2019：373-3733.
6	ALIZADEH A， MOGHADAM M， BICER Y，et al. Automated lane change decision making using deep reinforcement learning in dynamic and uncertain highway environment［C］.2019 IEEE Intelligent Transportation Systems Conference （ITSC） . IEEE，2019：1399-1404.
7	ZHANG S，PENG H，NAGESHRAO S，et al. Discretionary lane change decision making using reinforcement learning with model-based exploration［C］.2019 18th IEEE International Conference on Machine Learning And Applications （ICMLA），IEEE，2019： 844-850.
8	FU Y，LI C，YU F，et al. A decision-making strategy for vehicle autonomous braking in emergency via deep reinforcement learning［J］. IEEE Transactions on Vehicular Technology，2020，69（6）：5876-5888.
9	HUANG Z，WU J， LV C. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning［J］. IEEE Transactions on Intelligent Transportation Systems，2021.
10	WU Z，SUN L，ZHAN W，et al. Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving［J］. IEEE Robotics and Automation Letters，2020，5（4）：5355-5362.
11	金立生，韩广德，谢宪毅，等. 基于强化学习的自动驾驶决策研究综述［J］. 汽车工程，2023，45（4）： 527-540.
	JIN L，HAN G，XIE X，et al. Review of autonomous driving decision-making research based on reinforcement learning［J］. Automotive Engineering，2023，45（4）： 527-540.
12	SHI T，WANG P，CHENG X，et al. Driving decision and control for automated lane change behavior based on deep reinforcement learning［C］. 2019 IEEE Intelligent Transportation Systems Conference （ITSC） . IEEE，2019： 2895-2900.
13	YE F，CHENG X，WANG P，et al. Automated lane change strategy using proximal policy optimization-based deep reinforcement learning［C］. 2020 IEEE Intelligent Vehicles Symposium （IV） . IEEE，2020：1746-1752.
14	YUAN K，HUANG Y，YANG S，et al.Evolutionary decision-making and planning for autonomous driving based on safe and rational exploration and exploitation［J］.Engineering，2023，VISSN 2095-8099.
15	LI Z，XIONG L，LENG B，et al.Safe reinforcement learning of lane change decision making with risk-fused constraint［C］.2023 IEEE 26th International Conference on Intelligent Transportation Systems （ITSC），Bilbao，Spain，2023：1313-131.
16	高振海，闫相同，高菲，等. 仿驾驶员DDPG汽车纵向自动驾驶决策方法［J］. 汽车工程，2021，43（12）： 1737-1744.
	GAO Z，YAN X，GAO F，et al. A driver-like decision-making method for longitudinal autonomous driving based on DDPG［J］. Automotive Engineering，2021，43（12）： 1737-1744.
17	WANG P，LI H， CHAN C Y. Quadratic Q-network for learning continuous control for autonomous vehicles［J］. arXiv preprint arXiv：，2019.
18	LI X，QIU X，WANG J，et al. A deep reinforcement learning based approach for autonomous overtaking［C］. 2020 IEEE International Conference on Communications Workshops （ICC Workshops） . IEEE，2020：1-5.
19	TANG X，HUANG B，LIU T，et al. Highway decision-making and motion planning for autonomous driving via soft actor-critic［J］. IEEE Transactions on Vehicular Technology，2022， 71（5）：4706-4717.
20	TREIBER M，HENNECKE A， HELBING D. Congested traffic states in empirical observations and microscopic simulations［J］. Physical Review E，2000，62（2）：1805.
21	KESTING A，TREIBER M， HELBING D. General lane-changing model mobil for car-following models［J］. Transportation Research Record，2007，1999（1）：86-94.
22	KONG J，PFEIFFER M，SCHILDBACH G，et al. Kinematic and dynamic vehicle models for autonomous driving control design［C］. 2015 IEEE Intelligent Vehicles Symposium （IV） . IEEE，2015：1094-1099.
23	ATAEI M，KHAJEPOUR A，JEON S. Reconfigurable integrated stability control for four-and-three wheeled urban vehicles with flexible combinations of actuation systems［J］. IEEE/ASME Transactions on Mechatronics，2018，23（5）：2031-2041.
24	ZHOU M，QU X，JIN S. On the impact of cooperative autonomous vehicles in improving freeway merging： a modified intelligent driver model-based approach［J］. IEEE Transactions on Intelligent Transportation Systems，2016，18（6）：1422-1428.
25	LEURENT E. An environment for autonomous driving decision-making （version 1.4）［CP］. https：//github.com/eleurent/highway-env，2018.

算法名称	平均速度/（m·s^-1）	碰撞率/%	出界率/%
IDM+MOBIL	20.15	0	0
DQN	20.78	17	2
PPO	23.98	15	8
SAC	24.52	18	10
DDPG w/o BC	24.14	22	6
DDPG w/ BC	25.85	8	4
DRL-MOE	29.74	0	0

测试条件名称	不同参数设置	DRL-MOE 平均速度/（m·s^-1）	DRL-MOE 碰撞率/%	DRL-MOE 出界率%	DDPG w/ BC平均速度/（m·s^-1）	DDPG w/ BC碰撞率/%	DDPG w/ BC出界率/%
默认条件		29.74	0	0	25.85	8	4
行驶时间变化，其他环境条件不变	300时间步	28.71	2	1	25.22	13	6
	400时间步	28.54	3	2	24.88	15	7
	500时间步	28.51	5	1	24.17	19	10
车流密度变化，其他环境条件不变	1.1倍	27.76	2	1	24.11	12	8
	1.2倍	25.98	2	2	23.09	19	13
	1.3倍	25.46	3	2	21.97	31	15
车道数变化，其他环境条件不变	5车道	29.45	0	1	25.33	11	4
加速度范围变化，其他环境条件不变	［-3.5， 3.5］ m/s²	28.01	0	0	23.96	7	6
	［-3， 3］ m/s²	26.12	1	0	23.36	9	6
	［-2.5， 2.5］ m/s²	25.62	1	1	23.18	8	11

测试条件名称	不同参数设置	DRL-MOE 平均速度/（m·s^-1）	DRL-MOE碰撞率/%	DRL-MOE出界率/%	DDPG w/ BC平均速度/（m·s^-1）	DDPG w/ BC 碰撞率/%	DDPG w/ BC 出界率/%
默认条件（与训练集相同）（φ = 0.7， α = 0）		26.74	4	0	22.06	25	14
道路附着系数φ变化，其他环境条件不变	0.65	24.55	8	1	20.11	33	11
	0.60	23.89	7	3	19.56	29	17
	0.55	22.68	8	5	18.95	36	15
坡度α变化，其他环境条件不变	0.5%	26.23	4	0	21.76	26	15
	1.0%	26.28	4	0	21.92	24	14
	1.5%	25.54	4	0	21.74	26	14

[1]	Mengfan Li,Zhongxiang Feng,Weihua Zhang,Jingyu Li. Study on Driver's Visual Transfer Characteristics During the Takeover Process of Human-Computer Co-driving Mode [J]. Automotive Engineering, 2024, 46(5): 795-804.
[2]	Ting Chikit,Yafei Wang,Yichen Zhang,Mingyu Wu,Yile Wang. Energy-Saving Planning Method for Autonomous Driving Mining Trucks Based on Composite Dynamic Sampling [J]. Automotive Engineering, 2024, 46(4): 588-595.
[3]	Yiwei Zhou,Mo Xia,Bing Zhu. Multimodal Vehicle Trajectory Prediction Methods Considering Multiple Traffic Participants in Urban Road Scenarios [J]. Automotive Engineering, 2024, 46(3): 396-406.
[4]	Xiaocong Zhao,Shiyu Fang,Zirui Li,Jian Sun. Extraction and Application of Key Utility Term for Social Driving Interaction [J]. Automotive Engineering, 2024, 46(2): 230-240.
[5]	Yanli Ma, Qin Qin, Fangqi Dong, Yining Lou. Takeover Risk Assessment Model Based on Risk Field Theory Under Different Cognitive Secondary Tasks [J]. Automotive Engineering, 2024, 46(1): 9-17.
[6]	Weiguo Liu,Zhiyu Xiang,Weiping Liu,Daoxin Qi,Zixu Wang. Research on Vehicle Control Algorithm Based on Distributed Reinforcement Learning [J]. Automotive Engineering, 2023, 45(9): 1637-1645.
[7]	Ming Wang,Xiaolin Tang,Kai Yang,Guofa Li,Xiaosong Hu. A Motion Planning Method for Autonomous Vehicles Considering Prediction Risk [J]. Automotive Engineering, 2023, 45(8): 1362-1372.
[8]	Dongyu Zhao, Shuen Zhao. Autonomous Driving 3D Object Detection Based on Cascade YOLOv7 [J]. Automotive Engineering, 2023, 45(7): 1112-1122.
[9]	Jiahao Zhao,Zhiquan Qi,Zhifeng Qi,Hao Wang,Lei He. Calculation of Heading Angle of Parallel Large Vehicle Based on Tire Feature Points [J]. Automotive Engineering, 2023, 45(6): 1031-1039.
[10]	Lisheng Jin,Guangde Han,Xianyi Xie,Baicang Guo,Guofeng Liu,Wentao Zhu. Review of Autonomous Driving Decision-Making Research Based on Reinforcement Learning [J]. Automotive Engineering, 2023, 45(4): 527-540.
[11]	Qingyang Huang,Xiaoping Jin,Yikang Zhang. Analysis of Drivers' Driving Posture Change Rule Under the Condition of Automatic Driving Level Improvement [J]. Automotive Engineering, 2023, 45(3): 382-392.
[12]	Zhengfa Liu,Ya Wu,Peigen Liu,Rongqi Gu,Guang Chen. Cross-Domain Object Detection for Intelligent Driving Based on Joint Distribution Matching of Features and Labels [J]. Automotive Engineering, 2023, 45(11): 2082-2091.
[13]	Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios [J]. Automotive Engineering, 2023, 45(10): 1791-1802.
[14]	Pulei Xu,Yingfeng Cai,Yubo Lian,Xiaoqiang Sun,Hai Wang,Long Chen,Yilin Zhong. AFS/DYC Coordinated Control of Intelligent Vehicles Based on Improved Hierarchical Extensibility Theory [J]. Automotive Engineering, 2023, 45(1): 20-31.
[15]	Runhui Huang,Likun Hu,Mingfang Su,Daye Xu,Aoran Chen. Semantic Segmentation Method of LiDAR Point Cloud Based on 3D Conical Grid [J]. Automotive Engineering, 2022, 44(8): 1173-1182.

A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 25

Related Articles 15

Metrics

Comments

Recommended 10