基于强化学习的城市场景多目标生态驾驶策略

doi:10.19562/j.chinasae.qcgc.2023.10.002

Abstract

Abstract:

To improve the ride experience of connected and automated vehicle in complex urban traffic scenarios， this paper proposes a deep reinforcement learning based multi-objective eco-driving strategy that considers driving safety， energy economy， ride comfort， and travel efficiency. Firstly， the state space， action space， and multi-objective reward function of the eco-driving strategy are constructed based on the Markov decision process. Secondly， the car-following safety model and traffic light safety model are designed to provide safety speed suggestion for the eco-driving strategy. Thirdly， the composite multi-objective reward function design method that integrates safety constraints and shaping functions is proposed to ensure training convergence and optimization performance of the deep reinforcement learning agent. Finally， the effectiveness of the proposed method is verified through hardware-in-the-loop experiments. The results show that the proposed strategy can be applied in real-time on the onboard vehicle control unit. Compared to the eco-driving strategy based on the intelligent driver model， the proposed strategy improves energy economy， ride comfort， and travel efficiency of the vehicle while satisfying the driving safety constraints.

Key words: connected and automated vehicle, eco-driving, deep reinforcement learning, urban traffic scenario, multi-objective optimization

Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios[J].Automotive Engineering, 2023, 45(10): 1791-1802.

Figures/Tables 16

References 29

1	HUANG Y， NG E C， ZHOU J L， et al. Eco-driving technology for sustainable road transport： a review［J］. Renewable and Sustainable Energy Reviews， 2018， 93： 596-609.
2	ZHANG F， HU X， LANGARI R， et al. Energy management strategies of connected HEVs and PHEVs： recent progress and outlook［J］. Progress in Energy and Combustion Science， 2019， 73： 235-256.
3	钱立军，陈健，吴冰，等. 考虑驾驶员操作误差的混合动力汽车队列生态驾驶控制［J］. 汽车工程， 2021， 43（7）.
	QIAN L， CHEN J， WU B， et al. Eco⁃driving control for hybrid electric vehicle platoon with consideration of driver operation error［J］. Automotive Engineering， 2021， 43（7）.
4	魏小栋，刘波，冷江昊，等. 基于凸优化的燃料电池汽车节能驾驶研究［J］. 汽车工程， 2022， 44（6）.
	WEI X， LIU B， LENG J， et al. Research on eco-driving of fuel cell vehicles via convex optimization［J］. Automotive Engineering， 2022， 44（6）.
5	KIM J， AHN C. Real-time speed trajectory planning for minimum fuel consumption of a ground vehicle［J］. IEEE Transactions on Intelligent Transportation Systems， 2020， 21（6）： 2324-2338.
6	LI S E， PENG H， LI K， et al. Minimum fuel control strategy in automated car-following scenarios［J］. IEEE Transactions on Vehicular Technology， 2012， 61（3）： 998-1007.
7	KAMALANATHSHARMA R K， RAKHA H A. Leveraging connected vehicle technology and telematics to enhance vehicle fuel efficiency in the vicinity of signalized intersections［J］. Journal of Intelligent Transportation Systems， 2014， 20（1）： 1-12.
8	WU J， ZOU Y， ZHANG X， et al. A hierarchical energy management for hybrid electric tracked vehicle considering velocity planning with pseudospectral method［J］. IEEE Transactions on Transportation Electrification， 2020， 6（2）： 703-716.
9	SHAO Y， SUN Z. Eco-approach with traffic prediction and experimental validation for connected and autonomous vehicles［J］. IEEE Transactions on Intelligent Transportation Systems， 2021， 22（3）： 1562-1572.
10	LI L， WANG X， SONG J. Fuel consumption optimization for smart hybrid electric vehicle during a car-following process［J］. Mechanical Systems and Signal Processing， 2017， 87： 17-29.
11	SUTTON R S， BARTO A G. Reinforcement learning： an introduction［M］. MIT Press， 2018.
12	唐小林，陈佳信，刘腾，等. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究［J］. 机械工程学报， 2021， 57（22）： 237-246.
	TANG X， CHEN J， LIU T， et al. Research on deep reinforcement learning-based intelligent car-following control and energy management strategy for hybrid electric vehicles［J］. Journal of Mechanical Engineering， 2021， 57（22）： 237-246.
13	MNIH V， KAVUKCUOGLU K， SILVER D， et al. Human-level control through deep reinforcement learning［J］. Nature， 2015， 518（7540）： 529-533.
14	FUJIMOTO S， HOOF H， MEGER D. Addressing function approximation error in actor-critic methods［C］. International Conference on Machine Learning， 2018： 1587-1596.
15	LIU B， SUN C， WANG B， et al. Adaptive speed planning of connected and automated vehicles using multi-light trained deep reinforcement learning［J］. IEEE Transactions on Vehicular Technology， 2022， 71（4）： 3533-3546.
16	BAI Z， HAO P， SHANGGUAN W， et al. Hybrid reinforcement learning-based eco-driving strategy for connected and automated vehicles at signalized intersections［J］. IEEE Transactions on Intelligent Transportation Systems， 2022： 1-14.
17	张健，姜夏，史晓宇，等. 基于离线强化学习的交叉口生态驾驶控制［J］. 东南大学学报（自然科学版）， 2022（4）： 052.
	ZHANG J， JIANG X， SHI X， et al. Offline reinforcement learning for eco-driving control at signalized intersections［J］. Journal of Southeast University （Natural Science Edition）， 2022（4）： 052.
18	KRAUß S， WAGNER P， GAWRON C. Metastable states in a microscopic model of traffic flow［J］. Physical Review E， 1997， 55（5）： 5597.
19	LI J， WU X， XU M， et al. Deep reinforcement learning and reward shaping based eco-driving control for automated HEVs among signalized intersections［J］. Energy， 2022， 251： 123924.
20	NG A Y， HARADA D， RUSSELL S. Policy invariance under reward transformations： theory and application to reward shaping［C］. Icml， 1999： 278-287.
21	DEVLIN S M， KUDENKO D. Dynamic potential-based reward shaping［C］. Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems， 2012： 433-440.
22	LILLICRAP T P， HUNT J J， PRITZEL A， et al. Continuous control with deep reinforcement learning［J］. arXiv preprint arXiv：， 2015.
23	SILVER D， LEVER G， HEESS N， et al. Deterministic policy gradient algorithms［C］. International Conference on Machine Learning， 2014： 387-395.
24	Speedgoat real-time simulation and testing［EB/OL］. https：//www.speedgoat.com/.
25	HALBACH S， SHARER P， PAGERIT S， et al. Model architecture， methods， and interfaces for efficient math-based design and simulation of automotive control systems［C］. SAE Paper 2010-01-0241.
26	We are TAVF［EB/OL］. https：//tavf.hamburg/en/we-are-tavf.
27	TAVF-Hamburg Scenario on GitHub［EB/OL］. https：//github.com/DLR-TS/sumo-scenarios/tree/main/TAVF-Hamburg.
28	邹渊，张涛，张旭东，等. 考虑车流影响的网联车辆节能驾驶研究［J］. 汽车工程， 2020， 42（10）.
	ZOU Y， ZHANG T， ZHANG X， et al. Research on eco-driving of connected vehicles considering the influence of traffic flow［J］. Automotive Engineering， 2020， 42（10）.
29	TREIBER M， HENNECKE A， HELBING D. Congested traffic states in empirical observations and microscopic simulations［J］. Physical Review E， 2000， 62（2）： 1805.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	WANG Dan, LIU Zhong-Chang, TIAN Jing, HAN Yong-Qiang, TAN Man-Zhi. [J]. , 2016, 38(12): 1415 -1419 .
[3]	HAO Han, WANG Si-南, LI Xiao, LIU Zong-Wei, ZHAO Fu-Quan. [J]. , 2017, 39(1): 1 -8 .
[4]	XU Cheng-Shan, JIANG Fa-Chao, SONG Sen-Nan, TIAN Guang-Yu. [J]. , 2017, 39(1): 9 -14 .
[5]	ZHANG Lei, ZHANG Jin-Qiu, LUO Tao, BI Zhan-Dong, YAO Jun. [J]. , 2016, 38(12): 1494 -1499 .
[6]	LI Ling, MA Li, MOU Yu, XU Chao, LI Wen-Ru, SHI Shu-Ming. [J]. , 2016, 38(12): 1508 -1514 .
[7]	SHEN Deng-Feng, WANG Chen, YU Hai-Sheng, ZHANG Tong, YI Xian-Ke. [J]. , 2017, 39(1): 15 -22 .
[8]	Chen-Xin, FENG Xiao, SHEN Chuan-Liang, WANG Shuo, HU Cui-Song, LI Yan-Yang, YANG Chang-Hai-　. [J]. , 2017, 39(2): 206 -213 .
[9]	SONG Ke, ZHANG Tao, NIU Wen-Xu, ZHANG Tong. [J]. , 2017, 39(3): 249 -255 .
[10]	ZHANG Yong, GU Zheng-Qi, LIU Shui-Chang, MI Cheng-Ji. [J]. , 2017, 39(3): 275 -280 .

超参数			数值
记忆缓冲区容量			2²¹
高斯探索噪声标准差			0.8
批采样大小			256
折扣因子			0.99
Actor网络学习率			0.000 1
Critic网络学习率			0.000 15
目标网络更新率			0.005
Critic网络更新噪声标准差			0.2
策略更新延迟的频率			4
Critic网络更新噪声范围			0.5
网络结构	Actor网络		Critic网络1和2
网络结构	节点	激活函数	节点	激活函数
输入层	12	ReLU	13	ReLU
隐藏层	128		128
隐藏层	128		128
隐藏层	128		128
隐藏层	64		64
隐藏层	64		64
输出层	1	tanh	1

车辆参数	数值	单位
车辆质量	1 835	kg
电机峰值转矩	386	N·m
电机峰值功率	202	kW
电池电压	345.6	V
电池容量	173.5	A·h
主减速器传动比	9.1
轮胎半径	0.344	m

参数	IDM	DRL	数值对比
前车最小距离/m	0.884 7	0.976 8	+10.41 %
加速度绝对值平均/（m·s^-2）	0.551 7	0.495 0	-10.28 %
电池电耗/%	1.644	1.464	-10.94 %
电机损失电能/kJ	1 445	1 287	-10.93 %
总行驶时间/s	1 242	1 255	+1.047 %
累积红灯停车时间/s	337	56	-83.38 %

[1]	Meng Xiong,Dong Zhang,Guojian You,Tianfei Sun,Kai Sheng,Xuezhe Wei. Multi-objective Optimization Design of High Efficiency and High Utilization Magnetic Core of Wireless Charging of Electric Vehicles [J]. Automotive Engineering, 2023, 45(9): 1740-1752.
[2]	Jizheng Liu,Zhenpo Wang,Fengchun Sun,Lei Zhang. Research on Delay Compensation Control for Heterogeneous Connected and Automated Vehicle Platoons [J]. Automotive Engineering, 2023, 45(9): 1573-1582.
[3]	Yuan Zou,Wenjing Sun,Xudong Zhang,Ya Wen,Wanke Cao,Zhaolong Zhang. Multi-objective Optimization of In-Vehicle Ethernet Network Architecture for Time-Sensitive Network [J]. Automotive Engineering, 2023, 45(5): 746-758.
[4]	Yapeng Li,Xiaolin Tang,Xiaosong Hu. Study on Eco-driving of PHEVS Based on Hierarchical Control Strategy [J]. Automotive Engineering, 2023, 45(4): 551-560.
[5]	Zhiyong Duan,Jing Ma. Multi-objective Optimization of Lithium Battery Composite Cooling Structure Based on Heat Pipes and Liquid Cooling Plate [J]. Automotive Engineering, 2023, 45(11): 2047-2057.
[6]	Lijun Qian,Chen Chen,Jian Chen,Xinyu Chen,Chi Xiong. Discrete Platoon Control at an Unsignalized Intersection Based on Q-learning Model [J]. Automotive Engineering, 2022, 44(9): 1350-1358.
[7]	Xinyu Chen,Lijun Qian,Qidong Wang. Eco-driving Control at Signalized Intersections with Consideration of Time Delay [J]. Automotive Engineering, 2022, 44(7): 960-968.
[8]	Xiaodong Wei,Bo Liu,Jianghao Leng,Xingyu Zhou,Chao Sun,Fengchun Sun. Research on Eco-driving of Fuel Cell Vehicles via Convex Optimization [J]. Automotive Engineering, 2022, 44(6): 851-858.
[9]	Hao Zhang,Qinhao Fan,Wei Wang,Jin Huang,Zhi Wang. Reinforcement Learning Based Energy Management Strategy for Hybrid Electric Vehicles Using Multi⁃mode Combustion [J]. Automotive Engineering, 2021, 43(5): 683-691.
[10]	Jinghua Guo,Wenchang Li,Yugong Luo,Tao Chen,Keqiang Li. Driver Car⁃Following Model Based on Deep Reinforcement Learning [J]. Automotive Engineering, 2021, 43(4): 571-579.
[11]	Zhenhai Gao,Xiangtong Yan,Fei Gao,Tianjun Sun. A Driver-like Decision-making Method for Longitudinal Autonomous Driving Based on DDPG [J]. Automotive Engineering, 2021, 43(12): 1737-1744.
[12]	Zhonghua Tang,Yansong He,Tao Ma,Zhifei Zhang,Hongjie Pu,Yun Li,Zhao Chen. Lightweight Design of Automotive Sound Package [J]. Automotive Engineering, 2021, 43(1): 113-120.
[13]	Zhang Zhifei, Xue Haoxiang, Chen Zhao, Pu Hongjie, Xu Zhongming, He Yansong. Optimization of Front Suspension and Steering System Based on Grey Correlation TOPSIS Method [J]. Automotive Engineering, 2020, 42(8): 1082-1089.
[14]	Wu Jingwei, Zhang Guangya, Lü Juncheng, Li Qian. Methodology Research on Optimization Design of Vehicle Interior Space Under Constraint of Body Performance [J]. Automotive Engineering, 2020, 42(8): 1117-1123.
[15]	He Liangguo, Zhao Jie, Gu Xianguang. Lightweight and Crashworthiness Design of Vehicle BodyFront-end Based on Multi-cell Structure [J]. Automotive Engineering, 2020, 42(6): 832-839.

Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 29

Related Articles 15

Metrics

Comments

Recommended 10