基于逆强化学习的混合动力汽车能量管理策略研究

doi:10.19562/j.chinasae.qcgc.2023.10.016

Abstract

Abstract:

Energy management strategy is one of the key technologies for hybrid vehicles. With the continuous upgrading of computing power and hardware devices， more and more scholars have gradually carried out research on learning-based energy management strategies. In the study of reinforcement learning-based energy management strategies for hybrid electric vehicles， the orientation of the interaction between the intelligent agent and the environment is determined by the reward function. However， most of the current reward function design is subjectively determined or based on experience， which is difficult to objectively describe the expert's intention， so in that condition there is no guarantee that the intelligent body will learn the optimal driving strategy for a given reward function. To address these problems， an energy management strategy based on inverse reinforcement learning is proposed in this paper to obtain the reward function weights under the expert trajectory by means of inverse reinforcement learning and use them to guide the behavior of the engine and battery intelligent agents. Then， the modified weights are input again into the positive reinforcement learning training. The fuel consumption value， SOC variation curve， reward training process and power source torque are used to verify the accuracy of the weight value and its advantage in terms of fuel saving capability. In summary， the algorithm has improved the fuel saving effect by 5%~10%.

Key words: hybrid electric vehicle, maximum entropy reverse reinforcement learning, energy management strategy, positive reinforcement learning

Chunyang Qi,Chuanxue Song,Shixin Song,Liqiang Jin,Da Wang,Feng Xiao. Research on Energy Management Strategy for Hybrid Electric Vehicles Based on Inverse Reinforcement Learning[J].Automotive Engineering, 2023, 45(10): 1954-1964.

Figures/Tables 22

部件	参数	数值
整车	整备质量	1 449 kg
	轴距	2 700 mm
	风阻系数	0.26
	迎风面积	2.23 $m 2$
	滚动阻尼系数	0.013
MG1	最大功率	50 kW
	最大转矩	400 N·m
	最高转速	6 000 r/min
MG2	最大功率	37.8 kW
	最大转矩	75 N·m
	最高转速	10 000 r/min
发动机	最大输出功率	56 kW
	最大转矩	120 N·m
	最高转速	4 000 r/min
电池	电池容量	1.54 kW·h
电池	电压	237 V
变速器	传动比	3.93

算法：逆向强化学习参数标定

输入：发动机最优工作点与SOC最优变化范围

输出：强化学习多智能体奖励函数参数值

1. 初始化奖励函数权重 $θ$ ，发动机智能体，电池智能体

2. 计算专家期望

3. 计算 $D s$ ：

循环：

$Z a i, j = ∑ k P s k | s i, a i, j e x p ? (r θ (s i)) Z s k$

$Z s i = ∑ a i, j Z a i, j + [s i ∈ S t e r m i n a l]$

$P o l i c y = Z a i, j / Z s i$

$D s = ∑ s i ∑ a i, j P (s i = s i n i t i a l) P o l i c y P s k | s i, a i, j$

4. 通过梯度更新奖励函数权重：

$θ t + 1 = θ t + α (f ˉ - ∑ s ∈ S D s f s)$

5. 构造奖励函数并将其输入到正强化学习环境中

References 23

1	MAHAPATRA S， EGEL T， HASSAN R，et al. Model based design for hybrid electric vehicle systems［J］. Sensors， 2020，20.
2	李顶根，严索，邱君诚，等. 基于规则控制的PHHV能量管理策略研究［J］. 汽车工程学报， 2014，4（2）：116-124.
	LI D G， YAN S， QIU J C， et al. Rule-based power management strategy for parallel hydraulic hybrid vehicle［J］. Chinese Journal of Automotive Engineering，2014， 4（2）： 116-124.
3	庞玉涵. 基于分层强化学习的混合动力汽车能量管理策略研究［D］.重庆：重庆理工大学， 2022.
	PANG Y H. Research on energy management strategy of hybrid electric vehicle based on hierarchical reinforcement learning［D］.Chongqing： Chongqing University of Technoligy， 2022.
4	刘腾. 混合动力车辆强化学习能量管理研究［D］.北京：北京理工大学， 2017.
	LIU T. Reinforcement learning-based energy management for hybrid electric vehicles［D］.Beijing： Beijing Institute of Technology， 2017.
5	唐小林，陈佳信，刘腾，等. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究［J］. 机械工程学报， 2021， 57（22）：237-246.
	TANG X L， CHEN J X， LIU T， et al. Research on deep reinforcement learning-based intelligent car-following control and energy management strategy for hybrid electric vehicles［J］. Journal of Mechanical Engineering，2021， 57（22）：237-246.
6	LI Y， HE H， PENG J， et al. Energy management strategy for a series hybrid electric vehicle using improved deep Q-network learning algorithm with prioritized replay［C］. DEStech Transactions on Environment Energy and Earth Science， 2019（ICEEE）.
7	CHAOUI H， GUALOUS H， BOULON L， et al. Deep reinforcement learning energy management system for multiple battery based electric vehicles［C］. IEEE Vehicle Power & Propulsion Conference， 2018.
8	LIU C， YI L M. Power management for plug-in hybrid electric vehicles using reinforcement learning with trip information［C］. Transportation Electrification Conference & Expo， 2014.
9	LIU T， HU X， LI S E， et al. Reinforcement learning optimized look-ahead energy management of a parallel hybrid electric vehicle［J］. IEEE Asme Transactions on Mechatronics， 2017.
10	YANG N， HAN L， XIANG C， et al. Energy management for a hybrid electric vehicle based on blended reinforcement learning with backward focusing and prioritized sweeping［J］. IEEE Transactions on Vehicular Technology，2021，PP（99）.
11	LIAN R， TAN H， PENG J， et al. Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management［J］. IEEE Transactions on Vehicular Technology， 2020，69（8）：8367-8380.
12	QI Chunyang， ZHU Yiwen， SONG Chuanxue， et al. Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle［J］. Energy，2022，238.
13	DU G， ZOU Y， ZHANG X， et al. Heuristic energy management strategy of hybrid electric vehicle based on deep reinforcement learning with accelerated gradient optimization［J］. IEEE Transactions on Transportation Electrification， 2021，7（4）.
14	LIAN R， PENG J， WU Y， et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle［J］. Energy， 2020，197：117297.
15	HE D， ZOU Y， WU J， et al. Deep Q-learning based energy management strategy for a series hybrid electric tracked vehicle and its adaptability validation［C］. IEEE Transportation Electrification Conference & Expo， 2019.
16	GUO X. Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle［J］. IEEE Access， 2020，8： 165837-165848.
17	DU G， ZOU Y， ZHANG X， et al. Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning［J］. Applied Energy， 2019，251.
18	LIN X， ZHOU K， MO L， et al. Intelligent energy management strategy based on an improved reinforcement learning algorithm with exploration factor for a plug-in PHEV［J］. IEEE Transactions on Intelligent Transportation Systems， 2021，PP（99）：1-11.
19	XU B， HOU J， SHI J， et al. Learning time reduction using warm-start methods for a reinforcement learning-based supervisory control in hybrid electric vehicle applications［C］. IEEE， 2021.
20	XU B， HU X， TANG X， et al. Ensemble reinforcement learning-based supervisory control of hybrid electric vehicle for fuel economy improvement［J］. IEEE Transactions on Transportation Electrification， 2020，6（2）：717-727.
21	HAN X， HE H， WU J， et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle［J］. Applied Energy， 2019，254：113708.
22	ZHOU J， XUE S， XUE Y，et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning［J］. Energy，2021（5）：120118.
23	LI Y， TAO J，XIE L， et al. Enhanced Q-learning for real-time hybrid electric vehicle energy management with deterministic rule［J］. Measurement and Control： Journal of the Institute of Measurement and Control， 2020，53（7/8）.

[1]	ZHOU Wei, ZHANG Cheng-Ning, LI Jun-Qiu. [J]. , 2016, 38(12): 1407 -1414 .
[2]	HAO Han, WANG Si-南, LI Xiao, LIU Zong-Wei, ZHAO Fu-Quan. [J]. , 2017, 39(1): 1 -8 .
[3]	XU Cheng-Shan, JIANG Fa-Chao, SONG Sen-Nan, TIAN Guang-Yu. [J]. , 2017, 39(1): 9 -14 .
[4]	ZENG Bi-Qiang, GAO Ji-Dong, PENG Wei, SUN Zhen-Dong. [J]. , 2016, 38(12): 1446 -1451 .
[5]	LI Ling, MA Li, MOU Yu, XU Chao, LI Wen-Ru, SHI Shu-Ming. [J]. , 2016, 38(12): 1508 -1514 .
[6]	CUI An, LI Bin, WANG Xue-Liang, ZHANG Shi-Zhan. [J]. , 2016, 38(12): 1521 -1525 .
[7]	QIN Xiao-Hui, WANG Jian-Qiang, XIE Bo-Yuan, HU Man-Jiang, LI Ke-Qiang. [J]. , 2017, 39(1): 73 -78 .
[8]	ZHAO Liang, ZHANG Qiang, GUO Kong-Hui. [J]. , 2017, 39(1): 86 -91 .
[9]	YU Zhuo-Ping, HAN Wei, XIONG Lu. [J]. , 2017, 39(1): 52 -60 .
[10]	CUI Jun-Jia, YUAN Wei, LI Guang-Yao. [J]. , 2017, 39(1): 113 -120 .

[1]	Yapeng Li,Xiaolin Tang,Xiaosong Hu. Study on Eco-driving of PHEVS Based on Hierarchical Control Strategy [J]. Automotive Engineering, 2023, 45(4): 551-560.
[2]	Jinxia Liu,Zhihao Liang,Qiang Wang,Mingxing Wen. Research on the Equivalent Fuel Consumption Minimizing Energy Management Strategy of Through-the-Road Vehicle [J]. Automotive Engineering, 2023, 45(2): 285-292.
[3]	Yong Chen,Changyin Wei,Xiaoyu Li,Yanlin Li,Caixia Liu,Xiaozhe Lin. Research on Fuzzy Energy Management Strategy for Extended-Range Electric Vehicles with Driving Condition Identification [J]. Automotive Engineering, 2022, 44(4): 514-524.
[4]	Zhichao Liu,Tianlei Zheng,Shaohui Liu,Huiming Gong. Research on Energy Consumption and Driving Range Evaluation Method of Plug-in Hybrid Electric Passenger Car [J]. Automotive Engineering, 2022, 44(2): 161-170.
[5]	Junqiu Li,Jiwei Liu,Chaofeng Zhu. Research on Energy Management of Hybrid Electric Vehicle Based on Time-Varying Chassis Configuration [J]. Automotive Engineering, 2022, 44(12): 1866-1876.
[6]	Dehua Shi,Xiangwei Rong,Shaohua Wang,Kaimei Zhang,Long Chen,Chun Li. Research on the Optimal Dynamic Coordinated Control of Power Split Hybrid Electric Vehicles with Dual-Clutch Collaboration [J]. Automotive Engineering, 2022, 44(12): 1877-1888.
[7]	Lijun Qian,Jian Chen,Bing Wu,Liang Xuan,Chen Chen,Liangliang Chen. Eco⁃driving Control for Hybrid Electric Vehicle Platoon with Consideration of Driver Operation Error [J]. Automotive Engineering, 2021, 43(7): 1037-1045.
[8]	Ningkang Yang,Lijin Han,Hui Liu,Xin Zhang. Research on Efficiency Optimization Based Energy Management Strategy for a Hybrid Electric Vehicle with Reinforcement Learning [J]. Automotive Engineering, 2021, 43(7): 1046-1056.
[9]	Shaobo Xie,Kangkang Zhang,Qiankun Zhang,Huiran Luo. Study on Energy Management Strategy for Parallel Plug⁃in Hybrid Electric Vehicles Considering Battery Electric⁃Thermal⁃Depth⁃of⁃Discharge [J]. Automotive Engineering, 2021, 43(6): 791-798.
[10]	Changhui Wang,Zemin Liu,Yanfei Li,Shijin Shuai,Huifang Shao,Yinhui Wang. An Experimental Study on Emissions and Operation Characteristics of a Range Extender Under Charge Sustaining Mode [J]. Automotive Engineering, 2021, 43(5): 667-674.
[11]	Jie Hu,Ming Wang,Di Liu,Fuwu Yan,Kai Cao. Optimization of Control Strategy for Hybrid Power System Based on Traffic Information [J]. Automotive Engineering, 2021, 43(5): 675-682.
[12]	Hao Zhang,Qinhao Fan,Wei Wang,Jin Huang,Zhi Wang. Reinforcement Learning Based Energy Management Strategy for Hybrid Electric Vehicles Using Multi⁃mode Combustion [J]. Automotive Engineering, 2021, 43(5): 683-691.
[13]	Long Chen,Zhixing Chen,Xing Xu,Feng Wang,Yingfeng Cai,Tao Zhang. Energy Management Strategy of Plug⁃in Hybrid Electric Vehicle with Engine Start and Stop Optimization [J]. Automotive Engineering, 2021, 43(3): 313-322.
[14]	Xiaohua Zeng,Hongxu Chen,Dafeng Song,Chen Cui,Zhanjiang Li,Yuanguang Jiang. Braking Control Strategy Based on Optimal Regenerative Braking Torque Curve of Motor [J]. Automotive Engineering, 2021, 43(2): 162-170.
[15]	Yuejuan Li,Wei Qi,Cheng Wang,Bo Zhang,Qiang Lu. Study on Extraction Algorithm for Time⁃varying Equivalent Factor of ECMS for Parallel Hybrid Electric Vehicle [J]. Automotive Engineering, 2021, 43(2): 181-188.

Research on Energy Management Strategy for Hybrid Electric Vehicles Based on Inverse Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 22

References 23

Related Articles 15

Metrics

Comments

Recommended 10