基于逆强化学习的混合动力汽车能量管理策略研究

doi:10.19562/j.chinasae.qcgc.2023.10.016

摘要/Abstract

摘要：

能量管理策略是混合动力汽车关键技术之一。随着计算能力与硬件设备的不断升级，越来越多的学者逐步开展了基于学习的能量管理策略的研究。在基于强化学习的混合动力汽车能量管理策略研究中，智能体与环境相互作用的导向是由奖励函数决定。然而，目前的奖励函数设计多数是主观决定或者根据经验得来的，很难客观地描述专家的意图，所以在该条件不能保证智能体在给定奖励函数下学习到最优驾驶策略。针对这些问题，本文提出了一种基于逆向强化学习的能量管理策略，通过逆向强化学习的方法获取专家轨迹下的奖励函数权值，并用于指导发动机智能体和电池智能体的行为。之后将修改后的权重重新输入正向强化学习训练。从油耗值、SOC变化曲线、奖励训练过程、动力源转矩等方面，验证该权重值的准确性以及在节油能力方面具有一定的优势。综上所述，该算法的节油效果提高了5%~10%。

关键词: 混合动力汽车, 最大熵逆向强化学习, 能量管理策略, 正向强化学习

Abstract:

Energy management strategy is one of the key technologies for hybrid vehicles. With the continuous upgrading of computing power and hardware devices， more and more scholars have gradually carried out research on learning-based energy management strategies. In the study of reinforcement learning-based energy management strategies for hybrid electric vehicles， the orientation of the interaction between the intelligent agent and the environment is determined by the reward function. However， most of the current reward function design is subjectively determined or based on experience， which is difficult to objectively describe the expert's intention， so in that condition there is no guarantee that the intelligent body will learn the optimal driving strategy for a given reward function. To address these problems， an energy management strategy based on inverse reinforcement learning is proposed in this paper to obtain the reward function weights under the expert trajectory by means of inverse reinforcement learning and use them to guide the behavior of the engine and battery intelligent agents. Then， the modified weights are input again into the positive reinforcement learning training. The fuel consumption value， SOC variation curve， reward training process and power source torque are used to verify the accuracy of the weight value and its advantage in terms of fuel saving capability. In summary， the algorithm has improved the fuel saving effect by 5%~10%.

Key words: hybrid electric vehicle, maximum entropy reverse reinforcement learning, energy management strategy, positive reinforcement learning

齐春阳,宋传学,宋世欣,靳立强,王达,肖峰. 基于逆强化学习的混合动力汽车能量管理策略研究[J]. 汽车工程, 2023, 45(10): 1954-1964.

Chunyang Qi,Chuanxue Song,Shixin Song,Liqiang Jin,Da Wang,Feng Xiao. Research on Energy Management Strategy for Hybrid Electric Vehicles Based on Inverse Reinforcement Learning[J]. Automotive Engineering, 2023, 45(10): 1954-1964.

图/表 22

图1

表1

本文研究对象参数"

部件	参数	数值
整车	整备质量	1 449 kg
	轴距	2 700 mm
	风阻系数	0.26
	迎风面积	2.23 $m 2$
	滚动阻尼系数	0.013
MG1	最大功率	50 kW
	最大转矩	400 N·m
	最高转速	6 000 r/min
MG2	最大功率	37.8 kW
	最大转矩	75 N·m
	最高转速	10 000 r/min
发动机	最大输出功率	56 kW
	最大转矩	120 N·m
	最高转速	4 000 r/min
电池	电池容量	1.54 kW·h
电池	电压	237 V
变速器	传动比	3.93

表1

图2

表2

逆向强化学习获得奖励参数伪代码"

算法：逆向强化学习参数标定

输入：发动机最优工作点与SOC最优变化范围

输出：强化学习多智能体奖励函数参数值

1. 初始化奖励函数权重 $θ$ ，发动机智能体，电池智能体

2. 计算专家期望

3. 计算 $D s$ ：

循环：

$Z a i, j = ∑ k P s k | s i, a i, j e x p ? (r θ (s i)) Z s k$

$Z s i = ∑ a i, j Z a i, j + [s i ∈ S t e r m i n a l]$

$P o l i c y = Z a i, j / Z s i$

$D s = ∑ s i ∑ a i, j P (s i = s i n i t i a l) P o l i c y P s k | s i, a i, j$

4. 通过梯度更新奖励函数权重：

$θ t + 1 = θ t + α (f ˉ - ∑ s ∈ S D s f s)$

5. 构造奖励函数并将其输入到正强化学习环境中

表2

图3

图4

图5

图6

表3

图7

图8

图9

图10

图11

图12

图13

图14

表4

图15

图16

图17

图18

参考文献 23

1	MAHAPATRA S， EGEL T， HASSAN R，et al. Model based design for hybrid electric vehicle systems［J］. Sensors， 2020，20.
2	李顶根，严索，邱君诚，等. 基于规则控制的PHHV能量管理策略研究［J］. 汽车工程学报， 2014，4（2）：116-124.
	LI D G， YAN S， QIU J C， et al. Rule-based power management strategy for parallel hydraulic hybrid vehicle［J］. Chinese Journal of Automotive Engineering，2014， 4（2）： 116-124.
3	庞玉涵. 基于分层强化学习的混合动力汽车能量管理策略研究［D］.重庆：重庆理工大学， 2022.
	PANG Y H. Research on energy management strategy of hybrid electric vehicle based on hierarchical reinforcement learning［D］.Chongqing： Chongqing University of Technoligy， 2022.
4	刘腾. 混合动力车辆强化学习能量管理研究［D］.北京：北京理工大学， 2017.
	LIU T. Reinforcement learning-based energy management for hybrid electric vehicles［D］.Beijing： Beijing Institute of Technology， 2017.
5	唐小林，陈佳信，刘腾，等. 基于深度强化学习的混合动力汽车智能跟车控制与能量管理策略研究［J］. 机械工程学报， 2021， 57（22）：237-246.
	TANG X L， CHEN J X， LIU T， et al. Research on deep reinforcement learning-based intelligent car-following control and energy management strategy for hybrid electric vehicles［J］. Journal of Mechanical Engineering，2021， 57（22）：237-246.
6	LI Y， HE H， PENG J， et al. Energy management strategy for a series hybrid electric vehicle using improved deep Q-network learning algorithm with prioritized replay［C］. DEStech Transactions on Environment Energy and Earth Science， 2019（ICEEE）.
7	CHAOUI H， GUALOUS H， BOULON L， et al. Deep reinforcement learning energy management system for multiple battery based electric vehicles［C］. IEEE Vehicle Power & Propulsion Conference， 2018.
8	LIU C， YI L M. Power management for plug-in hybrid electric vehicles using reinforcement learning with trip information［C］. Transportation Electrification Conference & Expo， 2014.
9	LIU T， HU X， LI S E， et al. Reinforcement learning optimized look-ahead energy management of a parallel hybrid electric vehicle［J］. IEEE Asme Transactions on Mechatronics， 2017.
10	YANG N， HAN L， XIANG C， et al. Energy management for a hybrid electric vehicle based on blended reinforcement learning with backward focusing and prioritized sweeping［J］. IEEE Transactions on Vehicular Technology，2021，PP（99）.
11	LIAN R， TAN H， PENG J， et al. Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management［J］. IEEE Transactions on Vehicular Technology， 2020，69（8）：8367-8380.
12	QI Chunyang， ZHU Yiwen， SONG Chuanxue， et al. Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle［J］. Energy，2022，238.
13	DU G， ZOU Y， ZHANG X， et al. Heuristic energy management strategy of hybrid electric vehicle based on deep reinforcement learning with accelerated gradient optimization［J］. IEEE Transactions on Transportation Electrification， 2021，7（4）.
14	LIAN R， PENG J， WU Y， et al. Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle［J］. Energy， 2020，197：117297.
15	HE D， ZOU Y， WU J， et al. Deep Q-learning based energy management strategy for a series hybrid electric tracked vehicle and its adaptability validation［C］. IEEE Transportation Electrification Conference & Expo， 2019.
16	GUO X. Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle［J］. IEEE Access， 2020，8： 165837-165848.
17	DU G， ZOU Y， ZHANG X， et al. Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning［J］. Applied Energy， 2019，251.
18	LIN X， ZHOU K， MO L， et al. Intelligent energy management strategy based on an improved reinforcement learning algorithm with exploration factor for a plug-in PHEV［J］. IEEE Transactions on Intelligent Transportation Systems， 2021，PP（99）：1-11.
19	XU B， HOU J， SHI J， et al. Learning time reduction using warm-start methods for a reinforcement learning-based supervisory control in hybrid electric vehicle applications［C］. IEEE， 2021.
20	XU B， HU X， TANG X， et al. Ensemble reinforcement learning-based supervisory control of hybrid electric vehicle for fuel economy improvement［J］. IEEE Transactions on Transportation Electrification， 2020，6（2）：717-727.
21	HAN X， HE H， WU J， et al. Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle［J］. Applied Energy， 2019，254：113708.
22	ZHOU J， XUE S， XUE Y，et al. A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning［J］. Energy，2021（5）：120118.
23	LI Y， TAO J，XIE L， et al. Enhanced Q-learning for real-time hybrid electric vehicle energy management with deterministic rule［J］. Measurement and Control： Journal of the Institute of Measurement and Control， 2020，53（7/8）.

[1]	李亚鹏,唐小林,胡晓松. 基于分层式控制的混合动力汽车生态驾驶研究[J]. 汽车工程, 2023, 45(4): 551-560.
[2]	刘晋霞,梁志豪,王强,温明星. TTR汽车等效燃油消耗最小的能量管理策略研究[J]. 汽车工程, 2023, 45(2): 285-292.
[3]	陈勇,魏长银,李晓宇,李彦林,刘彩霞,林霄喆. 融合工况识别的增程式电动汽车模糊能量管理策略研究[J]. 汽车工程, 2022, 44(4): 514-524.
[4]	刘志超,郑天雷,柳邵辉,龚慧明. 插电式混合动力乘用车能耗和续驶里程评价方法研究[J]. 汽车工程, 2022, 44(2): 161-170.
[5]	李军求,刘吉威,朱超峰. 基于时变底盘构型的混动车辆能量管理研究[J]. 汽车工程, 2022, 44(12): 1866-1876.
[6]	施德华,容香伟,汪少华,张开美,陈龙,李春. 双离合器协同的功率分流式混合动力汽车动态协调优化控制研究[J]. 汽车工程, 2022, 44(12): 1877-1888.
[7]	钱立军,陈健,吴冰,宣亮,陈晨,陈亮亮. 考虑驾驶员操作误差的混合动力汽车队列生态驾驶控制[J]. 汽车工程, 2021, 43(7): 1037-1045.
[8]	杨宁康,韩立金,刘辉,张欣. 基于效率优化的混合动力车辆强化学习能量管理策略研究[J]. 汽车工程, 2021, 43(7): 1046-1056.
[9]	解少博,罗慧冉,张乾坤,张康康. 智能网联混合动力车辆速度规划的多目标协同控制研究[J]. 汽车工程, 2021, 43(7): 953-961.
[10]	解少博,张康康,张乾坤,罗慧冉. 考虑电池电-热-放电深度的并联PHEV能量管理策略研究[J]. 汽车工程, 2021, 43(6): 791-798.
[11]	王长卉,刘泽民,李雁飞,帅石金,邵慧芳,王银辉. 电量保持阶段增程器的排放与工作特性的试验研究[J]. 汽车工程, 2021, 43(5): 667-674.
[12]	胡杰,王明,刘迪,颜伏伍,曹恺. 基于交通信息的复合电源系统控制策略优化[J]. 汽车工程, 2021, 43(5): 675-682.
[13]	张昊,范钦灏,王巍,黄晋,王志. 基于强化学习的多燃烧模式混合动力能量管理策略[J]. 汽车工程, 2021, 43(5): 683-691.
[14]	陈龙,陈智星,徐兴,王峰,蔡英凤,张涛. 考虑发动机起停优化的PHEV能量管理策略[J]. 汽车工程, 2021, 43(3): 313-322.
[15]	李跃娟,齐巍,王成,张博,卢强. 并联混合动力汽车ECMS的时变等效因子提取算法的研究[J]. 汽车工程, 2021, 43(2): 181-188.