融合PER和TL的燃料电池客车能量管理策略更新方法研究

doi:10.19562/j.chinasae.qcgc.2025.12.006

Abstract

Abstract:

For the problems of low training efficiency and delayed updating in deep reinforcement learning-based energy management strategies （EMSs）， taking the fuel cell bas as the research object， an intelligent EMS updating method integrating prioritized experience replay （PER） and transfer learning （TL） for fuel cell buses is proposed in this paper. A sampling mechanism-enhanced soft actor-critic （ESAC） algorithm is designed to improve EMS training efficiency by incorporating PER into the SAC framework. Furthermore， a TL-based EMS updating method is proposed to enhance the updating efficiency and long-term optimization performance by leveraging the knowledge-sharing mechanism for cross-cycle knowledge transfer and policy reuse of the ESAC-based EMS. Finally， the updated EMS is deployed to the energy management controller for online power distribution optimization. The experimental simulation results show that， compared with SAC， the proposed ESAC algorithm improves training efficiency by 58.33%. Additionally， the proposed updating method enhances EMS updating efficiency by 63.01% and fuel economy by 5.24% over baseline methods， while demonstrating real-time application potential.

Key words: transfer learning, prioritized experience replay, soft actor-critic, energy management strategy updating, fuel cell bus

Ruchen Huang,Hongwen He. Research on Updating Method of Energy Management Strategy for Fuel Cell Bus with Integrated PER and TL[J].Automotive Engineering, 2025, 47(12): 2336-2345.

Figures/Tables 17

输入：训练工况输出：EMS策略网络参数 $?$
1.	初始化：神经网络参数 $?$ ， $θ 1$ ， $θ 2$ ， $θ ˉ 1 ← θ 1$ ， $θ ˉ 2 ← θ 2$ 初始化：经验池 $D$ ，二叉树，优先级 $p$ =1，指针point=0，最大训练回合数M，批次样本数N
2.	forepis = 1 to M，do
3.	获取FCB的初始状态 $s 0$
4.	fort = 1 to T，do
5.	基于当前状态选择动作 $a t = π ? ? \| s t$
6.	执行动作 $a t$ ，得到奖励 $r t$ ，观察新状态 $s t + 1$
7.	将四元组 $s t, a t, r t, s t + 1$ 存储进 $D$ ，并更新二叉树
8.	point = point + 1
9.	ifpoint ≥ $B$ ，do
10.	fork = 1 to N，do
11.	根据式（13）计算采样概率
12.	通过二叉树从 $D$ 中采样四元组 $s k, a k, r k, s k + 1$
13.	根据式（14）计算重要性采样权重
14.	end for
15.	选择目标动作 $a t + 1 = π ? ? \| s t + 1$
16.	根据式（7）计算目标Q值
17.	根据式（15）更新价值网络参数
18.	根据式（9）更新策略网络参数
19.	根据式（16）更新目标价值网络参数
20.	根据式（10）更新温度因子
21.	根据式（11）重新计算TD误差
22.	根据式（12）在二叉树中更新样本优先级
23.	end if
24.	end for
25.	end for

1.	预训练：ESAC智能体从源任务的工况中学习特征表示，得到初始EMS，表示为 $π E S A C O, i i = 1,2, …, n$ ， $n$ 表示预训练的次数
2.	网络迁移：将初始EMS的浅层DNN参数迁移至目标任务ESAC智能体的相应DNN层，并随机初始化其它层的参数
3.	微调：在目标工况下对步骤2中已完成初始化的ESAC智能体进行微调再训练，直至稳定收敛，便得到目标EMS，表示为 $π E S A C T, i$ ，并将微调过程表示为 $T E S A C ?$

References 25

[1]	AJRIAN S， MOSTAFAEE A， SAFARZADEH S. Optimal pricing， subsidies， and cost-sharing rate in fuel cell bus supply chain： a nash and stackelberg framework analysis［J］. Energy， 2025： 135603.
[2]	WU J D， PENG J K， LI M L， et al. Enhancing fuel cell electric vehicle efficiency with TIP-EMS： a trainable integrated predictive energy management approach［J］. Energy Conversion and Management， 2024， 310： 118499.
[3]	王亚雄，范依莹，欧凯，等. 融合交通信息的燃料电池汽车能量管理研究进展［J］. 汽车工程， 2024， 46（12）： 2314-2328，2338.
	WANG Y X， FAN Y Y， OU K， et al. Research progress on traffic information-integrated energy management for fuel cell vehicles［J］. Automotive Engineering， 2024， 46（12）： 2314-2328，2338.
[4]	魏小栋，刘波，冷江昊，等. 基于凸优化的燃料电池汽车节能驾驶研究［J］. 汽车工程， 2022， 44（6）： 851-858.
	WEI X D， LIU B， LENG J H， et al. Research on eco-driving of fuel cell vehicles via convex optimization［J］. Automotive Engineering， 2022， 44（6）： 851-858.
[5]	LIN X Y， XU X H， WANG Z R. Deep Q-learning network based trip pattern adaptive battery longevity-conscious strategy of plug-in fuel cell hybrid electric vehicle［J］. Applied Energy， 2022， 321： 119378.
[6]	TANG X L， CHEN J X， QIN Y C， et al. Reinforcement learning-based energy management for hybrid power systems： state-of-the-art survey， review， and perspectives［J］. Chinese Journal of Mechanical Engineering， 2024， 37（1）： 43.
[7]	LIU T， ZOU Y， LIU D X， et al. Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle［J］. IEEE Transactions on Industrial Electronics， 2015， 62（12）： 7837-7846.
[8]	YANG D， WANG L， YU K J， et al. A reinforcement learning-based energy management strategy for fuel cell hybrid vehicle considering real-time velocity prediction［J］. Energy Conversion and Management， 2022， 274： 116453.
[9]	YANG N K， HAN L J， LIU R， et al. Multiobjective intelligent energy management for hybrid electric vehicles based on multiagent reinforcement learning［J］. IEEE Transactions on Transportation Electrification， 2023， 9（3）： 4294-4305.
[10]	WURMAN P R， BARRETT S， KAWAMOTO K， et al. Outracing champion Gran Turismo drivers with deep reinforcement learning［J］. Nature， 2022， 602（7896）： 223-228.
[11]	LI J W， WANG H X， HE H W， et al. Battery optimal sizing under a synergistic framework with DQN-based power managements for the fuel cell hybrid powertrain［J］. IEEE Transactions on Transportation Electrification， 2021， 8（1）： 36-47.
[12]	ZHENG C H， ZHANG D F， XIAO Y， et al. Reinforcement learning-based energy management strategies of fuel cell hybrid vehicles with multi-objective control［J］. Journal of Power Sources， 2022， 543： 231841.
[13]	WU J D， WEI Z B， LIU K L， et al. Battery-involved energy management for hybrid electric bus based on expert-assistance deep deterministic policy gradient algorithm［J］. IEEE Transactions on Vehicular Technology， 2020， 69（11）： 12786-12796.
[14]	HUANG R C， HE H W， ZHAO X Y， et al. Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm［J］. Applied Energy， 2022， 321： 119353.
[15]	XIAO B Y， YANG W W， WU J M， et al. Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle［J］. Energy， 2022， 253： 124105.
[16]	WANG Z X， HE H W， PENG J K， et al. A comparative study of deep reinforcement learning based energy management strategy for hybrid electric vehicle［J］. Energy Conversion and Management， 2023， 293： 117442.
[17]	LIANG Z W， RUAN J G， WANG Z P， et al. Soft actor-critic-based EMS design for dual motor battery electric bus［J］. Energy， 2024， 288： 129849.
[18]	ZOU Y， LIU T， LIU D X， et al. Reinforcement learning-based real-time energy management for a hybrid tracked vehicle［J］. Applied Energy， 2016， 171： 372-382.
[19]	DU G D， ZOU Y， ZHANG X D， et al. Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning［J］. Applied Energy， 2019， 251： 113388.
[20]	ZHANG B， ZOU Y， ZHANG X D， et al. Online updating energy management strategy based on deep reinforcement learning with accelerated training for hybrid electric tracked vehicles［J］. IEEE Transactions on Transportation Electrification， 2022， 8（3）： 3289-3306.
[21]	HE H W， NIU Z G， WANG Y， et al. Energy management optimization for connected hybrid electric vehicle using offline reinforcement learning［J］. Journal of Energy Storage， 2023， 72： 108517.
[22]	HUANG R C， HE H W， ZHAO X Y， et al. Longevity-aware energy management for fuel cell hybrid electric bus based on a novel proximal policy optimization deep reinforcement learning framework［J］. Journal of Power Sources， 2023， 561： 232717.
[23]	李佩哲，张文彪. 基于改进经验回放策略的路径规划算法［J/OL］. 控制与决策， 1-8 ［2025-04-05］.
	LI P Z， ZHANG W B. A path planning algorithm based on improved experience replay strategy［J/OL］. Control and Decision， 1-8 ［2025-04-05］.
[24]	周湘淇，付忠广，高玉才. 无监督域适应迁移学习在旋转机械故障诊断中的应用［J］. 振动与冲击， 2024， 43（10）： 106-113.
	ZHOU X Q， FU Z G， GAO Y C. Unsupervised domain adaptation transfer learning for the fault diagnosis in rotating machinery［J］. Journal of Vibration and Shock， 2024， 43（10）： 106-113.
[25]	TAN Y Q， XU J Y， MA J Y， et al. A transferable perception-guided EMS for series hybrid electric unmanned tracked vehicles［J］. Energy， 2024， 306： 132367.

部件	参数	数值
车辆	质量/kg	15 900
	滚动半径/m	0.47
	主减速比	6.20
燃料电池	额定功率/kW	60
动力电池	额定能量/（kW·h）	24.29
动力电池	额定容量/（A·h）	36.80
驱动电机	最大转速/（r·min^-1）	3 000
	最大转矩/（N·m）	2 400
	最大功率/kW	200

超参数	数值
策略网络结构	［4，200，100，50，2］
价值网络结构	［5，200，100，50，1］
策略网络学习率	3×10^-4
价值网络学习率	3×10^-4
训练回合数	100
折扣因子	0.99
优先级因子	0.6

智能体	最高奖励	最低奖励	收敛所需回合	收敛后平均奖励
ESAC	-0.22	-47.50	30	-0.32
SAC	-0.28	-66.98	72	-0.45

EMS	SOC末值	氢耗量/g	燃油经济性/ （kg·100 km^-1）	性能
TU-EMS	0.501 2	817.34	5.61	97.33%
U-EMS	0.502 0	849.59	5.83	93.65%
S-EMS	0.501 8	862.46	5.92	92.23%
DP-EMS	0.500 2	795.82	5.46	100%

EMS	每个控制步的处理器运行时间/ms
TU-EMS	1.32
U-EMS	1.30
S-EMS	1.34
DP-EMS	526.85

Research on Updating Method of Energy Management Strategy for Fuel Cell Bus with Integrated PER and TL

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 25

Related Articles 2

Metrics

Comments

Recommended 0

[1]	Wei Gong,Yafei Wang,Bowen Wang,Zexing Li,Jiaming Sun. Study on Unsupervised Learning Method for Roadside 3D Object Detection Based on Cross Domain Spatiotemporal Feature Matching [J]. Automotive Engineering, 2025, 47(9): 1665-1673.
[2]	Bin Zhang,Yuan Zou,Xudong Zhang,Guodong Du,Wenjing Sun,Wei Sun. Energy Management Strategy Based on TD3-PER for Hybrid Electric Tracked Vehicle [J]. Automotive Engineering, 2022, 44(9): 1400-1409.