基于混合深度强化学习的ICV任务卸载与资源分配

doi:10.19562/j.chinasae.qcgc.2025.01.004

Abstract

Abstract:

With the development of Intelligent Connected Vehicle （ICV） technology， ICVs with limited computing resources face the problem of significantly increased computational demand. ICVs can offload tasks to Mobile Edge Computing （MEC） servers via Roadside Units （RSU）. However， the dynamic and complex nature of vehicular networks makes task offloading and resource allocation highly challenging. In this paper， it is proposed to minimize task computing energy consumption by controlling task offloading decision， communication power， and computing resource allocation under environmental and resource constraints. To address the coexistence of discrete and continuous control variables in the problem， a Hybrid Deep Reinforcement Learning （HDRL） algorithm is designed. The algorithm employs the Double Deep Q-Network （DDQN） to generate task offloading decisions and the Deep Deterministic Policy Gradient （DDPG） to determine communication power and MEC resource allocation. Furthermore， an Improved Prioritized Experience Replay （IPER） mechanism is integrated to evaluate and select actions， outputting the optimal strategy. Simulation results show that the method achieves faster and more stable decision convergence than comparative algorithms， minimizes the energy consumption for task computation offloading， and effectively adapts to changes in the number of ICVs and task sizes， demonstrating high real-time performance and excellent environmental adaptability.

Key words: mobile edge computing, deep reinforcement learning, task offloading, resource allocation, priority experience replay

Jiahui Liu,Yuan Zou,Wei Sun,Yihao Meng,Xiaoran Lu,Yuanyuan Li. ICV Task Offloading and Resource Allocation Based on Hybrid Deep Reinforcement Learning[J].Automotive Engineering, 2025, 47(1): 35-43.

Figures/Tables 8

输入：系统环境参数HDRL 算法参数

输出：HDRL网络参数

1）初始化DDQN网络参数 $ω$ 和 $ω'$ ， DDPG网络参数 $θ$ 、 $θ'$ 、 $φ$ 、 $φ'$ ，经验回放池 $Φ$ ；

2） for episode = 1 to EP_maxdo

3）初始化环境、奖励值R=0，获取环境初始状态s₀；

4） fort = 1 to Tdo

5）获取系统状态s_t；

6）利用DDQN选择动作 $x i j (t)$ ；

7）利用DDPG选择动作 $p i j (t)$ 和 $β i j (t)$ ；

8）执行动作 $a t = x i j (t), p i j (t), β i j (t)$ ，获得奖励值 $r t$ 和下一系统状态 $s t + 1$ ；

9）计算系统累积回报R = R + r_t；

10）将四元组 $(s t, a t, r t, s t + 1)$ 存入经验回放池 $Φ$ ；

11）利用IPER机制从经验回放池 $Φ$ 中采样；

12）根据式（18）和式（19）计算得到DDQN最大Q值对应的动作和目标值；

13）根据式（20）最小化损失函数，并根据式（21）更新DDQN网络参数；

14）根据式（22）最小化DDPG的损失函数；

15）根据式（23）更新Actor训练网络参数和 Critic训练网络参数；

16）根据式（24）更新Actor目标网络参数和Critic目标网络参数；

17） end for

18） end for

符号	含义	单位	数值
$d i (t)$	任务数据量大小	Mb	［2，7］
$c i (t)$	任务计算复杂度	Mcycles/Mb	［1 000，1 500］
$τ i (t)$	任务执行允许最大时延	s	3
$f i v e h (t)$	ICV本地可用计算能力	MHz	［750，1 500］
$f j M E C (t)$	MEC计算能力	GHz	［7，10］
B	通信带宽	MHz	20
$α 0$	单位参考距离下的信道增益	dB	-50
$σ 2$	高斯白噪声功率	dBm	-60
P_loss	传输损耗	dB	20
P_max	ICV和MEC配备的RSU 间的最大通信功率	W	1
$κ$	能量系数		10^-26

References 24

1	WAN S， GU R， UMER T， et al. Toward offloading internet of vehicles applications in 5G networks［J］. IEEE Transactions on Intelligent Transportation Systems， 2020， 22（7）： 4151-4159.
2	MUNAWAR S， ALI Z， WAQAS M， et al. Cooperative computational offloading in mobile edge computing for vehicles： a model-based DNN approach［J］. IEEE Transactions on Vehicular Technology， 2022， 72（3）： 3376-3391.
3	TANG F， MAO B， KATO N， et al. Comprehensive survey on machine learning in vehicular network： technology， applications and challenges［J］. IEEE Communications Surveys & Tutorials， 2021， 23（3）： 2027-2057.
4	CHEN J， XING H， XIAO Z， et al. A DRL agent for jointly optimizing computation offloading and resource allocation in MEC［J］. IEEE Internet of Things Journal， 2021， 8（24）： 17508-17524.
5	赵海涛，张唐伟，陈跃，等.基于DQN的车载边缘网络任务分发卸载算法［J］.通信学报，2020，41（10）：172-178.
	ZHAO H T， ZHANG T W， CHEN Y， et al. Task distribution offloading algorithm of vehicle edge network based on DQN ［J］. Journal on Communications， 2020，41（10）：172-178.
6	HUANG L， BI S， ZHANG Y J A. Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks［J］. IEEE Transactions on Mobile Computing， 2019， 19（11）： 2581-2593.
7	ZHOU H， JIANG K， LIU X， et al. Deep reinforcement learning for energy-efficient computation offloading in mobile-edge computing［J］. IEEE Internet of Things Journal， 2021， 9（2）： 1517-1530.
8	卢海峰，顾春华，罗飞，等.基于深度强化学习的移动边缘计算任务卸载研究［J］.计算机研究与发展，2020，57（7）：1539-1554.
	LU H F， GU C H， GU F， et al. Research on task offloading based on deep reinforcement learning in mobile edge computing［J］. Journal of Computer Research and Development， 2020，57（7）：1539-1554.
9	LIU Y， YU H， XIE S， et al. Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks［J］. IEEE Transactions on Vehicular Technology， 2019， 68（11）： 11158-11168.
10	KE H， WANG J， DENG L， et al. Deep reinforcement learning-based adaptive computation offloading for MEC in heterogeneous vehicular networks［J］. IEEE Transactions on Vehicular Technology， 2020， 69（7）： 7916-7929.
11	LIN J， HUANG S， ZHANG H， et al. A deep-reinforcement-learning-based computation offloading with mobile vehicles in vehicular edge computing［J］. IEEE Internet of Things Journal， 2023， 10（17）： 15501-15514.
12	QIU B， WANG Y， XIAO H， et al. Deep reinforcement learning-based adaptive computation offloading and power allocation in vehicular edge computing networks［J］. IEEE Transactions on Intelligent Transportation Systems， 2024.
13	何杰，马强.基于深度强化学习的C-V2X任务卸载研究［J/OL］.计算机工程， 1-11［2024-07-20］. https：//doi.org/10.19678/j.issn.1000-3428.0068425.
	HE J， MA Q. Research on C-V2X task offloading based on deep reinforcement learning［J］. Computer Engineering， 1-11［2024-07-20］. https：//doi.org/10.19678/j.issn.1000-3428.0068425.
14	YANG H， WEI Z， FENG Z， et al. Intelligent computation offloading for MEC-based cooperative vehicle infrastructure system： a deep reinforcement learning approach［J］. IEEE Transactions on Vehicular Technology， 2022， 71（7）： 7665-7679.
15	沈乐. 基于DQN-DDPG的空地协作边缘计算任务卸载与资源分配研究［J/OL］.软件导刊： 1-8［2024-05-09］. http：//kns.cnki.net/kcms/detail/42.1671.TP.20240130.1638.016.html.
	SHEN L. Task offloading and resource allocation based on DQN-DDPG for aerial-ground cooperative mobile edge computing［J/OL］. Software Guide： 1-8［2024-05-09］. http：//kns.cnki.net/kcms/detail/42.1671.TP.20240130.1638.016.html.
16	LIN N， TANG H， ZHAO L， et al. A PDDQNLP algorithm for energy efficient computation offloading in UAV-assisted MEC［J］. IEEE Transactions on Wireless Communications， 2023， 22（12）： 8876-8890.
17	MEI H， YANG K， LIU Q， et al. 3D-trajectory and phase-shift design for RIS-assisted UAV systems using deep reinforcement learning［J］. IEEE Transactions on Vehicular Technology， 2022， 71（3）： 3020-3029.
18	SEID A M， BOATENG G O， ANOKYE S， et al. Collaborative computation offloading and resource allocation in multi-UAV-assisted IoT networks： a deep reinforcement learning approach［J］. IEEE Internet of Things Journal， 2021， 8（15）： 12203-12218.
19	SHI H， TIAN Y， LI H， et al. Task offloading and trajectory scheduling for UAV-enabled MEC networks： an MADRL algorithm with prioritized experience replay［J］. Ad Hoc Networks， 2024， 154： 103371.
20	GUO Y， MA D， SHE H， et al. Deep deterministic policy gradient-based intelligent task offloading for vehicular computing with priority experience playback［J］. IEEE Transactions on Vehicular Technology， 2024. doi： 10.1109/TVT.2024.3378919.
21	HE X， LU H， DU M， et al. Qoe-based task offloading with deep reinforcement learning in edge-enabled internet of vehicles［J］. IEEE Transactions on Intelligent Transportation Systems， 2020， 22（4）： 2252-2261.
22	刘国志，代飞，莫启，等.车辆边缘计算环境下基于深度强化学习的服务卸载方法［J］.计算机集成制造系统，2022， 28（10）： 3304-3315.
	LIU G Z， DAI F， MO Q， et al. Service offloading method with deep reinforcement learning in edge computing empowered Internet of vehicles［J］. Computer Integrated Manufacturing Systems，2022， 28（10）： 3304-3315.
23	FENG J， YU F R， PEI Q， et al. Cooperative computation offloading and resource allocation for blockchain-enabled mobile-edge computing： a deep reinforcement learning approach［J］. IEEE Internet of Things Journal， 2019， 7（7）： 6214-6228.
24	喻鹏，张俊也，李文璟，等. 移动边缘网络中基于双深度Q学习的高能效资源分配方法［J］. 通信学报， 2020， 41（12）： 148-161.
	YU P， ZHANG J Y， LI W J， et al. Energy-efficient resource allocation method in mobile edge network based on double deep Q-learning［J］. Journal on Communications， 2020， 41（12）： 148-161.

算法	计算时间/s
HDRL	3.29×10^-3
DDQN+DDPG	4.17×10^-3
DQN+DDPG	3.06×10^-3
GA	24.04

[1]	Fuxing Yao,Chao Sun,Yungang Lan,Bing Lu,Bo Wang,Haiyang Yu. A Lane Change Decision Method for Intelligent Connected Vehicles Based on Mixture of Expert Model [J]. Automotive Engineering, 2024, 46(5): 882-892.
[2]	Xiaohu Geng,Yao Fu,Jie Wang,Yulong Lei,Weidong Liu,Yuhai Wang,Ke Liu. Predictive Cruise Control for Commercial Vehicles Considering Different Time Domains [J]. Automotive Engineering, 2024, 46(11): 2046-2058.
[3]	Jie Li,Xiaodong Wu,Min Xu,Yonggang Liu. Reinforcement Learning Based Multi-objective Eco-driving Strategy in Urban Scenarios [J]. Automotive Engineering, 2023, 45(10): 1791-1802.
[4]	Duanfeng Chu,Zhuoer Wang,Yujie Qiu,Liping Lu,Zhihao Xu,Xinyu Ji. Research on Transmission Strategy for Real⁃time Video Stream Under LTE⁃V2X [J]. Automotive Engineering, 2021, 43(6): 815-824.
[5]	Hao Zhang,Qinhao Fan,Wei Wang,Jin Huang,Zhi Wang. Reinforcement Learning Based Energy Management Strategy for Hybrid Electric Vehicles Using Multi⁃mode Combustion [J]. Automotive Engineering, 2021, 43(5): 683-691.
[6]	Jinghua Guo,Wenchang Li,Yugong Luo,Tao Chen,Keqiang Li. Driver Car⁃Following Model Based on Deep Reinforcement Learning [J]. Automotive Engineering, 2021, 43(4): 571-579.
[7]	Zhenhai Gao,Xiangtong Yan,Fei Gao,Tianjun Sun. A Driver-like Decision-making Method for Longitudinal Autonomous Driving Based on DDPG [J]. Automotive Engineering, 2021, 43(12): 1737-1744.
[8]	Zhou Nan, Chen Gang. Gearshifting Strategy for Robot-driven Vehicles Based on Deep Reinforcement Learning [J]. Automotive Engineering, 2020, 42(11): 1473-1481.

ICV Task Offloading and Resource Allocation Based on Hybrid Deep Reinforcement Learning

RichHTML

PDF (PC)

Like

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 24

Related Articles 8

Metrics

Comments

Recommended 0