Co-MAPPO: Enabling Horizontal Collaborative Task Offloading in Multi-access Edge Computing
-
摘要: 任务卸载作为多接入边缘计算(MEC)的关键技术，通常被研究并应用于终端设备向边缘节点的纵向卸载，由于边缘节点之间存在计算能力差异，导致系统出现负载不均衡。针对该问题，该文引入软件定义网络(SDN)技术构建支持边缘节点横向协作的任务卸载框架，将时延与负载均衡的多目标优化描述为随机整数规划问题，并提出了基于协作式多智能体近端策略优化(MAPPO)的任务卸载策略解决该问题。设计了共享Actor-Critic网络，通过SDN获得全局状态信息，在线更新网络并获得卸载策略。面对共享网络的表达能力不足，提出了观测独热编码与状态增广的方法。仿真结果表明，所提策略在不同任务数据量与网络带宽下相较于对比策略，均能使系统保持低时延与高负载均衡，同时具备动态感知网络拓扑的能力。
-
关键词:
- 多接入边缘计算 /
- 横向任务卸载 /
- 软件定义网络 /
- 多智能体近端策略优化 /
- 负载均衡
Abstract:Objective The rapid development of intelligent IoT has spawned numerous novel applications with low-latency requirements, where vertical task offloading from terminal devices to Edge Node (EN) serves as a critical technology for Multi-access Edge Computing (MEC) to ensure latency guarantees. Typically constrained by limited-service coverage areas, ENs struggle to provide computational resources to terminal devices outside their service regions, resulting in significant load disparities among different nodes. However, existing vertical task offloading approaches primarily optimize terminal-side metrics such as latency and energy consumption, while insufficiently addressing load balancing at the EN layer. Although reinforcement learning algorithms like the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) have been widely adopted for vertical task offloading, they are better suited for competitive-cooperative hybrid environments, whereas horizontal task offloading between ENs constitutes a fully cooperative scenario. Furthermore, conventional vertical task offloading strategies relying on IP networks inherently face challenges of partial observability and global information deficiency due to the distributed architecture of IP infrastructure. To address these limitations, this research introduces Software-Defined Networking (SDN) technology to establish a global view for resolving information asymmetry issues, and proposes a collaborative multi-agent proximal policy optimization (Co-MAPPO)-based task offloading strategy that effectively supports horizontal task offloading, thereby achieving comprehensive EN load balancing. Methods This research first integrates SDN technology into MEC-enabled horizontal task offloading, establishing a multi-hop routing communication architecture and a multi-timeslot computational framework where the SDN controller deploys virtual agents across EN. A load index metric is proposed with its variance serving as the quantitative measure for load balancing, while the multi-objective optimization problem simultaneously addressing task processing latency reduction and system-wide load balancing is formulated as a stochastic integer programming model. Subsequently, a Partially Observable Markov Decision Process (POMDP) is developed for collaborative horizontal task offloading, accompanied by the Co-MAPPO based task offloading strategy. Given the homogeneous state-value functions inherent in cooperative agents, a shared Critic network is implemented to approximate the global state-value function. To address heterogeneous observations and actions across agents, a shared Actor network is developed to approximate the global policy function. Agent identification capability is enhanced through one-hot encoding of observational data and state augmentation techniques, enabling the shared Actor-Critic network to effectively discriminate between heterogeneous agents. The SDN controller aggregates decentralized local observations to reconstruct global system states, computes compound loss functions encompassing both local and global optimization objectives from the shared Actor-Critic network outputs, and executes real-time parameter updates through online learning mechanisms. This iterative optimization process progressively refines the task offloading policy towards achieving balanced computational load distribution across EN. Results and Discussions The Co-MAPPO strategy demonstrates superior convergence performance and outperforms comparative task offloading approaches across varying task data volumes and network environments. (1) Co-MAPPO exhibits advantages over QMix and baseline strategies in convergence speed, policy update stability, and converged outcomes, attributable to its integration of proximal policy optimization that ensures monotonic policy improvement during training ( Fig.3 ). Furthermore, the strategy achieves horizontal collaboration among EN by offloading computationally intensive tasks from resource-constrained ENs to high-capacity ENs (Fig.4 ). (2) As task data volume increases, Co-MAPPO maintains lower average latency compared to other strategies, with the smallest latency escalation rate. At peak data loads, Co-MAPPO reduces latency by 40.3%～84.3% relative to other strategies. Concurrently, its load balancing index remains consistently lower than comparative methods without abrupt increases under heavy workloads (Fig.5 ). (3) Although system latency is observed to increase with reduced transmission bandwidth, Co-MAPPO retains significant superiority. Under minimal bandwidth conditions, it achieves latency reductions of 62.2～88.5% compared to other strategies. The load balancing index exhibits gradual growth under bandwidth constraints while maintaining optimal levels (Fig.6 ). (4) While latency and load balancing demonstrate positive correlation when offloading tasks from low-capacity to high-capacity EN, excessive emphasis on load balancing may inversely increase communication latency due to neglected bandwidth limitations and network topology, thereby establishing a negative correlation under such conditions (Fig.7 ,Fig.8 ).Conclusions The Co-MAPPO strategy utilizes the global network view provided by SDN to dynamically perceive network topology, enabling the MEC system to maintain low latency and achieve load balancing under varying task data volumes and network bandwidth conditions. Additionally, a complex coupling relationship exists between latency and load balancing: (1) Positive Correlation Phase: When tasks are offloaded from computationally weak EN to high-capacity EN, latency reduction is achieved alongside improved load balancing, demonstrating a positive correlation between the two metrics. (2) Negative Correlation Phase: While prioritizing load balancing optimization, Co-MAPPO may inadvertently increase communication latency due to unaddressed transmission bandwidth limitations and network topology constraints, resulting in a negative correlation between latency and load balancing performance. -
1 Co-MAPPO训练伪代码
输入：系统状态$ {\boldsymbol{s}}_{t}=[{\boldsymbol{z}}_{t},\mathbf{\delta }_{t}^{\text{queue-t}},\mathbf{\delta }_{\mathrm{t}}^{\text{queue-c}},{\boldsymbol{b}}_{t},{\boldsymbol{H}}_{t}] $，与EN的观测$ \boldsymbol{o}_{t}^{n}=[z_{t}^{n},\delta _{n,t}^{\text{queue-t}},\mathbf{\delta }_{n,\mathrm{t}}^{\text{queue-c}},{\boldsymbol{b}}_{n,t},{\boldsymbol{H}}_{n,t}] $； 输出：由联合动作$ {\boldsymbol{a}}_{t} $概率分布生成的卸载策略$ {\boldsymbol{x}}_{t} $ (1) 初始化共享Critic网络与共享Actor网络参数，设置Adam优化器的学习率； (2) for episode:1 to $ I $ do (3) for $ t $:1 to $ T $do (4) SDN控制器获取所有EN的观测$ \boldsymbol{o}_{t}^{n} $与系统状态$ {\boldsymbol{s}}_{t} $，根据式(18)对$ \boldsymbol{o}_{t}^{n} $编码得到$ {\hat{\boldsymbol{o}}}_{t}^{n} $，根据式(19)得到$ {{\hat{\boldsymbol{s}}}}_{t} $； (5) for Agent:1 to $ N $do (6) 将$ {\hat{\boldsymbol{o}}}_{t}^{n} $输入Actor网络得到$ \boldsymbol{a}_{t}^{n} $； (7) end for (8) 获得联合动作$ {\boldsymbol{a}}_{t} $，并生成卸载策略$ {\boldsymbol{x}}_{t} $，与环境交互得到奖励$ {r}_{t} $； (9) end for (10) 将本episode获得的状态$ \boldsymbol{s} $与$ \mathbf{{\boldsymbol{s}}^{\prime}} $，观测$ \boldsymbol{o} $与$ \mathbf{{\boldsymbol{o}}^{\prime}} $，动作$ \boldsymbol{a} $与奖励$ \boldsymbol{r} $，组成一条经验$ \boldsymbol{E} $并存储至经验回放池； (11) if buffer-size = $ {C}_{\text{buffer}} $do (12) for epoch:1 to $ {K}_{\text{epoch}} $do (13) 从经验池随机采样$ {N}_{\text{batch}} $样本； (14) for Agent:1 to N do (15) 根据式(26)计算Critic网络局部损失，根据式(27)计算Actor网络局部损失； (16) end for (17) 根据式(31)计算Critic网络全局损失，根据式(32)计算Actor网络全局损失，更新网络参数； (18) end for (19) 清空经验回放池； (20) end if (21) end for 
下载: 导出CSV
表 1 系统环境仿真参数
参数 数值 参数 数值 时隙长度$ \Delta $(s) 0.1 每跳路由平均时延$ \eta $(s) 0.01 时隙个数T 100 CPU处理周期数
$ {\rho }_{n} $(Gigacycles/Mb)0.297 最小任务数据量$ {z}_{\min } $(Mb) 0.5 边缘节点个数N 10 最大任务数据量$ {z}_{\max } $(Mb) 5.5 权重因子$ \beta $ 0.8 最大传输带宽$ {B}_{\max } $(Mbps) 44 量纲平衡因子k 100
下载: 导出CSV
表 2 Co-MAPPO超参数
参数 数值 参数 数值 训练样本个数$ {N}_{\text{batch}} $ 8 平衡方差系数$ \lambda $ 0.95 经验回放池容量$ {C}_{\text{buffer}} $ 32 权重系数$ c $ 0.01 单次训练次数$ {K}_{\text{epoch}} $ 15 clip函数参数$ \varepsilon $ 0.2 回报折扣率$ \gamma $ 0.95 episode个数I 10 000
下载: 导出CSV
-
[1] LETAIEF K B, SHI Yuanming, LU Jianmin, et al. Edge artificial intelligence for 6G: Vision, enabling technologies, and applications[J]. IEEE Journal on Selected Areas in Communications, 2022, 40(1): 5–36. doi: 10.1109/JSAC.2021.3126076. [2] 周晓天, 孙上, 张海霞, 等. 多接入边缘计算赋能的AI质检系统任务实时调度策略[J]. 电子与信息学报, 2024, 46(2): 662–670. doi: 10.11999/JEIT230129.ZHOU Xiaotian, SUN Shang, ZHANG Haixia, et al. Real-time task scheduling for multi-access edge computing-enabled AI quality inspection systems[J]. Journal of Electronics & Information Technology, 2024, 46(2): 662–670. doi: 10.11999/JEIT230129. [3] 张冰雪, 李希胜, 尤佳. 多接入边缘计算网络中动态资源感知与任务卸载方案设计[J]. 电子与信息学报, 2026, 48(5): 2199–2209. doi: 10.11999/JEIT250640.ZHANG Bingxue, LI Xisheng, and YOU Jia. Design of dynamic resource awareness and task offloading schemes in multi-access edge computing networks[J]. Journal of Electronics & Information Technology, 2026, 48(5): 2199–2209. doi: 10.11999/JEIT250640. [4] 杨守义, 韩昊锦, 郝万明, 等. 边缘计算中面向缓存的迁移决策和资源分配[J]. 电子与信息学报, 2024, 46(12): 4391–4398. doi: 10.11999/JEIT240427.YANG Shouyi, HAN Haojin, HAO Wanming, et al. Cache oriented migration decision and resource allocation in edge computing[J]. Journal of Electronics & Information Technology, 2024, 46(12): 4391–4398. doi: 10.11999/JEIT240427. [5] LU Siliang, LU Jingfeng, AN Kang, et al. Edge computing on IoT for machine signal processing and fault diagnosis: A review[J]. IEEE Internet of Things Journal, 2023, 10(13): 11093–11116. doi: 10.1109/JIOT.2023.3239944. [6] CRUZ P, ACHIR N, and VIANA A C. On the edge of the deployment: A survey on multi-access edge computing[J]. ACM Computing Surveys, 2023, 55(5): 99. doi: 10.1145/3529758. [7] DONG Shi, TANG Junxiao, ABBAS K, et al. Task offloading strategies for mobile edge computing: A survey[J]. Computer Networks, 2024, 254: 110791. doi: 10.1016/j.comnet.2024.110791. [8] FERRAG M A, FRIHA O, KANTARCI B, et al. Edge learning for 6G-enabled internet of things: A comprehensive survey of vulnerabilities, datasets, and defenses[J]. IEEE Communications Surveys & Tutorials, 2023, 25(4): 2654–2713. doi: 10.1109/COMST.2023.3317242. [9] BEJARBANEH E Y, DU Haiping, and NAGHDY F. Exploring shared perception and control in cooperative vehicle-intersection systems: A review[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(11): 15247–15272. doi: 10.1109/TITS.2024.3432634. [10] WALIA G K, KUMAR M, and GILL S S. AI-empowered fog/edge resource management for IoT applications: A comprehensive review, research challenges, and future perspectives[J]. IEEE Communications Surveys & Tutorials, 2024, 26(1): 619–669. doi: 10.1109/COMST.2023.3338015. [11] CUI Laizhong, XU Chong, YANG Shu, et al. Joint optimization of energy consumption and latency in mobile edge computing for Internet of Things[J]. IEEE Internet of Things Journal, 2019, 6(3): 4791–4803. doi: 10.1109/JIOT.2018.2869226. [12] WANG Yuchen, HUANG Zishan, WEI Zhongcheng, et al. MADDPG-based offloading strategy for timing-dependent tasks in edge computing[J]. Future Internet, 2024, 16(6): 181. doi: 10.3390/fi16060181. [13] TRAN T X and POMPILI D. Joint task offloading and resource allocation for multi-server mobile-edge computing networks[J]. IEEE Transactions on Vehicular Technology, 2019, 68(1): 856–868. doi: 10.1109/TVT.2018.2881191. [14] HUANG Liang, BI Suzhi, and ZHANG Y J A. Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[J]. IEEE Transactions on Mobile Computing, 2020, 19(11): 2581–2593. doi: 10.1109/TMC.2019.2928811. [15] TANG Ming and WONG V W S. Deep reinforcement learning for task offloading in mobile edge computing systems[J]. IEEE Transactions on Mobile Computing, 2022, 21(6): 1985–1997. doi: 10.1109/TMC.2020.3036871. [16] HE Xiao, PANG Shanchen, GUI Haiyuan, et al. Multi-agent DRL-based large-scale heterogeneous task offloading for dynamic IoT systems[J]. IEEE Transactions on Network Science and Engineering, 2025, 12(2): 982–996. doi: 10.1109/TNSE.2024.3521885. [17] MAITI R, MADHUKUMAR A S, and ERNEST T Z H. MACU: A multiagent cache updating framework for IIoT networks[J]. IEEE Internet of Things Journal, 2025, 12(5): 5219–5232. doi: 10.1109/JIOT.2024.3487913. [18] ABEDI M R, MOKARI N, JAVAN M R, et al. Low complexity and mobility-aware robust radio, storage, computing, and cost management for cellular vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2025, 74(2): 3327–3344. doi: 10.1109/TVT.2024.3480996. [19] DAS D, RANA M K, SARDAR B, et al. A comparative analysis of distributed mobility management schemes for 5G-based intelligent transportation systems[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(2): 2434–2448. doi: 10.1109/TITS.2024.3506662. [20] LV Zhihan and XIU Wenqun. Interaction of edge-cloud computing based on SDN and NFV for next generation IoT[J]. IEEE Internet of Things Journal, 2020, 7(7): 5706–5712. doi: 10.1109/JIOT.2019.2942719. [21] XU Xiaolong, HUANG Qihe, ZHU Haibin, et al. Secure service offloading for internet of vehicles in SDN-enabled mobile edge computing[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3720–3729. doi: 10.1109/TITS.2020.3034197. [22] DAS R K, AHMED N, MAJI A K, et al. Edge controller-assisted SDN architecture for internet of things[J]. IEEE Sensors Journal, 2023, 23(22): 28182–28190. doi: 10.1109/JSEN.2023.3317841. [23] BAKER T, AL AGHBARI Z, KHEDR A M, et al. EDITORS: Energy-aware dynamic task offloading using deep reinforcement transfer learning in SDN-enabled edge nodes[J]. Internet of Things, 2024, 25: 101118. doi: 10.1016/j.iot.2024.101118. [24] TANG Chaogang, ZHU Chunsheng, ZHANG Ning, et al. SDN-assisted mobile edge computing for collaborative computation offloading in industrial internet of things[J]. IEEE Internet of Things Journal, 2022, 9(23): 24253–24263. doi: 10.1109/JIOT.2022.3190281. [25] RASHID T, SAMVELYAN M, DE WITT C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. The Journal of Machine Learning Research, 2020, 21(1): 178. doi: 10.5555/3455716.3455894. [26] LOWE R, WU Yi, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[C]. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 6382–6393. doi: 10.5555/3295222.3295385. [27] JIANG Bingqing, DU Jun, JIANG Chunxiao, et al. Underwater searching and multiround data collection via AUV swarms: An energy-efficient AoI-aware MAPPO approach[J]. IEEE Internet of Things Journal, 2024, 11(7): 12768–12782. doi: 10.1109/JIOT.2023.3336055. [28] 陈盈果, 王斐然, 胡云鹏, 等. 融合大语言模型与强化学习的敏捷卫星任务分配算法设计[J]. 电子与信息学报, 2025, 47(12): 4959–4972. doi: 10.11999/JEIT250991.CHEN Yingguo, WANG Feiran, HU Yunpeng, et al. Automating algorithm design for agile satellite task assignment with large language models and reinforcement learning[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4959–4972. doi: 10.11999/JEIT250991. [29] 张冰雪, 李希胜, 尤佳. 多接入边缘计算网络中动态资源感知与任务卸载方案设计[J]. 电子与信息学报, 2026, 48(5): 2199–2209. doi: 10.11999/JEIT250640.ZHANG Bingxue, LI Xisheng, and YOU Jia. Design of dynamic resource awareness and task offloading schemes in multi-access edge computing networks[J]. Journal of Electronics & Information Technology, 2026, 48(5): 2199–2209. doi: 10.11999/JEIT250640. -
下载:
图(10) / 表(3)
计量
- 文章访问数: 10
- HTML全文浏览量: 1
- PDF下载量: 0
- 被引次数: 0
