Mobile Edge Computing (MEC) has been considered very potential in overcoming the limited computing resource constraint of mobile users (MUs)  where MUs can offload partially or fully their computation load (CL) to the fog servers (FSs) at the network edge, which will be called partial and binary offloading, respectively. Such computation offloading (CO) can result in smaller energy consumption and/or better execution latency [2, 3]. However, CO typically requires transmission of some incurred data from MUs to the fog servers. From the MU’s perspective, CO can reduce the local CL but it also consumes extra energy for the data transmission. Therefore, joint CO decision and resource allocation (RA) becomes a major challenge in MEC systems which has been studied in several recent works [4, 3, 5, 6, 7, 8, 9, 10]. In particular, the papers [4, 3, 5, 6] consider MEC systems in which one FS serves multiple MUs. Moreover,  designs a binary CO scheme to improve energy-efficiency considering radio interference while  proposes fair binary CO algorithms aiming to minimize the maximum weighted energy consumption. The partial CO strategy is considered in [5, 6]. Both papers focus on minimizing the total energy consumption of all MUs with the delay constraint where  investigates the offloading for the frequency division multiple access (FDMA) while  studies the offloading for both OFDMA and TDMA scenarios.
A joint CO and RA design is recently investigated in multiple-FS MEC systems where a group of FSs is available to help the MUs reduce their CLs [8, 9, 10, 11]. In particular,  studies the fair partial CO for the multiple-FS MEC system to minimize the task execution delay. In this work, a task from an MU is first partitioned into multiple subtasks which are offloaded independently to different FSs within their coverage using the corresponding wireless links. In , a binary CO strategy in the multi-fog system is proposed where CL from an MU can be executed at its FS or at one of its neighboring access points. Similarly, the authors in  propose a joint CO and RA framework which maximizes the weighted sum of the reductions in task completion time and energy consumption. These papers do not consider cooperation among the FSs. Considering FSs’ cooperation,  studies the multiple-cloud load balancing problem for the system with limited backhaul capacity where minimization of the maximum CL of all cloud servers is the design objective. However, optimization of the CO decisions is not addressed in this work.
Our current paper studies the joint CO and RA in a cooperative MEC system with multiple FSs where cooperative computation load balancing among different FSs is considered to efficiently serve the offloaded CLs. Specifically, an overloaded FS can offload its CL to neighboring FSs through the capacity limited backhaul links. Our design aims at minimizing the weighted sum of energy consumption considering constraints on the maximum task execution latency and computation resource (CR) at the MUs and FSs as well as the backhaul capacity. The design can be applied to any multi-FS MEC systems with arbitrary backhaul network topologies. The optimization problem is formulated as a non-convex problem which is hard to solve due to the complex coupling among the variables. To overcome this challenge, we first study the relationship among the variables based on which the problem is convexified by using the penalty method. Then, a duality-based algorithm is proposed to solve the problem optimally. For comparison purpose, a greedy algorithm is also discussed. Numerical results are then presented to illustrate the various insights and significant performance gains compared to conventional design without exploiting cooperation among FSs and the greedy algorithm.
Ii System Model and Problem Formulation
Consider a multi-FS system consisting FSs deployed at one-antenna base stations (BS). Each BS is connected with a corresponding co-located FS which provide wireless communication and edge computation services to MUs inside its coverage. Let denote the set of all FSs (or BSs), and denote the set of MUs served by FS/BS . Furthermore, it is assumed that each FS can offload its CL to other FSs in a FS set through the corresponding backhaul links. The helping FS set for each FS can be determined from the underlying backhaul network topology, which is denoted as for FS . For convenience, we also define as the set of FSs which can offload their CLs to FS .
Ii-a Local-Computing and Offloading Model
Assume that MU in has to execute a computation task with -bit input data within the maximum latency of seconds. The partial offloading strategy is considered in this paper. In particular, bits of the incurred data are executed locally at the MU while the remaining bits will be offloaded to its FS for processing where . Let be the CPU clock speed of MU associated with BS , which can be chosen in the range . The local execution energy per CPU cycle can be expressed as  where represents the energy coefficient specified in the CPU model. Denote as the number of CPU cycles required for executing data bit of MU . Then, the energy and time consumed for local execution at MU can be expressed, respectively as
Once MU decides to offload bits to FS , it transmits these data bits to BS over the wireless link. The TDMA transmission strategy is employed in this paper. In cell (corresponding to FS ), an interval of seconds is divided into time slots each of which is then assigned to one corresponding MU in the set to support the computation offloading. Let be the channel gain between MU and BS , which is assumed to be unchanged during the offloading duration; and denote the transmission power of MU . To avoid the strong inter-cell interference, we assume that different frequency bands are allocated to different BSs . Then, the data rate corresponding to the transmission from MU to its BS can be expressed as
where is the bandwidth and is the noise power. Let denote the duration of the time slot assigned to MU , which must satisfy due to the employed TDMA strategy. Hence, the transmission power of MU in BS can be set so that . Thus, can be expressed as where , and . Then, the energy consumed by offloading MU can be expressed as
Ii-B Multi-Fog Cooperation Computing Model
The offloaded CL from MU (corresponding to bits) can be executed at FS or be further offloaded to FSs in the set . Let denote the number of bits, which is a part of bits, offloaded to FS () for processing, we have . Assume that the allocated backhaul rate between FSs and is fixed at , and the CPU clock speed that FS allocates for executing is . Then, the time consumed for further offloading and executing bits at BS can be given, respectively as
Note that stands for the number of bits processed directly at BS ; hence, should be set to be zero. To yield this required outcome, we set as , . For many practical applications, the amount of data describing the computation outcome to be sent back to every MU, is usually much smaller than that due to the task offloading. Therefore, we omit the downlink transmission in this paper. Hence, the total time required for task execution of MU can be written as
Ii-C Problem Formulation
In this paper, we focus on minimizing the weighted sum energy consumption of all MUs and our design can be formulated in the following problem:
where ’s are the weighted coefficients, ’s denotes the set of all variables. Here, the constraints represent the backhaul capacity, computing resource, and different delay constraints where is the maximum clock speed of FS .
Iii Optimal Joint Computation Offloading and Resource Allocation
Iii-a Penalty based Convex Transformation
This section describes how to convexify problem which is non-convex due to the non-linear term in and the non-convex constraint (8f). Toward this end, we first study the relationship between the optimal value of with other variables in the following proposition.
The optimal value of for any values of and can be expressed as
As can be seen, constraint (8f) can provide the lower bound on as . Besides, it can be verified from the local computing energy consumption given in (1) that the objective function of problem is a monotonic increasing function of . Therefore, the optimal value of for any value of and can be expressed in (9). ∎
By replacing and using Proposition 1, the variable set can be reduced to and can be rewritten as which is a convex function with respect to . However, problem is still non-convex due to non-convex term in constraint (8f). To deal with this challenge, we introduce a new variable which satisfies . Using these new variables, problem can be rewritten as
where ’s. A can be seen, the left hand side of constraint (11c) is a convex function of . Thus, problem is convex which can be solved optimally if one omits constraint (11c). Therefore, we employ the Penalty-Function method to solve problem by dealing with the following problem
In particular, the optimal solution of can be obtained by repeatedly solving problem and increasing until constraint (11c) holds. In the next section, we study how to tackle problem and propose an optimal algorithm to solve problem . Problem is convex thanks to its convex-form objective function and constraints. Therefore, it can be solved by employing the standard duality method as presented in the following section.
Iii-B Duality-based Optimal Algorithm
The Lagrangian of problem is expressed in (17) where ’s, ’s are the Lagrangian multipliers associated with the constraints of problem . Then, the dual function can be defined as
and the dual problem can be stated as
Let . For given , the value of can be expressed as
where is the Lambert function .
Due to the space constraint, the proof is given briefly as follows. Applying the KKT conditions, we first take the derivative of with respect to , set it to zero, which results in an equation of . Substituting into this equation yields a Lambert function of . Solving this equation as in , we can obtain as given in (16). ∎
Let where . For given , the following value of satisfy the KKT conditions:
where and is determined by solving the following equation
Due to the space constraint, the proof is given briefly as follows. First, we set
Applying the KKT conditions, we first take the derivative of with respect to all variables in and set the results equal to zeros to form a system of equations based on which the variables in can be stated as a function of as in (17)-(19). Substituting (19) and (17) into (21) yields (3). Then, hence can be expressed as in (17)-(19) after obtaining by solving (3). Since are calculated from the KKT conditions, must satisfy these conditions. ∎
The dual function is a concave function and its sub-gradient at is , and at is where is obtained from the previous section.
Due to proposition 4, the dual variables can be solved using the following iterative updates:
where and are suitable small step-sizes. If , the above sub-gradient based updates are guaranteed to converge to the optimal solution of problem (15). By iteratively updating , we can obtain the optimal solution. The proposed solution approach is summarized in the Algorithm 1 where .
Iv Greedy Algorithm
For a comparison purpose, a greedy algorithm is described in Algorithm 2. This algorithm aims to perform load balancing among FSs. Specifically, we attempt to determine if one FS is overload and there exists a Ready-To-Help (RTH) FS with available computing resource (CR) as follows. First, we optimize the CO and RA at each BS/FS separately, from which the available CR of each FS is calculated as . Then, the FS having the largest available CR is selected as RTH FS . This RTH FS will utilize all available CR to help the FS that has used all CR and its MUs spend the highest energy consumption, named as Asking-For-Help (AFH) FS . Specifically, FS is allowed to offload its CL to FS . The CPU clock speed that FS assigns MUs in cell can be determined as in (26). Then, the offloaded data size can be determined to achieve time balancing as follows:
which yields . Thanks to the load balancing between FSs and as in Step 11 of Algorithm 2, the processing time at the FS for MUs in cell is reduced by . This time reduction can be exploited for increasing the offloading time of all MUs in cell . Hence, the new offloading duration can be defined as . Then, the join CO and RA design for cell can be re-optimized by solving the following problem:
where . Problem is applied for one FS which can be solved by employing the algorithms in  or our proposed algorithm with low complexity. Since the previous values of is a feasible solution of , solving results in a new solution with smaller weighted sum of power consumption at cell . Step 4-Step 13 of Algorithm 2 are repeated until there is no RTH FS or AFH FS.
V Numerical Results
The performance of the proposed algorithms for backhaul limited cooperative MEC system is investigated via numerical studies. We consider a simple 4-cell network where the distance between two nearest BSs is as illustrated in Fig. 1. In each cell, we randomly place 7 MUs (except for Fig. 5) so that the distance from the cell center to every MU is in the range . The channel gains are generated by considering both Rayleigh fading and path loss which is modelled as where is the distance from MU to FS and . The noise power is set equal to . In the simulation, we choose equal to for all MUs, MHz (except for Fig. 3), and Mbits (except for Fig. 6). Each MU needs to execute a task with data size of Mbits () within a duration of ms () (except Fig. 4). The CPU computation capacity for each MU () is randomly selected from the set GHz and the local computing energy per cycle is J/cycle. The number of CPU cycles per data bit is set cycles/bit. The computation capacities of four FSs are chosen as GHz.
First, we examine the convergence of our proposed optimal algorithm by showing the variations of total energy consumption (TEC), the energy of local execution (ELE), and offloading energy (OE) of all MUs over iterations by using Algorithm 1 with in Fig. 2. In this simulation, we consider the cooperation among the FSs under the full mesh backhaul topology as shown in Fig. 1-(a). It can be observed that the TEC converges after iterations which confirms the convergence of Algorithm 1. In addition, the ELE decreases while the TE increases over iterations before becoming saturated.
In Fig. 3, we show the benefit of cooperation among the FSs where the TEC versus the frequency bandwidth () is plotted for three schemes, Algorithm 1 with full cooperation (Full Mesh) and no-cooperation, and Algorithm 2 with full cooperation. As can be observed, the TEC achieved by our proposed optimal algorithm with full cooperation is much lower than those due to the schemes without FSs’ cooperation and the greedy algorithm. In addition, the TEC achieved by all algorithms decrease as increases, which demonstrates the impact of wireless links capacity on the CO design.
Then, we illustrate the variations of TEC achieved by Algorithm 1 with different values of allowable task latency () and application data size () in Fig. 4. As can be seen from this figure, the TEC decreases when increases while the TEC increases when becomes larger. This is because the larger value of gives more freedom for the computation offloading and the larger data size requires higher energy for data transmission and task execution.
The performance of our proposed algorithm under different backhaul topologies is examined in Fig. 5 where the TEC achieved by employing Algorithm 1 versus the number of MUs is plotted in four backhaul network topologies: Full Mesh, Ring, Star-Max (the most powerful FS is the center), and Star-Min (the weakest FS is the center). This figure confirms that the Full-Mesh backhaul network results in the lowest TEC while Star-Max scenario achieves better performance than the Ring scenario, and the Star-Max is the worst. The figure also shows that the TEC achieved by all schemes increases with the number of MUs as expected.
Finally, we demonstrate the impact of backhaul capacity on the number of bits offloaded to FSs with and without cooperation among the FSs in Fig. 6. In this simulation, we set . As can be seen, the cooperation topologies encourage MUs to offload more data to FSs than the no-cooperation one does. In particular, the number of offloaded bits in Full-Mesh scheme is the highest, that number in Ring scheme is slightly smaller than that due to Star-Max scheme while the Star-Min scheme gains the lowest number of offloaded bits among all cooperation schemes. Considering the results in both Figs. 5 and 6, one interestingly confirms that the schemes having the higher number of offloaded bits can save more TEC. In addition, the amount of offloaded data in each scheme tends to increase as the backhaul capacity increases but it becomes saturated when is sufficiently large.
In this paper, we have considered the joint CO and RA design for backhaul limited cooperative MEC systems. This design aims to minimize the weighted sum of energy consumption of all MUs. We have developed the optimal algorithm using the convexification and duality methods and have described the greedy algorithm. Numerical results have confirmed the great performance gains of the proposed design compared to offloading design without cooperation among FSs and the greedy design.
-  E. Portal. (2014) Mobile-edge computing introductory technical white paper. [Online]. Available: https://portal.etsi.org
-  X. Lyu, H. Tian, L. Jiang, A. Vinel, S. Maharjan, S. Gjessing, and Y. Zhang, “Selective offloading in mobile edge computing for the green internet of things,” IEEE Network, vol. 32, no. 1, pp. 54–60, Jan 2018.
T. T. Nguyen, L. Le, and Q. Le-Trung, “Computation offloading in MIMO based mobile edge computing systems under perfect and imperfect CSI estimation,”IEEE Trans. Services Comput., vol. PP, no. 99, pp. 1–1, 2019.
-  C. Wang, F. R. Yu, C. Liang, Q. Chen, and L. Tang, “Joint computation offloading and interference management in wireless cellular networks with mobile edge computing,” IEEE Trans. Veh. Technol., vol. 66, no. 8, pp. 7432–7445, Aug 2017.
-  Y. Wang, M. Sheng, X. Wang, L. Wang, and J. Li, “Mobile-edge computing: Partial computation offloading using dynamic voltage scaling,” IEEE Trans. Commun., vol. 64, no. 10, pp. 4268–4282, Oct 2016.
-  C. You, K. Huang, H. Chae, and B. Kim, “Energy-efficient resource allocation for mobile-edge computation offloading,” IEEE Trans. Wireless Commun., vol. 16, no. 3, pp. 1397–1411, March 2017.
-  L. Mashayekhy, M. M. Nejad, and D. Grosu, “Cloud federations in the sky: Formation game and mechanism,” IEEE Trans. Cloud Comput., vol. 3, no. 1, pp. 14–27, Jan 2015.
-  G. Zhang, F. Shen, Y. Yang, H. Qian, and W. Yao, “Fair task offloading among fog nodes in fog computing networks,” in Proc. IEEE Int. Conf. Commun., May 2018, pp. 1–6.
-  X. He, Y. Chen, and K. K. Chai, “Delay-aware energy efficient computation offloading for energy harvesting enabled fog radio access networks,” in Proc. IEEE Veh. Technol. Conf., June 2018, pp. 1–6.
-  T. X. Tran and D. Pompili, “Joint task offloading and resource allocation for multi-server mobile-edge computing networks,” IEEE Trans. Veh. Technol., vol. 68, no. 1, pp. 856–868, Jan 2019.
-  J. Oueis, E. Calvanese-Strinati, A. De Domenico, and S. Barbarossa, “On the impact of backhaul network on distributed cloud computing,” in Proc. IEEE Wireless Commun. and Networking Conf. IEEE, 2014, pp. 12–17.
-  W. Zhang, Y. Wen, K. Guan, D. Kilper, H. Luo, and D. O. Wu, “Energy-optimal mobile cloud computing under stochastic wireless channel,” IEEE Trans. Wireless Commun., vol. 12, no. 9, pp. 4569–4581, Sep. 2013.
-  C. Kosta, B. Hunt, A. U. Quddus, and R. Tafazolli, “On interference avoidance through inter-cell interference coordination (icic) based on ofdma mobile systems,” IEEE Commun. Surveys Tuts., vol. 15, no. 3, pp. 973–995, Third 2013.
-  S. Boyd and L. Vandenberghe, Convex optimization. Cambridge university press, 2004.
-  R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey, and D. E. Knuth, “On the lambertw function,” Advances in Computational Mathematics, vol. 5, no. 1, pp. 329–359, Dec 1996. [Online]. Available: https://doi.org/10.1007/BF02124750