Cloud computing has emerged originally as a business model ; therefore, public Cloud services are provided mainly to meet the technological and economic requirements from business enterprises. However, a previous study shows that Cloud computing is also widely accepted as a potential and encouraging paradigm to solve scientific problems , for the same benefits of on-demand resource provisioning and cost effectiveness. In fact, the existing public Cloud services can be improved for scientific computing through appropriate optimizations [6, 22]. On the other hand, once commercial Cloud vendors pay more attention to academic requirements, they can make the current Cloud more scientific-oriented by slightly changing their infrastructures .
Driven by the diversity of requirements , Google launched its infrastructure service, namely Google Compute Engine (GCE), and recently made it available to the public . Being “designed to run high-performance, computationally intensive virtual compute clusters” , GCE seems to be a promising public Cloud service that can satisfy the requirement of scientific computing. Unfortunately, to the best of our knowledge, no formal GCE evaluation work has been done to date. Although a set of relevant evaluation studies can be found in technical websites, most of them lack comprehensive experiments to reveal GCE’s performance, not to mention summarizing GCE’s characteristics for scientific computing. To help understand GCE for dealing with scientific issues, we decided to systematically evaluate GCE following the Cloud Evaluation Experiment Methodology (CEEM) .
When it comes to evaluating public Cloud services for scientific computing, the efforts can be distinguished between two types according to their evaluation strategies. The first type is to reveal fundamental performance of Cloud services by using traditional and/or scientific benchmarks, which happens usually during the early stage of investigating new Cloud services. For example, data durability, availability and access performance of Amazon S3 were particularly analyzed for supporting science grids ; and the NAS Parallel Benchmarks (NPB) was used to verify Amazon EC2 for high performance computing [1, 26]. The second type is to investigate sophisticated Cloud applications as real scientific cases, which is normally based on the aforementioned early-stage evaluation. For example, satellite image processing was studied on Windows Azure ; and a metabolic flux analysis was implemented by employing multiple types of Amazon services .
Given the recently available GCE, this paper reports an early-stage performance evaluation. By employing four popular benchmarks, our study exhibits the fundamental performance of four GCE types, and also compares them with nine Amazon EC2 types. Based on the experimental results and analyses, we further show the potential advantages of, and possible threats to applying GCE to scientific computing. For example, GCE would be particularly suitable for applications that require frequent disk operations, while it may not well support single VM-based parallel computing. Following the same evaluation methodology, even different evaluators would be able to replicate and/or supplement this fundamental evaluation of GCE. Moreover, according to the outcome of this study, researchers and engineers can establish suitable GCE environments to carry out sophisticated and scientific case studies. Overall, the contribution of this work is twofold. Firstly, to our best knowledge, this is the first study that systematically reveals GCE’s performance for scientific computing. Secondly, this evaluation practice can be viewed as a case study for validating the methodology CEEM.
The remainder of this paper is organized as follows. Section II summarizes the existing practices related to GCE evaluation. Section III specifies our pre-experimental evaluation activities following the CEEM. The experimental results and analyses around GCE properties including communication, memory, storage, and computation are respectively reported in Section IV. Conclusions and some future work are discussed in Section V.
Ii Related Work
As revealed in , the requirements in the Cloud market are diverse; Infrastructure as a Service (IaaS, e.g., Amazon EC2) and Platform as a Service (PaaS, e.g., Google AppEngine) would then serve different types of requirements, and they cannot be replaced with each other. This could be the motive for Google to finally launch its infrastructure service GCE . Although GCE has been widely viewed as a strong competitor against the other IaaS services, especially against Amazon EC2, not much evaluation work can be found for understanding GCE’s performance. The very early GCE evaluation studies tended to be qualitative discussions around service level agreement (SLA) [4, 27]. More quantitative evaluations appeared in the latter GCE studies. For example, Zencoder compared GCE’s n1-standard-8-d with two Amazon cluster instance types for video transcoding . As argued in , however, this work did not reveal suitable EC2 types for “apple-to-apple” comparison to GCE. On the contrary, it is able to identify comparable EC2 types for GCE n1-standard-4 in a serial of evaluation experiments [19, 20]. Nevertheless, as suggested by the author himself, those ad hoc and short-time experiments could result in biases and misleading conclusions. A various-type virtual machine (VM) evaluation of GCE and EC2 was given in . However, the non-standard benchmarks and unclear evaluation procedure make this study hard for experimental replication and comparison. Given the lack of a systematic GCE evaluation, we use this paper to report our empirical study to help understand the fundamental performance of GCE, particularly for scientific computing.
Iii Evaluation Methodology
To achieve convincing evaluation results, we followed the ten-step CEEM  to evaluate the selected GCE and EC2 instances (cf. VM Type in III-C), as illustrated in Fig. I. By providing systematic guidelines together with evaluation experiences, CEEM is supposed to help reduce human bias and facilitate implementations of Cloud services evaluation. Here we briefly introduce a set of pre-experimental evaluation activities instructed by CEEM.
Iii-a Requirement Recognition and Service Feature Identification
As mentioned previously, this work is to investigate the fundamental performance of GCE for scientific computing. Since the GCE service is offered as virtual machine (VM) instances, four main characteristics (network I/O, memory, storage, and processing capabilities) can be used to distinguish between different instance types . By using the taxonomy of Cloud services evaluation , we formally identify the requirement and service features to be evaluated as: Given particular benchmarks, how capable are GCE instances in terms of the following properties? Note that a service feature is defined as a combination of a service property and its capability .
Iii-B Metrics/Benchmarks Listing and Selection
Available metrics and benchmarks for Cloud services evaluation can be conveniently explored in the existing metric catalogue . According to the past evaluation experiences, we decided to select relatively lightweight and popular benchmarks recommended in , as listed in Table I. In detail, Iperf is able to deliver more precise results by consuming less system resources , which is usually employed together with Ping for communication evaluation; the sequential disk operations simulated by Bonnie++ are more typical to scientific computing ; STREAM is the de facto memory evaluation benchmark included in the HPC Challenge Benchmark (HPCC) suite ; while the NPB suite (including eight benchmarks: BT, CG, EP, FT, IS, LU, MG, and SP) has been widely adopted in the early-stage evaluation of Cloud services for high performance computing [1, 10, 26]. In particular, considering LU is the most process-flexible pseudo application benchmark in the suite , we chose the benchmarking result of LU to calculate the computation performance/price ratio for different VM types.
|Service Property||Capability Metric||Benchmark||Version|
|Computation||Performance/Price Ratio||LU in NPB-MPI||3.3|
Iii-C Experimental Factor Listing and Selection
Similarly, candidate factors that may influence the measured performance of GCE can be identified within an experimental factor framework . In general, we considered at least two input factors and one output factor in the evaluation of every service property, as specified below.
|VM Name||Virtual CPUs||GCEUs||Memory||Scratch Disk|
|Image||SCSI-enabled GCEL 12.04 LTS (gcel-12-04-v20130104)|
VM Type: At the time of writing, Google supplies 10 types of VM instances (with local disks) categorized into three groups . Given the limitation of quota (8 CPUs) , it is impossible to evaluate all the VM types during the same experimental time window. Therefore, we chose four types (the first four in Table II) as the variable values of VM Type to cover the three instance groups for memory, storage and computation evaluation. With regard to the communication evaluation, we only chose the type n1-standard-1 for the consistent environment and the lowest price. Note that, for the purpose of comparison, we also evaluated nine 64bit-Linux-image EC2 types located in Amazon’s North Virginia data center (us-east-1b zone). Due to the limit of space, we directly report the EC2-related experimental results without listing the individual EC2 types here.
Duration: For each evaluation experiment, we decided to take a whole-day observation including one-hour warming up. In other words, we assign the value of Duration as 24 hours.
In particular, we supplemented three more input factors to communication evaluation.
(GCE) Geographical Location: Currently GCE service has five available zones within three geographical locations (i.e. us-central1, us-central2 and europe-west1). Due to the quota limitation again, we only selected three *-a zones as the VM-side locations.
(Client) Geographical Location: We varied the client-side locations by using a local machine in our NICTA Canberra Lab and an EC2 micro instance in Amazon’s us-east-1b zone.
Communication Scope: Given the two communication patterns, intra-Cloud and wide-area , we decided to respectively observe the data transferring performance between GCE instances and between a GCE instance and a client. Note that, in all the communication experiments, we only visited the external IP addresses of GCE instances to force their network address translation (NAT). As such, the result of the intra-Cloud evaluation would be more comparable to the wide-area scenario.
When it comes to computation evaluation, we were further concerned with Workload Size and Process Number as two experimental factors.
Process Number: To decide the number of processes for experimental design, we were concerned with two sides. From the side of resource, as specified in , the selected GCE instance types have either one or two virtual CPU cores. From the side of workload, although six benchmarks in the NPB-MPI suite run on a power-of-2 number of processes (1, 2, 4, …), BT and SP run on a square number of processes (1, 4, 9, …) 
. To satisfy the process variance of all the benchmarks, we naturally distinguished between requesting one and four processes for running BT and SP, and requesting one, two and four processes for running the other NPB-MPI benchmarks.
Iii-D Experimental Design
By using Design of Experiment (DOE) techniques , the evaluation experiments can be prepared based on the pre-identified factors. Recall that Duration has been constrained to be a single value (i.e. 24 hours), a simple and general design is to run each benchmark on different VM Type instances independently for a whole day plus one hour. When it comes to a combination of input factors with various values, more sophisticated experimental design could be required, as specified below.
In the case of communication evaluation that combines Geographical Location with Communication Scope, the intra-Cloud pattern has six paths (, e.g., “GCE-central1-a to GCE-west1-a” and “GCE-west1-a to GCE-central1-a” are treated as the same path) among three server-side locations, and the wide-area pattern also has six paths (=) between three server-side and two client-side locations. The design result is directly listed in Table III, where an X indicates a communication path that needs to be evaluated.
As for the computation evaluation that combines Workload Size and Process Number, we employed the Full-factorial Design technique  to statistically investigate those factors’ influences on the performance of GCE instances. This design technique adjusts one factor at a time. Also recall that there are eight individual benchmarks with two patterns of process variance in the NPB-MPI suite; a single round of experiment on a VM then consists of 44 (=) different benchmarking trials.
Iv Evaluation Results and Analyses
Iv-a Communication Evaluation Result and Analysis
To save space, we show the 12-path communication evaluation results together in Fig. 2. In general, the data throughputs of communication within US GCE data centers are approximately 2.5 times higher than that between US and Europe, while the lowest one is around 107Mb/s between GCE-central2 and GCE-west1. Recall that NAT is enforced by visiting only external IP addresses in this communication evaluation. If using internal IP addresses, according to our ad hoc experiments, we may achieve nearly double of the communication performance inside or between GCE data centers. Such a level of communication data throughput implies that GCE employs the 1000Mb/s Ethernet.
In particular, the communication performance from NICTA Canberra Lab (also 1000Mb/s Ethernet) to GCE data centers are much worse compared to the measurement from Amazon EC2. This could be caused of much longer routing traces over the connection from our side. Similar to the communication between GCE’s US and European data centers, there is also a significant performance decrease when visiting GCE-west1 from the local machine. However, we did not observe this clear trend when measuring from the EC2 instance. Surprisingly, the communication between EC2-us-east and GCE-west1 even has shorter round-trip latency than that between GCE-centrals and GCE-west1. Another surprising observation is that the communication data throughput between GCE-central1 and GCE-central2 is averagely higher than inside the GCE data centers, although it also shows the most variation.
Iv-B Memory Evaluation Result and Analysis
As mentioned previously, we also evaluated nine EC2 instances from four type families for comparison to the GCE instances. Corresponding to the single-CPU GCE type (i.e. n1-standard-1-d), we only show the single-thread memory performance of different VM types. As illustrated in Fig. 3, the GCE instances displayed stronger memory performance against most of the evaluated EC2 instances. Only Amazon’s second generation (M3) standard family can relatively come up to the four GCE types, as specified in Table IV. This observation is also consistent with the study , although the author evaluated a different GCE type (i.e. n1-standard-4) from this work. In addition, interestingly, EC2 m3.2xlarge seems to be one of the most memory-stable VM types, while EC2 m3.xlarge exposed the most variability in terms of memory data throughput.
Furthermore, it seems that Google adopts different memory virtualization strategy from Amazon. Fig. 3 shows that Amazon varies memory capabilities for different families of VM types. Even in the same type family, Amazon may have deliberately decreased the memory data throughput for higher-SLA VM instances (e.g., EC2 m1.xlarge in the first generation (M1) standard family). On the contrary, this work along with the related study  show that Google may offer relatively consistent memory performance for all the GCE instance types. As such, only take memory into account, Google supplies an easier and clearer condition for VM type selection. Moreover, since even the cheapest GCE instance (n1-standard-1-d in this case) shows high memory performance, GCE service may be considered as an economic candidate for memory-intensive computing.
|(Std. Dev.)||(Std. Dev.)||(Std. Dev.)||(Std. Dev.)|
Iv-C Storage Evaluation Result and Analysis
It has been identified that sequential disk operations are more typical to scientific computing . To save space, here we only show the sequential storage performance of the evaluated VM types. In particular, we did not use Bonnie++ to benchmark Amazon’s second generation (M3) standard instances. The M3 instances employ persistent disks of Elastic Block Store (EBS) only. Since we did not consider GCE’s persistent storage in this study, EBS evaluation is then out of the scope of this paper.
Firstly, we assessed the storage transaction speed of each instance with sequential write/read per character. When it comes to character write/read, in fact, Bonnie++ only measures the amount of data performed per second. Considering that each byte of data incurs a transaction in this case, we can directly translate Bonnie++’s result into storage transaction speed, as shown in Fig. (a)a. Among the evaluated VM types, it is clear that GCE has overwhelming advantage over EC2 on writing/reading small size of data, although there is considerable variability in reading on GCE instances. Given the approximately 3- to 4.5-times faster storage transaction speed, GCE could be particularly suitable for applications that need frequent disk access.
Secondly, we assessed the storage data throughput of each instance with sequential write/read per block. Interestingly, the benchmarking result as illustrated in Fig. (b)b shows opposite performance trends in block writing and reading: GCE instances seem better at writing in general, while EC2 instances generally win at reading. Moreover, unlike EC2, individual GCE instance’s (except for n1-standard-1-d’s) reading and writing data throughputs are nearly the same. A possible explanation for this is that Amazon has implemented storage cache for EC2, while GCE seems not to be the case.
Similar to accessing small size of data, block writing/reading on GCE instances also displays considerable variance. To better observe the storage variability of different VM instances, we plotted the benchmarking results against trial sequence of the whole day. Note that Bonnie++ generates test files that are at least twice the memory size of the evaluated machine to invalidate the memory’s interference, while bigger test files inevitably take longer time in a trial. Therefore, the number of trials during a whole day would be different for different memory-size VM types. Surprisingly, all the four GCE types have regular patterns of performance jitter in block writing, rewriting and reading. This paper only reports their block reading patterns that are the clearest, as illustrated in Fig. 5. As we know, the hard disk surface is divided into a set of concentrically circular tracks. Given the same rotational speed of a hard disk drive (HDD), the outer tracks would have higher data throughput than the inner ones. As such, the regular patterns may indicate that the HDD heads sequentially shuttle between outer and inner tracks when consecutively writing/reading block data on GCE instances.
For comparison, we additionally show the plots of two EC2 instance types. In detail, the m1.large instance had the most variable block data throughput among those EC2 VMs (cf. Fig. (e)e), while some tiny and regular jitters in block reading also appeared on the m2.2xlarge instance (cf. Fig. (f)f). However, contrasted with GCE, there is no clearly fluctuating pattern on EC2 instances. This phenomenon may confirm the aforementioned hypothesis that Amazon have employed storage cache to iron the block writing/reading fluctuations. Overall, this study verifies that EC2 supplies more stable storage service than GCE.
Iv-D Computation Evaluation Result and Analysis
Given the literature investigation , NPB could be one of the most well-known and widely-accepted benchmark suites for early-stage scientific computing evaluation. In fact, the NPB suite can cover various Cloud service properties in addition to computation, for example, BT is a disk I/O-intensive benchmark while MG is communication-intensive. In this study, however, we use NPB to emphasize computation only. In other words, the eight NPB benchmarks are employed together to reflect the overall computation performance of a particular VM type.
After finishing the experiments following the previous design, we found that the GCE n1-highcpu-2-d instance failed in running FT with workload Class B. This could be due to its relatively small size of memory. Therefore, we only show the benchmarking result with workload Class A, as shown in Fig. 6. As claimed by Google, since individual virtual cores of different GCE VM types have the same computation power (i.e. 2.75 GCEUs) , the single-process performance of the evaluated four GCE instances did not reveal much difference. Nevertheless, when increasing the benchmark processes, only the high memory GCE type gave notable improvement of computation performance, which is surprising. With virtually dual-core processors, except for n1-standard-1-d, the other two GCE types should have also exhibited significant scalability from one process to two processes.
Unlike the phenomenon on GCE, the expected rule of scalability can be clearly observed on EC2 instances. Similarly, here we only specify the experimental results of two EC2 types: m1.large and m3.xlarge. Among the evaluated EC2 instances, m1.large is the lowest-SLA EC2 type that is comparable to GCE at two-process performance (cf. Fig. (e)e), while m3.xlarge is the lowest-SLA EC2 type for comparison to GCE at single-process performance (cf. Fig. (f)f). As can be seen, multi-core EC2 instances can nearly double their computation performance when switching the benchmarks’ process number from one to two. As a result, even EC2 m1.large with relatively poor single-process performance can still come up with most GCE types when running multi-process benchmarks.
To straight portray the original scalability  of different GCE and EC2 instances, we employed the typical scalability metric Speedup Over a Baseline , as shown in Fig. 7. In particular, we set the baseline as the single-process performance of EC2 m1.large with the LU benchmark (recall that LU is the only pseudo application benchmark that runs on a power-of-2 number of processes ). It is then clear that multi-core EC2 instances are generally more originally-scalable than GCE instances. Moreover, although the high memory GCE type appeared to have better original scalability, its performance would become significantly variable under multi-process circumstances (cf. Fig. (c)c). These findings reveal that GCE may not be ready yet for single VM-based parallel computing.
Given the current prices of the evaluated GCE and EC2 types, we further calculated their performance/price ratio, as listed in Table V. The performance here refers to the average computation transaction speed of running LU at two processes. Interestingly, even with poor original scalability, GCE seems to have generally economic advantage over Amazon EC2. Only EC2 c1.medium beats all the evaluated GCE types in this case. This finding suggests that GCE is a relatively economic option in terms of computation, and it could be even more economic if improving GCE to be as originally scalable as EC2.
|VM Type||Price||LU Computation||Performance/Price|
|EC2 m1.medium||$0.120/hour||1337.34 Mop/s||401202.71 Mop/cent|
|EC2 m1.large||$0.240/hour||2777.3 Mop/s||416595 Mop/cent|
|EC2 m1.xlarge||$0.480/hour||2725.9 Mop/s||204442.61 Mop/cent|
|EC2 m2.xlarge||$0.410/hour||3639.32 Mop/s||319549.96 Mop/cent|
|EC2 m2.2xlarge||$0.820/hour||3629.46 Mop/s||159342.33 Mop/cent|
|EC2 m3.xlarge||$0.500/hour||4210.94 Mop/s||303188.03 Mop/cent|
|EC2 m3.2xlarge||$1.000/hour||4340.91 Mop/s||156272.65 Mop/cent|
|EC2 c1.medium||$0.145/hour||2690.39 Mop/s||667960.02 Mop/cent|
|EC2 c1.xlarge||$0.580/hour||2773.06 Mop/s||172120.72 Mop/cent|
|GCE standard-1-d||$0.132/hour||2343.21 Mop/s||639058.31 Mop/cent|
|GCE standard-2-d||$0.265/hour||2833.51 Mop/s||384930.02 Mop/cent|
|GCE highmem-2-d||$0.305/hour||3787.07 Mop/s||446997.85 Mop/cent|
|GCE highcpu-2-d||$0.163/hour||2836.06 Mop/s||626368.94 Mop/cent|
V Conclusions and Future Work
Cloud computing with public Cloud services has been widely regarded as a potential and encouraging paradigm for scientific computing. Since the recently available GCE is supposed to meet high-performance and computationally intensive workloads , we performed an early-stage evaluation of GCE to verify its fundamentally scientific capabilities. By contrasting with a set of Amazon EC2 types, the experimental results and analyses exhibit both possible advantages of, and threats to employing GCE, as listed below.
The potential advantages of using GCE:
Relatively high memory data throughput. Different GCE types seem to have consistent type of virtual memory.
Relatively fast storage transaction speed.
Relatively fast computation transaction speed with single process.
Relatively computationally economic, given generally high computation performance/price ratio.
The possible threats to using GCE:
Relatively low communication data throughput between US and European data centers. Distributed computing crossing both data centers may need to be well balanced.
Considerable variability of storage performance. GCE seems to have a lack of dedicated storage cache.
Considerable variability of computation transaction speed on high memory VM type.
Relatively poor (original) scalability when switching process numbers on individual VMs. GCE’s hyper-thread-based virtual core seems not a suitable mechanism for parallel computing.
Based on the outcomes of this study, our future work will be unfolded along two directions. The first direction is to supplement the fundamental evaluation of GCE. By applying more resources, we will try to evaluate the other VM types to investigate GCE’s vertical scalability, and evaluate VM clusters to investigate GCE’s horizontal scalability. The second direction is to deploy and evaluate scientific applications within relatively sophisticated GCE environment. The performance of the evaluated applications would then reflect the scientific capability of the entire GCE environment.
NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program.
-  S. Akioka and Y. Muraoka, “HPC benchmarks on Amazon EC2,” Proc. 24th IEEE Int. Conf. Advanced Information Networking and Applications Workshops (WAINA 2010), IEEE Computer Society, Apr. 2010, pp. 1029–1034.
-  CloudHarmony, “CloudHarmony benchmarks,” http://cloudharmony.com/benchmarks, 2013.
-  T. Dalman, T. Doernemann, E. Juhnke, M. Weitzel, M. Smith, W. Wiechert, K. Noh, and B. Freisleben, “Metabolic flux analysis in the Cloud,” Proc. 6th IEEE Int. Conf. e-Science (e-Science 2010), IEEE Computer Society, Dec. 2010, pp. 57–64.
-  J. M. Emison, “Google Compute Engine: Hands-on review,” http://www.informationweek.com/cloud-computing/infrastructure/google-compute-engine-hands-on-review/240002899?pgno=1, 2012.
-  J. M. Emison, “Google IaaS vs. Amazon EC2: New benchmarks,” http://www.informationweek.com/cloud-computing/infrastructure/google-iaas-vs-amazon-ec2-new-benchmarks/240004282, 2012.
-  C. Evangelinos and C. N. Hill, “Cloud computing for parallel scientific HPC applications: Feasibility of running coupled atmosphere-ocean climate models on Amazon s EC2,” Proc. 1st Workshop on Cloud Computing and its Applications (CCA 2008), Oct. 2008, pages 1 -6.
-  J. Geelan, “Google Compute Engine now available for all,” http://cloudcomputing.sys-con.com/node/2661914, 2013.
-  Google, “Google Compute Engine instances,” https://developers.google.com/compute/docs/instances, 2013.
-  Google, “Google Compute Engine Quota Policy” https://developers.google.com/compute/docs/quota, 2013.
-  Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, “Case study for running HPC applications in public Clouds,” Proc. 1st Workshop on Scientific Cloud Computing (ScienceCloud 2010) in conjunction with 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp. 395–401.
-  M. Humphrey, Z. Hill, K. Jackson, C. van Ingen, and Y. Ryu, “Assessing the value of Cloudbursting: A case study of satellite image processing on Windows Azure,” Proc. 7th IEEE Int. Conf. eScience (eScience 2011), IEEE Computer Society, Dec. 2011, pp. 126–133.
-  A. Iosup, S. Ostermann, N. Yigitbasi, R. Prodan, T. Fahringer, and D.H.J. Epema, “Performance analysis of Cloud computing services for many-tasks scientific computing,” IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 6, Jun. 2011, pp. 931–945.
-  Z. Li, L. O’Brien, R. Cai, and H. Zhang, “Towards a taxonomy of performance evaluation of commercial Cloud services,” Proc. 5th Int. Conf. Cloud Computing (IEEE CLOUD 2012), IEEE Computer Society, Jun. 1012, pp. 344–351.
-  Z. Li, L. O’Brien, H. Zhang, and R. Cai, “On a catalogue of metrics for evaluating commercial Cloud services,” Proc. 13th ACM/IEEE Int. Conf. Grid Computing (Grid 2012), IEEE Computer Society, Sept. 2012, pp. 164–173.
-  Z. Li, L. O’Brien, H. Zhang, and R. Cai, “A factor framework for experimental design for performance evaluation of commercial Cloud services,” Proc. 4th IEEE Int. Conf. Cloud Computing Technology and Science (CloudCom 2012), IEEE Computer Society, Dec. 2012, pp. 169–176.
-  Z. Li, L. O’Brien, and H. Zhang, “CEEM: A practical methodology for Cloud services evaluation,” Proc. 3rd IEEE Int. Workshop on the Future of Software Engineering for/in Cloud (FOSEC 2013) in conjunction with IEEE 9th World Congr. Services (IEEE SERVICES 2013), IEEE Computer Society, Jun.-Jul. 2013, to appear.
-  Z. Li, H. Zhang, L. O’Brien, R. Cai, and S. Flint, “On evaluating commercial Cloud services: A systematic review,” J. Syst. Software, in press.
-  D. C. Montgomery, Design and Analysis of Experiments, 7th ed. Hoboken, NJ: John Wiley & Sons, Inc., Jan. 2009.
-  S. Murphy, “GCE versus EC2: A comparison of instance pricing and characteristics, part 1,” http://datacommunitydc.org/blog/2013/01/gce-versus-ec2-a-comparison-of-instance-pricing-and-characteristics-part-1/, 2013.
-  S. Murphy, “Google Compute Engine vs Amazon EC2 Part 2: Synthetic CPU and Memory Benchmarks,” http://datacommunitydc.org/blog/2013/01/google-compute-engine-vs-amazon-ec2-part-2-synthetic-cpu-and-memory-benchmarks/, 2013.
-  NASA, “NAS Parallel Benchmarks,” http://www.nas.nasa.gov/publications/npb.html, 2013.
-  S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D.H.J. Epema, “A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing,” Proc. 1st Int. Conf. Cloud Computing (CloudComp 2009), Springer-Verlag, Oct. 2009, pp. 115–131.
-  M. R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, “Amazon S3 for science grids: A viable solution?,” Proc. 2008 Int. Workshop on Data-Aware Distributed Computing (DADC 2008), ACM Press, Jun. 2008, pp. 55–64.
-  J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz, “Runtime measurements in the Cloud: Observing, analyzing, and reducing variance,” VLDB Endowment, vol. 3, no. 1-2, Sept. 2010, pp. 460–471.
-  S. Stadil, “By the numbers: How Google Compute Engine stacks up to Amazon EC2,” http://gigaom.com/2013/03/15/by-the-numbers-how-google-compute-engine-stacks-up-to-amazon-ec2/, 2013.
-  E. Walker, “Benchmarking Amazon EC2 for highperformance scientific computing,” Usenix ;Login, vol. 33, no. 5, Oct. 2008, pp. 18–23.
-  P. Wayner, “Review: Google Compute Engine rocks the Cloud,” http://www.infoworld.com/d/cloud-computing/review-google-compute-engine-rocks-the-cloud-200591, 2012.
-  Zencoder, “First look at Google Compute Engine for video transcoding,” http://blog.zencoder.com/2012/07/23/first-look-at-google-compute-engine-for-video-transcoding/, 2012.
-  Q. Zhang, L. Cheng, and R. Boutaba, “Cloud computing: State-of-the-art and research challenges,” J. Internet Serv. Appl., vol. 1, no. 1, May 2010, pp. 7 -18.