The Virginia-based Standard Performance Evaluation Corp. (SPEC) has introduced a new Cloud benchmark, named SPEC Cloud IaaS to gauge the performance of IaaS (Infrastructure-as-a-Service) Cloud deployments. The SPEC Cloud IaaS 2018 benchmark tests replicated application instances, overall performance, relative scalability, and mean instance provisioning time of IaaS Cloud implementations, both public and private. This new Cloud benchmark supersedes the 2016 version that will be retired in three months.
Cloud vendors, users of Cloud services, providers of Cloud-related hardware and software, and researchers can practice the benchmark to assess how diverse implementations affect Cloud performance. Two multi-instance workloads are inserted within the SPEC Cloud IaaS framework that is the Yahoo Cloud Serving Benchmark (YCSB), which utilizes the Cassandra NoSQL database to store and recover data in a manner representative of social media applications; and the second, big data analytics based on a K-means clustering workload utilizing Hadoop MapReduce. The workloads are intended to experiment compute, storage and network performance of an IaaS Cloud subject to strict Quality of Service (QoS) metrics. Each workload operates as a distributed application involving six or seven Cloud instances that make up an application instance. Adding new application instances doubles the stress on the cloud infrastructure’s instance-provisioning capabilities and on its available resources, including CPU, memory, disk, and network.
Chair of the SPEC Cloud subcommittee, Paula Smith stated that SPEC Cloud IaaS 2018 integrates enhanced methodologies that assist ensure fairness, reliability and real-world rigor. It offers metrics that exceed relative performance and capacity measurement to demonstrate how typical Cloud application workloads perform as underlying resources approach full capacity. SPEC Cloud IaaS 2018 developed on the 2016 release with a range of updates, including easier installation of CBTOOL and workload images; revised internal methodologies for scoring, reporting and metric calculations; enhanced support for public clouds, OpenStack, and containers and container managers; an increased I/O workload that checks the limit of current storage technologies; and new versions of Hadoop for K-means and Cassandra for YCSB.