HPCwire: Cloud for Academia?

Since 1986 - Covering the Fastest Computers in the World and the People Who Run Them

HPCwire >> Special Features >> HPC in the Cloud >> HPC in the Cloud Features

November 02, 2009

Cloud for Academia?

by Adrian Mouat

Page: 1 of 2
1 | 2 All »

Grid computing was born in academia and was originally designed to support scientific and research computing. In contrast, cloud computing has a business background and is designed to enable the delivery of scalable Web applications.

The BEinGRID project has looked into how Grid is appropriate for business use (and has run several successful business experiments proving this proposition), but what about looking at whether the cloud is useful for academia? Can it be effectively used to run scientific codes, such as those found in climate modelling, fluid dynamics or molecular physics simulations, which have traditionally required the use of supercomputers?

On the face of it, cloud services offer a compelling, simple and relatively cost-effective HPC proposition -- just pay for as many CPUs as you want, when you want them. Of course, the truth isn't quite as simple as that.

Take a look at the Google Cloud offering, App Engine. Users access App Engine through an API, which places quite a lot of restrictions on the code that can be run, including:

It must be written either in Python or Java (or use a JVM based interpreter or compiler) -- meaning any C or Fortran codes have to be ported.
It can't start any threads (instead the API is used to start a new task).
Any single request/task must complete in 30 seconds.
It has to stay within quotas on CPU, bandwidth and storage usage.

For full details, see the App Engine documentation. Note that there are different quotas for the free and billed service, and that it is possible to negotiate increases to the quotas.

This doesn't make scientific computing impossible, but it does put in place a lot of barriers. It would be interesting to see what could be achieved computationally, given the above restrictions, by an academic research group that chose to use App Engine. However, the bottom line is that Google App Engine is more suited to creating dynamic web applications, such as photo and document editing tools, than to processing long-lived scientific computations.

Amazon's offering, EC2, is a lot more promising. EC2 gives users much more access and control over the system through the use of virtualisation. Users are free to install whatever software and applications they need on EC2.

Users provide "virtual images", instances of which can be launched at any time and will normally be running in under 10 minutes. By default, only 20 instances (per region) can normally be launched, but users can apply to increase this limit, potentially allowing thousands of instances to be launched. Amazon also supports Hadoop, Condor and OpenMPI for batch/parallel processing. The Data Wrangling blog has some in-depth information on using Amazon to set up an MPI cluster.

For data storage, the Amazon S3 service can be used to store the enormous amounts of data produced by some scientific applications. Access to the data is controlled via Access Control Lists (ACLs) and data is encrypted during transmission using SSL. Users are encouraged to encrypt any sensitive data being stored in S3. It is important to note that Amazon does not guarantee that data will not be lost or compromised (see point 7.2 of the AWS Customer Agreement).

So, it should be fairly easy to get most scientific computing codes running in parallel on EC2. But what's the performance like? There has been some research and the results are mixed. Comparing a roughly equivalent amount of CPU resources, super-computer clusters are typically much faster at processing scientific codes, largely due to having a better interconnect (see this article by Edward Walker). However, if we include the amount of time it takes to get the code running (i.e., to request and boot the images on EC2 and to wait in the queue on a super-computer cluster), EC2 is likely to be faster in many cases, dependent on the size of the job and scheduling policies (as shown by Ian Foster on his blog). In the future, EC2 may offer an even more competitive service if they upgrade their systems.

Page: 1 of 2
1 | 2 All »

Article Tools

Share Options

(Digg, Technorati, more)

Subscribe

Subscribe to HPCwire

Discussion

There are 1 discussion items posted.

CloudBerry Explorer - freeware client for Amazon S3
Submitted by cloudberryman on 11/25/2009 - 3:15AM

If you want to explorer Amazon S3 storage, upload data or configure ACL check out CloudBerry Explorer that helps to
manage S3 on Windows . It is a freeware. http://cloudberrylab.com/

Post #1

RSS Feeds

Feeds by Topic

Feeds by Industry

Feeds by Content Type

Feature Articles

The Week in Review

The National Science Foundation has awarded funding to four projects as part of the Future Internet Architecture program; and the 3PAR bidding war is won by HP. We recap those stories and more in our weekly wrapup.
Read More...

Intel Flexes Parallel Programming Muscles

Intel Corp has released Parallel Studio 2011, a set of four tools designed to mainstream software development on multicore x86 architectures. The update folds in a number of parallel programming technologies that the company has acquired or developed independently over the past few years, including the Cilk Arts and RapidMind technologies, and Intel's own Ct data parallel language framework.
Read More...

Startup Makes Liquid Cooling an Immersive Experience

There's nothing like a blazing hot summer to focus one's attention on the best ways to keep cool. That goes for datacenter operators as well, who are equally worried about keeping their servers properly chilled. While there is no shortage of innovative cooling solutions being proffered by various vendors, a new liquid immersion cooling solution from startup Green Revolution Cooling could end up being the best of them all.
Read More...

Read more HPCwire features...

Around the Web

HP, Hynix Start Memristor on Path to Commercialization

Sep 02 | Could see first products in three years. Read more...

TED Talks for the IT Crowd

Sep 01 | A hand-picked selection of video presentations from the TED conference -- because the next big thing has to start somewhere. Read more...

LHC Compute Grid Teaches Some Valuble Lessons

Aug 30 | CERN project adapts its computation and storage strategy as hardware gets cheaper and better. Read more...

Godson CPUs Groomed for Supercomputing Duty

Aug 26 | Chinese-made chip adds vector SIMD unit; delivers 128 gigaflops in 40 watts. Read more...

Power7 Hub Chip Key to IBM's PERCS Super

Aug 25 | Hot Chips presentation offers insights on supercomputer design. Read more...

Read more headlines...

Featured Whitepapers

Effective Backup and Restore

Jul 29 | | Panasas storage solutions deliver high throughput with many concurrent backup IO streams to standard backup applications such as Veritas NetBackup™ or EMC® NetWorker™. Download this whitepaper to understand the essential elements for effective backup and restore: the tape subsystem, networking, file system workload and administrative policy.

GPU Cluster Realities Whitepaper from Platform Computing

Jul 28 | | As compelling economics and performance drive GPUs into HPC clusters, developers are scrambling to catch up. Download this whitepaper from Platform Computing to understand how to capture the benefits of exciting new GPU capabilities.

View the White Paper Library

Multimedia

Webcast: Are you drowning in data?

In this webinar you will hear about the current storage challenges facing the HPC community, how Panasas storage solutions provide exceptional performance, scalability, and manageability, and how you can achieve the lowest total Cost of Ownership with a system that installs and configures in 15 minutes.

Webcast: Virtualized Data Center Roundtable

Join this online panel discussion for live Q&A with leading industry experts, analysts, and end-users to discuss the latest innovations, best practices, barriers to implementation, and measurable benefits of server virtualization with a particular focus on today's real world solutions.

Webcast: Watch SC09 Birds of a Feather Video: Scalable Fault-Tolerant HPC Supercomputers

Learn about scalable fault-tolerant architectures and examples of energy efficient and scalable supercomputing clusters using dual QDR InfiniBand to combine capacity computing with network failover capabilities with the help of programming languages such as MPI and a robust Linux cluster management package.