Bad Moon Rises Over Cloud Perceptions

By Nicole Hemsoth

April 28, 2011

Let’s pretend for a moment that you are the owner or technical lead on a web application that recently captured the public’s attention and swelled in popularity—to the point that serving those visitors or customers without building a new data center would be impossible.

Instead of resorting to that up-front investment, what you need, of course, is infrastructure that will scale with your traffic or transactions—a resource that will allow you to avoid the cost and expertise required to maintain a system robust enough to handle the flood. What you need, in other words, is a cloud-based infrastructure provider—preferably one that offers an attractive guarantee of uptime and continual reassurance that no matter what, that data is backed up and replicated to death in the event of disaster.

So, there you have it; you’re all set to serve users and wash your hands of the whole infrastructure problem. In one relatively swift move you’ve shed the need for expensive, cumbersome hardware and can roll ahead with your business.

What could be more convenient? The resources simply scale along with demand, you pay for that demand as it happens, and outside of your maintenance of the core business applications, you can sit back and relax.


According to CTO and founder of the cloud management firm Rightscale, Thorsten von Eicken, a former colleague of AWS chief Werner Vogels,  this assumption is part of what sparked some of the trouble for those who had their business lifeblood in the cloud. Those guarantees and assurances backing any cloud computing outlet were as good as gold, weren’t they?

Following the initial Amazon Web Services outage, von Eicken wrote that although many customers were able to resort to a solid “Plan B” in the case of an outage, some were not adequately prepared for such an event. He claims that “because of Amazon’s reliability has been incredible, many users were not well-prepared leading to widespread outages. Additionally, some users got caught by unforeseen failure modes rendering their failure plans ineffective.”

So is this to say that if you experienced extensive, damaging outages and data loss the burden ultimately falls on you due to lack of disaster recovery plans? Not necessarily. However, what Von Eicken and other notable experts on the cloud movement suggest is that there might have been some over-confidence in Amazon’s ability to take care of everything. That aside, one has to wonder if even AWS as a whole had been a little too confident.

Now let’s go back to the scenario with you at helm from before. You’ve prepared yourself in any way you thought was necessary or appropriate given Amazon’s very solid track record of performance and uptime, the extensive service level agreements (SLAs) and their several success stories of mission-critical cloud operations.

Nonetheless, you woke up this morning (if you were lucky enough to sleep following the outage) to the following message that Amazon sent out to some of its customers today:

A few days ago we sent you an email letting you know that we were working on recovering an inconsistent data snapshot of one or more of your Amazon EBS volumes.  We are very sorry, but ultimately our efforts to manually recover your volume were unsuccessful.  The hardware failed in such a way that we could not forensically restore the data.
What we were able to recover has been made available via a snapshot, although the data is in such a state that it may have little to no utility…
If you have no need for this snapshot, please delete it to avoid incurring storage charges.
We apologize for this volume loss and any impact to your business.


Amazon Web Services, EBS Support

This message was produced and distributed by Amazon Web Services LLC, 410 Terry Avenue North, Seattle, Washington 98109-5210

You read this a few times. It doesn’t really sink in at first since after all, wasn’t everything protected under, like 50 layers of different protection and duplication efforts on Amazon’s side?

You see key phrases and nothing else… “failed in such a way that we could not forensically restore the data…”

“ is in such a state that it may have little to no utlilty…”

And your favorite line of all after you’ve had a few moments to really think about it:

 “If you have no need for this snapshot, please delete it to avoid incurring storage charges…”

At these lines, you cock a brow over your left eye, which started twitching occasionally a few days ago in the wake of the outage and now appears to be possessed.

Seriously?… This apologetic letter about your complete and total loss of my data and you’re warning me that I am going to be incurring storage charges?

(insert select profanities here)

While certainly Amazon could have massaged this message, lending a spoonful of sugar for such medicine, this is not the only way that communication has played a significant role in the increasingly bad press the normally stable infrastructure provider is receiving today.

As von Eicken noted of the initial outage, Amazon receives an “F” for its ability to effectively communicate with users throughout the first signs of trouble. Many are now claiming that failure extends to the data loss matter at hand now. At a time like this, however, good communication is needed more than ever—it seems that either they have no idea of the extent and cause of the loss or they are afraid to let people know how bad it is and how far it extends. Either way, this does not bode well for the company as it has opened the door to a bumrush of negative speculation.

The problem is, in Amazon’s defense, this is the first major problem with multiple, compounding failures that it has ever experienced. There have been latencies and delays in select zones in the past but nothing on this level—not even close to it.

In addition to opening the door for widespread criticism and speculation, it has also allowed competitors (not to mention the six or more backup/recovery companies that are rapid-firing press, barely able to contain their excitement over this outage and loss) to claim dominance—to state that they are immune from such disasters.

But we all know, no one is immune. If Amazon suffers this type of problem, Rackspace could suffer the same issue. If Rackspace trips momentarily, so could Microsoft’s services.

And the point that no one brings up here is that this same problem—and worse—could happen in your very own data center if you chose not to hop on the cloud bandwagon. And it could be far more destructive and expensive.

We’ll spare Amazon for a moment and say that this was written in haste. After all, they have been raked over the coals since news of sporadic data destruction broke…and broke in a very public way.

Just as cloud computing hit the mainstream media outlets in a big way over the past year, so too did news of the problems that could arise when you push your core business into the ether.

The announcement today that data was not only lost or temporarily unavailable—that instead it was actually destroyed—certainly doesn’t bode well for the future of mission critical applications being exclusively hosted on cloud computing infrastructure. It is unfortunate for IaaS providers that this should happen right at that much-anticipated golden moment of growing comfort with cloud computing, but perhaps we should consider this event in light of a few points that major media outlets aren’t talking about.

By the way, the media that I’ve encountered this morning paints the picture of what happened with the Amazon cloud in some rather black and white terms. Mainstream outlets necessarily be condemned for this however—after all, it’s not easy to come up with live news that is approachable for the folks that just a few months ago learned that clouds were more than just the puff upstairs while still painting a picture of the outage that is technically dense and thus in better context.

With that said, the media really doesn’t have a leg to stand (nor does anyone else at this point) when Amazon has been (notoriously) uncommunicative about what actually happened. Aside from sporadic updates following the initial outage and some updates that were dense but not necessarily revealing, the public, not to mention the users whose data may have been chewed up, are in the dark.

If an infrastructure provider of any size has a problem like this, the first item on the agenda should be communication. This not only protects them from wild speculation across media outlets, but it also protects the very notion of the cloud as a reasonable solution for everyone—no matter who they choose to rent hardware from.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Geospatial Data Research Leverages GPUs

August 17, 2017

MapD Technologies, the GPU-accelerated database specialist, said it is working with university researchers on leveraging graphics processors to advance geospatial analytics. The San Francisco-based company is collabor Read more…

By George Leopold

Intel, NERSC and University Partners Launch New Big Data Center

August 17, 2017

A collaboration between the Department of Energy’s National Energy Research Scientific Computing Center (NERSC), Intel and five Intel Parallel Computing Centers (IPCCs) has resulted in a new Big Data Center (BDC) that Read more…

By Linda Barney

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last week the cloud giant released deeplearn.js as part of that in Read more…

By John Russell

HPE Extreme Performance Solutions

Leveraging Deep Learning for Fraud Detection

Advancements in computing technologies and the expanding use of e-commerce platforms have dramatically increased the risk of fraud for financial services companies and their customers. Read more…

Spoiler Alert: Glimpse Next Week’s Solar Eclipse Via Simulation from TACC, SDSC, and NASA

August 17, 2017

Can’t wait to see next week’s solar eclipse? You can at least catch glimpses of what scientists expect it will look like. A team from Predictive Science Inc. (PSI), based in San Diego, working with Stampede2 at the Read more…

By John Russell

Microsoft Bolsters Azure With Cloud HPC Deal

August 15, 2017

Microsoft has acquired cloud computing software vendor Cycle Computing in a move designed to bring orchestration tools along with high-end computing access capabilities to the cloud. Terms of the acquisition were not disclosed. Read more…

By George Leopold

HPE Ships Supercomputer to Space Station, Final Destination Mars

August 14, 2017

With a manned mission to Mars on the horizon, the demand for space-based supercomputing is at hand. Today HPE and NASA sent the first off-the-shelf HPC system i Read more…

By Tiffany Trader

AMD EPYC Video Takes Aim at Intel’s Broadwell

August 14, 2017

Let the benchmarking begin. Last week, AMD posted a YouTube video in which one of its EPYC-based systems outperformed a ‘comparable’ Intel Broadwell-based s Read more…

By John Russell

Deep Learning Thrives in Cancer Moonshot

August 8, 2017

The U.S. War on Cancer, certainly a worthy cause, is a collection of programs stretching back more than 40 years and abiding under many banners. The latest is t Read more…

By John Russell

IBM Raises the Bar for Distributed Deep Learning

August 8, 2017

IBM is announcing today an enhancement to its PowerAI software platform aimed at facilitating the practical scaling of AI models on today’s fastest GPUs. Scal Read more…

By Tiffany Trader

IBM Storage Breakthrough Paves Way for 330TB Tape Cartridges

August 3, 2017

IBM announced yesterday a new record for magnetic tape storage that it says will keep tape storage density on a Moore's law-like path far into the next decade. Read more…

By Tiffany Trader

AMD Stuffs a Petaflops of Machine Intelligence into 20-Node Rack

August 1, 2017

With its Radeon “Vega” Instinct datacenter GPUs and EPYC “Naples” server chips entering the market this summer, AMD has positioned itself for a two-head Read more…

By Tiffany Trader

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

Leading Solution Providers

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

Six Exascale PathForward Vendors Selected; DoE Providing $258M

June 15, 2017

The much-anticipated PathForward awards for hardware R&D in support of the Exascale Computing Project were announced today with six vendors selected – AMD Read more…

By John Russell

Top500 Results: Latest List Trends and What’s in Store

June 19, 2017

Greetings from Frankfurt and the 2017 International Supercomputing Conference where the latest Top500 list has just been revealed. Although there were no major Read more…

By Tiffany Trader

IBM Clears Path to 5nm with Silicon Nanosheets

June 5, 2017

Two years since announcing the industry’s first 7nm node test chip, IBM and its research alliance partners GlobalFoundries and Samsung have developed a proces Read more…

By Tiffany Trader

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

  • arrow
  • Click Here for More Headlines
  • arrow
Share This