Bad Moon Rises Over Cloud Perceptions

By Nicole Hemsoth

April 28, 2011

Let’s pretend for a moment that you are the owner or technical lead on a web application that recently captured the public’s attention and swelled in popularity—to the point that serving those visitors or customers without building a new data center would be impossible.

Instead of resorting to that up-front investment, what you need, of course, is infrastructure that will scale with your traffic or transactions—a resource that will allow you to avoid the cost and expertise required to maintain a system robust enough to handle the flood. What you need, in other words, is a cloud-based infrastructure provider—preferably one that offers an attractive guarantee of uptime and continual reassurance that no matter what, that data is backed up and replicated to death in the event of disaster.

So, there you have it; you’re all set to serve users and wash your hands of the whole infrastructure problem. In one relatively swift move you’ve shed the need for expensive, cumbersome hardware and can roll ahead with your business.

What could be more convenient? The resources simply scale along with demand, you pay for that demand as it happens, and outside of your maintenance of the core business applications, you can sit back and relax.

Right?

According to CTO and founder of the cloud management firm Rightscale, Thorsten von Eicken, a former colleague of AWS chief Werner Vogels,  this assumption is part of what sparked some of the trouble for those who had their business lifeblood in the cloud. Those guarantees and assurances backing any cloud computing outlet were as good as gold, weren’t they?

Following the initial Amazon Web Services outage, von Eicken wrote that although many customers were able to resort to a solid “Plan B” in the case of an outage, some were not adequately prepared for such an event. He claims that “because of Amazon’s reliability has been incredible, many users were not well-prepared leading to widespread outages. Additionally, some users got caught by unforeseen failure modes rendering their failure plans ineffective.”

So is this to say that if you experienced extensive, damaging outages and data loss the burden ultimately falls on you due to lack of disaster recovery plans? Not necessarily. However, what Von Eicken and other notable experts on the cloud movement suggest is that there might have been some over-confidence in Amazon’s ability to take care of everything. That aside, one has to wonder if even AWS as a whole had been a little too confident.

Now let’s go back to the scenario with you at helm from before. You’ve prepared yourself in any way you thought was necessary or appropriate given Amazon’s very solid track record of performance and uptime, the extensive service level agreements (SLAs) and their several success stories of mission-critical cloud operations.

Nonetheless, you woke up this morning (if you were lucky enough to sleep following the outage) to the following message that Amazon sent out to some of its customers today:

Hello,
 
A few days ago we sent you an email letting you know that we were working on recovering an inconsistent data snapshot of one or more of your Amazon EBS volumes.  We are very sorry, but ultimately our efforts to manually recover your volume were unsuccessful.  The hardware failed in such a way that we could not forensically restore the data.
 
What we were able to recover has been made available via a snapshot, although the data is in such a state that it may have little to no utility…
 
If you have no need for this snapshot, please delete it to avoid incurring storage charges.
 
We apologize for this volume loss and any impact to your business.

Sincerely,

Amazon Web Services, EBS Support

This message was produced and distributed by Amazon Web Services LLC, 410 Terry Avenue North, Seattle, Washington 98109-5210

You read this a few times. It doesn’t really sink in at first since after all, wasn’t everything protected under, like 50 layers of different protection and duplication efforts on Amazon’s side?

You see key phrases and nothing else… “failed in such a way that we could not forensically restore the data…”

“..data is in such a state that it may have little to no utlilty…”

And your favorite line of all after you’ve had a few moments to really think about it:

 “If you have no need for this snapshot, please delete it to avoid incurring storage charges…”

At these lines, you cock a brow over your left eye, which started twitching occasionally a few days ago in the wake of the outage and now appears to be possessed.

Seriously?… This apologetic letter about your complete and total loss of my data and you’re warning me that I am going to be incurring storage charges?

(insert select profanities here)

While certainly Amazon could have massaged this message, lending a spoonful of sugar for such medicine, this is not the only way that communication has played a significant role in the increasingly bad press the normally stable infrastructure provider is receiving today.

As von Eicken noted of the initial outage, Amazon receives an “F” for its ability to effectively communicate with users throughout the first signs of trouble. Many are now claiming that failure extends to the data loss matter at hand now. At a time like this, however, good communication is needed more than ever—it seems that either they have no idea of the extent and cause of the loss or they are afraid to let people know how bad it is and how far it extends. Either way, this does not bode well for the company as it has opened the door to a bumrush of negative speculation.

The problem is, in Amazon’s defense, this is the first major problem with multiple, compounding failures that it has ever experienced. There have been latencies and delays in select zones in the past but nothing on this level—not even close to it.

In addition to opening the door for widespread criticism and speculation, it has also allowed competitors (not to mention the six or more backup/recovery companies that are rapid-firing press, barely able to contain their excitement over this outage and loss) to claim dominance—to state that they are immune from such disasters.

But we all know, no one is immune. If Amazon suffers this type of problem, Rackspace could suffer the same issue. If Rackspace trips momentarily, so could Microsoft’s services.

And the point that no one brings up here is that this same problem—and worse—could happen in your very own data center if you chose not to hop on the cloud bandwagon. And it could be far more destructive and expensive.

We’ll spare Amazon for a moment and say that this was written in haste. After all, they have been raked over the coals since news of sporadic data destruction broke…and broke in a very public way.

Just as cloud computing hit the mainstream media outlets in a big way over the past year, so too did news of the problems that could arise when you push your core business into the ether.

The announcement today that data was not only lost or temporarily unavailable—that instead it was actually destroyed—certainly doesn’t bode well for the future of mission critical applications being exclusively hosted on cloud computing infrastructure. It is unfortunate for IaaS providers that this should happen right at that much-anticipated golden moment of growing comfort with cloud computing, but perhaps we should consider this event in light of a few points that major media outlets aren’t talking about.

By the way, the media that I’ve encountered this morning paints the picture of what happened with the Amazon cloud in some rather black and white terms. Mainstream outlets necessarily be condemned for this however—after all, it’s not easy to come up with live news that is approachable for the folks that just a few months ago learned that clouds were more than just the puff upstairs while still painting a picture of the outage that is technically dense and thus in better context.

With that said, the media really doesn’t have a leg to stand (nor does anyone else at this point) when Amazon has been (notoriously) uncommunicative about what actually happened. Aside from sporadic updates following the initial outage and some updates that were dense but not necessarily revealing, the public, not to mention the users whose data may have been chewed up, are in the dark.

If an infrastructure provider of any size has a problem like this, the first item on the agenda should be communication. This not only protects them from wild speculation across media outlets, but it also protects the very notion of the cloud as a reasonable solution for everyone—no matter who they choose to rent hardware from.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “pre-exascale” award), parsed out additional information ab Read more…

By Tiffany Trader

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid whoops and hollers from the crowd, Thomas Sterling presented t Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out plans to push deeper into climate science and develop more gran Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale companies and their embrace of AI and deep learning – tha Read more…

By Doug Black

HPE Extreme Performance Solutions

Creating a Roadmap for HPC Innovation at ISC 2017

In an era where technological advancements are driving innovation to every sector, and powering major economic and scientific breakthroughs, high performance computing (HPC) is crucial to tackle the challenges of today and tomorrow. Read more…

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network designed to emulate and compete with the human brain. In thi Read more…

By Doug Black

Cray Brings AI and HPC Together on Flagship Supers

June 20, 2017

Cray took one more step toward the convergence of big data and high performance computing (HPC) today when it announced that it’s adding a full suite of big data and artificial intelligence software to its top-of-the-l Read more…

By Alex Woodie

AMD Charges Back into the Datacenter and HPC Workflows with EPYC Processor

June 20, 2017

AMD is charging back into the enterprise datacenter and select HPC workflows with its new EPYC 7000 processor line, code-named Naples, announced today at a “global” launch event in Austin TX. In many ways it was a fu Read more…

By John Russell

Hyperion: Deep Learning, AI Helping Drive Healthy HPC Industry Growth

June 20, 2017

To be at the ISC conference in Frankfurt this week is to experience deep immersion in deep learning. Users want to learn about it, vendors want to talk about it, analysts and journalists want to report on it. Deep learni Read more…

By Doug Black

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Tsinghua Crowned Eight-Time Student Cluster Champions at ISC

June 22, 2017

Always a hard-fought competition, the Student Cluster Competition awards were announced Wednesday, June 21, at the ISC High Performance Conference 2017. Amid wh Read more…

By Kim McMahon

GPUs, Power9, Figure Prominently in IBM’s Bet on Weather Forecasting

June 22, 2017

IBM jumped into the weather forecasting business roughly a year and a half ago by purchasing The Weather Company. This week at ISC 2017, Big Blue rolled out pla Read more…

By John Russell

Intersect 360 at ISC: HPC Industry at $44B by 2021

June 22, 2017

The care, feeding and sustained growth of the HPC industry increasingly is in the hands of the commercial market sector – in particular, it’s the hyperscale Read more…

By Doug Black

At ISC – Goh on Go: Humans Can’t Scale, the Data-Centric Learning Machine Can

June 22, 2017

I've seen the future this week at ISC, it’s on display in prototype or Powerpoint form, and it’s going to dumbfound you. The future is an AI neural network Read more…

By Doug Black

Cray Brings AI and HPC Together on Flagship Supers

June 20, 2017

Cray took one more step toward the convergence of big data and high performance computing (HPC) today when it announced that it’s adding a full suite of big d Read more…

By Alex Woodie

AMD Charges Back into the Datacenter and HPC Workflows with EPYC Processor

June 20, 2017

AMD is charging back into the enterprise datacenter and select HPC workflows with its new EPYC 7000 processor line, code-named Naples, announced today at a “g Read more…

By John Russell

Hyperion: Deep Learning, AI Helping Drive Healthy HPC Industry Growth

June 20, 2017

To be at the ISC conference in Frankfurt this week is to experience deep immersion in deep learning. Users want to learn about it, vendors want to talk about it Read more…

By Doug Black

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Just how close real-wo Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the cam Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its a Read more…

By Tiffany Trader

Google Pulls Back the Covers on Its First Machine Learning Chip

April 6, 2017

This week Google released a report detailing the design and performance characteristics of the Tensor Processing Unit (TPU), its custom ASIC for the inference Read more…

By Tiffany Trader

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Nvidia’s Mammoth Volta GPU Aims High for AI, HPC

May 10, 2017

At Nvidia's GPU Technology Conference (GTC17) in San Jose, Calif., this morning, CEO Jensen Huang announced the company's much-anticipated Volta architecture a Read more…

By Tiffany Trader

Facebook Open Sources Caffe2; Nvidia, Intel Rush to Optimize

April 18, 2017

From its F8 developer conference in San Jose, Calif., today, Facebook announced Caffe2, a new open-source, cross-platform framework for deep learning. Caffe2 is the successor to Caffe, the deep learning framework developed by Berkeley AI Research and community contributors. Read more…

By Tiffany Trader

Leading Solution Providers

MIT Mathematician Spins Up 220,000-Core Google Compute Cluster

April 21, 2017

On Thursday, Google announced that MIT math professor and computational number theorist Andrew V. Sutherland had set a record for the largest Google Compute Engine (GCE) job. Sutherland ran the massive mathematics workload on 220,000 GCE cores using preemptible virtual machine instances. Read more…

By Tiffany Trader

Google Debuts TPU v2 and will Add to Google Cloud

May 25, 2017

Not long after stirring attention in the deep learning/AI community by revealing the details of its Tensor Processing Unit (TPU), Google last week announced the Read more…

By John Russell

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

Russian Researchers Claim First Quantum-Safe Blockchain

May 25, 2017

The Russian Quantum Center today announced it has overcome the threat of quantum cryptography by creating the first quantum-safe blockchain, securing cryptocurrencies like Bitcoin, along with classified government communications and other sensitive digital transfers. Read more…

By Doug Black

Groq This: New AI Chips to Give GPUs a Run for Deep Learning Money

April 24, 2017

CPUs and GPUs, move over. Thanks to recent revelations surrounding Google’s new Tensor Processing Unit (TPU), the computing world appears to be on the cusp of Read more…

By Alex Woodie

DOE Supercomputer Achieves Record 45-Qubit Quantum Simulation

April 13, 2017

In order to simulate larger and larger quantum systems and usher in an age of “quantum supremacy,” researchers are stretching the limits of today’s most advanced supercomputers. Read more…

By Tiffany Trader

Messina Update: The US Path to Exascale in 16 Slides

April 26, 2017

Paul Messina, director of the U.S. Exascale Computing Project, provided a wide-ranging review of ECP’s evolving plans last week at the HPC User Forum. Read more…

By John Russell

Knights Landing Processor with Omni-Path Makes Cloud Debut

April 18, 2017

HPC cloud specialist Rescale is partnering with Intel and HPC resource provider R Systems to offer first-ever cloud access to Xeon Phi "Knights Landing" processors. The infrastructure is based on the 68-core Intel Knights Landing processor with integrated Omni-Path fabric (the 7250F Xeon Phi). Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This