From Rendering LOTR to Animating HPC Clouds

By Nicole Hemsoth

September 16, 2013

Much has been written about the incredible animation work that went on behind the scenes of the Lord of the Rings series, but without the rendering horsepower of high performance systems, none of that would have been possible. It took an entire “supercomputing center” and unique cloud build-out for the team to continue—a process that led to the creation of Greenbutton—one of the few companies that identify as HPC cloud oriented.

In this audio interview, HPCwire editor, Nicole Hemsoth, talked with Scott Houston, the former CIO for WETA Digital, which provided the IT power infrastructure that powered the stunning visual effects for the films. Scott is currently the CEO of Greenbutton, which spun out of this work.

HPCwire: Scott, we talked back in 2011 at the Supercomputing conference, about many of your experiences on the Lord of the Rings films. Can you give us a sense of the background, what some of the decision making processes were when they were looking at different IT solutions?

Houston: It’s really been an interesting journey for me. In fact, back in 2003, the cloud really wasn’t available, so we literally had to go out and buy a thousand processors for one particular shot – it’s the battle of Pelennor Fields. I still have shivers up my spine every time I see that shot. It’s the shot that’s actually midway through the Return of the King, and it’s when the 14,000 rides of the Rohirrim (the horse warriors), clash on the battlefield with 83,000 orcs.

And the reality is that even though we had four data centers and clustered two and a half thousand processors, we literally had to go out and buy another thousand processors and build a brand new data center just to get that one shot done right at the end of the production of the movie. It was at that time, at the conclusion of the movie back in 2003, and early 2004 that I realized that there had to be a better way. And in fact, that infrastructure was sitting idle at the end of the movie. The next production for WETA was King Kong, so we didn’t really need those extra thousand processors.

So I formed a consortium called the New Zealand Supercomputer center, and we rented out time in that. We did some biotech work, we did some seismic processing, we helped render the movie Happy Feet. But the problem was for a number of our users, they just wanted to run small jobs, so that’s really where the genesis of green button was born. I saw an opportunity in the market to create some software to automate the process of using this capacity on demand, well before the cloud came around.

And that’s been great and a fantastic story to close this loop is that today we’re working with a company in Mexico that is rendering a full feature animation – a full length movie – that is rendered entirely on the cloud. So in the future CIOs and CTOs that are making movies and doing large computationally intensive projects, may not necessarily have to go out and buy that capacity in the future.

HPCwire: What is the status of the New Zealand Supercomputer center now? Are you running a balance between research/scientific workloads and entertainment/media workloads? Where does that stand?

Houston: That environment was shut down in 2009 and we’ve been cloud only since then. So today, we have jobs running on Windows Azure, Amazon, vCloud environments, and recently, earlier this year we announced support for Open Stack.

HPCwire: Since you’ve been heavy users from the beginning of both Amazon and Azure, what do you think some of the differences are – advantages and disadvantages between those platforms? Where is the advantage of Amazon over Azure, and Azure over Amazon?

Houston: Good question. We have customers, I think at last count in 77 different countries, so in many cases it just comes down to geography – where is the nearest data center and how fat can I have – you know – what is bandwidth of the pipe to get to that datacenter. So a number of those decisions are driven by where the nearest datacenter is.

Almost invariably the technical decisions – so from the processing point of view, from the support point of view…We do have workloads that run on Amazon and primarily one of them is a seismic processing on demand service that we had called Cloud Claritas, and that’s running on Amazon primarily because Amazon supports 10 gigabit Ethernet. The application is MPI-based. So that has driven that decision based on the technical requirements.

Sometimes it’s come down to memory. So there’s new large memory instances on Azure, now. Some of those workloads were running on Amazon, and some of those have been ported over to Azure. There’s a company that has just been setup called ProfitBricks, and they have InfiniBand support, and today not many other cloud providers have InfiniBand, so we’re starting to see some workloads running there.

To answer your question, often it will come down to the geographic location of the customer, and then also the technical performance, or the requirements of the workload.

HPCwire: If you’re talking about seismic processing for instance, not only is that computationally intensive, it’s data intensive. If you’re talking about using public cloud resources, I would imagine that the data movement costs would be pretty significant – how do you balance that out?

Houston: Good question. The reality is – and I’m embarrassed to say this – when we’re talking about moving 20 or 30 terabytes, which is a large scale 3D seismic processing run – we literally still ship the drives. You can’t beat the bandwidth of a Phoenix truck to be honest.

Interestingly enough, we’ve developed a product, which is part of our cloud fabric product called Cloud Sync. It’s a downloadable tool that in the background will move the data to the cloud. We just added an FTP capability that will use UDP protocols and do bulk upload. We’ve just done some recent testing, and we’re getting close to 2.4 gigabits per second. That’s theoretical performance, but that means I could technically, in the right environment, move a terabyte of data in under an hour, with the right connection from the customers site to the cloud. That may be – that will start to be a game changer.

I think we’re starting to see increased bandwidth into the datacenter – there are more cloud datacenters, and so I think that will be overcome.  Not today, or tomorrow, or even this year, but over the next couple of years, I think the bandwidth problem will be solved. The whole concept of GreenButton is to push the green button from within your desktop application, and your job will start running. Well, clearly we need to get the data there, so Cloud Sync will manage that data synchronization in the background so that when you push the button, hopefully all of the files are there and we just do a quick synchronization and the job will start running.

HPCwire: That problem you mentioned is one that everyone is trying to tackle – it seems pretty tricky.  The theoretical performance you just cited there is pretty impressive. What’s the actual performance based on some use cases?

Houston: Well, we’ll be launching the product later this month, so we’ll probably go back out. The reality is, it will clearly depend on what sort of connection the customer has from their network provider into the datacenter. Many of the datacenters have large pipes into them, so it’s actually going to come down to the performance from the customer to the datacenter. But we’ve just started working with a company that is putting in a dedicated pipe between their facility and their cloud provider, so I would think in a couple of months we will have some real world use cases.

HPCwire: That’s interesting. We’ll keep our eyes open for it. Let’s talk about GreenButton the company for a second. Let me sure I have this straight – you’re a platform as a service company dedicated specifically to HPC workloads. I know today you had an announcement around higher end analytics services. Can you describe the company’s focus in terms of the types of applications and needs that you’re serving specifically, and what is it that makes GreenButton unique and distinctly HPC oriented.

Houston: Often it depends on who we’re talking to. We’re in this growth stage, and HPC hasn’t been sexy. It’s always been sexy to me, of course, but it hasn’t been particularly sexy in the marketplace. The growth in the industry hasn’t been particularly investable, and I know that we have an HPC audience, but – The interesting thing for us is that we don’t just focus on rendering seismic, genomic sequencing, Monte Carlos, and CFD.  A number of our customers – and we just had an announcement today – one of our largest customers is in the social media space. So really what we look at at Green Button is any type of workload, but it has to be computationally intensive.

Interestingly enough, we’re doing a lot of work on video indexing. So we’re taking an hour of video content in, and using key algorithms and processing that video and making that indexable. And that’s not a traditional HPC type workload, but it is a big compute workload.  And that’s a product that we’ve created called inCus that uses an algorithm called Microsoft called MAVIS. So it’s not just traditional HPC, we’ve got social media customers, we’re processing video, yes we’re rendering movies and we’re doing seismic processing work, but we’ve also been putting some work into running Hadoop workloads and big data analytics through our GreenButton cloud fabric engine as well. So for us, it’s not just a particular vertical market, it’s anything that is computationally intensive, or big data intensive, we can take that workload, run it on GreenButton cloud fabric and deploy that workload to any cloud platform.

HPCwire: That comment that HPC’s definition is expanding beyond the research, scientific computing workloads – that’s getting more and more common as data intensive / computationally intensive sort of merge. But when it comes to cloud computing, especially if you’re using a public cloud resource, the concerns really don’t change – they get more intense actually whether it’s computationally intensive or data intensive – where there’s a performance gap, and so people that require high, high performance, have to really, I would imagine, take a very close look at whatever cloud service they’re going to use because they’re taking a pretty big hit in the virtualization side – data movement side. How do you help customers work through that, and how do you prove the ROI of this over just buying a bunch of infrastructure, which is a big up front cost.

Houston: I think that’s an ongoing process. The reality is, from early on, we’ve talk to customers, because they’ll run a job on their local server environment, then they’ll run it on the cloud and they’ll go, it runs slower. And there’s no denying that. One, we have to get the data up there, and there’s nothing like a dedicated environment that is running in their own datacenter.

That said, we’re seeing significant investment from the cloud providers in their infrastructures. So while there may never be parity in running the job offsite, from a high performance dedicated environment on site – from a financial perspective, it really does make sense. I think that the sort of workloads that we’re talking about, it would be a very brave CIO that’s says “I’m going all into the cloud.” I’ve been in the cloud for 10 years, and I’m not sure that I would say it in their position.

The reality is that the cloud makes a heck of a lot of sense to take specific workloads that either take a long time, or they’re very bursty – they’re only project based – and use the cloud to scale out the business.  So I’d encourage IT managers to use the cloud to support either very intensive workloads that they really don’t have the bandwidth or capability to run in-house, and not necessarily just go all into the cloud.  We’d love to talk to those folks, but I think we’re still really a couple years away from the cloud being all encompassing and customers not needing datacenters anymore.

To give you an example of that, a number of our customers are using Green Button to render their jobs. And I did say one of them rendered an entire movie, but most of our customers will have really hard shots that the render jobs will run often for a matter of days, and they’re able to take those jobs, run them offsite on the cloud, and continue using their data center more efficiently for smaller jobs or the jobs that they need a quick turnaround on.

HPCwire: To go back to some of the use cases we talked about earlier, you said you’re seeing a lot of growth on the social media side, on the analytics side, big data – whatever that encompasses – it’s very big – what are some of the emerging markets that you think balance computationally intensive workload needs with whatever advantages the cloud can offer. Where is the real hot growth right now?

Houston: it’s a little bit of both. We can’t talk about this customer publically, but they’re a Fortune 500 manufacturing customer – they have a large Monte Carlo simulation that was running in their internal cluster and taking 3 hours to run. We are able to take those workloads using Green Button Cloud Fabric, and spin up 1,100 processors and have that job completed at under 10 minutes. And that’s a traditional HPC workload, and just the power of our spinning up 100 processors and running that job and getting it turned around in 10 minutes is actually quite transformational for this customer.

We’re certainly seeing great growth in traditional HPC workloads – biotech, seismic, rendering. We’re starting to see CFD workloads – we have a customer in Germany that is doing fire dynamics simulation and that’s a traditional HPC MPI based workload and they’re running those jobs in the cloud.

But also balancing that are some really interesting opportunities around video and social media. The press release you’ll see on our website today is with a company called Tout, and effectively they provide 15 second video “tweets” (if you like) through the mobile phone, and they’re using Green Button to provide analytics for those customers – what sort of things they’re talking about, what are the themes, what are the ideas, what are the hot subjects.

So  balancing out the traditional HPC workloads are a whole lot of new workloads that are computationally intensive but not viewed as traditional HPC.

HPCwire: In the case like the manufacturing one you cited – cloud pricing is already tricky enough to figure out for a lot of users – how does this layer factor into pricing. How do you work this out?

Houston: The good news is that it’s pretty darn cheap. This particular workload – just think about the economics of it – if I’m spinning up 1,100 processors, most cloud providers are charging by the minute, and I want to run a job, and I’m ten minutes. So that’s $20 or $30 dollars to run the job. So to take that turnaround from under 3 hours to 10 minutes, and it costs $20 or $30 dollars to run a job – and they’ll negotiate that with their cloud provider – that’s a pretty compelling return on investment or business case to make.

I think the challenge is for the cloud provider in that the processing is so darned attractive for the end user. The end user just pays for the compute and storage they use for a particular job. It is becoming more and more compelling for many customers. The challenge for the cloud provider is how will they make money out of that. And if you’ve seen what’s happening in the cloud, you’ve seen the transition of the SAAS type applications, or hosting websites, and you know the cloud makes good sense for that. Increasingly organizations are using the cloud for offsite storage, and that makes sense.

None of those are particularly profitable for the cloud providers. The real money and profit for the cloud providers is in on demand computational work. The challenge for them is around utilization. If they can ensure their cloud is fully utilized, or even 80% utilized, it’s a profitable business, so how do you attract enough customers to keep that utilization. And it’s a win-win if you can do it from a customer point of view – from only using the cloud 10 or 20 or 30 percent of my time – it just makes sense. In fact the analytics that we’re doing right now with current cloud pricing, it’s almost 50%. If I built a datacenter and I’m running at less than 50% of the time, depending on my licensing costs, it may well be cheaper to run those jobs on the cloud with the current pricing model. So that’s an interesting tipping point that I’m sure most CIOs and CFOs are considering at the moment.

HPCwire: Right. So as with everything in high end infrastructure, it all depends.

Houston: I don’t think you want to argue that the pricing is compelling. The biggest problem, and the problem that we’ve been really focused on is governance. Anybody that is running their jobs in the cloud – hopefully they’re having a good experience. What is a really bad experience for people is not the costs, but where to appoint the costs.

If you get a bill from a cloud provider today – and it doesn’t matter who it is, whether it’s Amazon or Microsoft or anybody – it’s like a phone bill.  You’ve got no idea who ran the jobs, what department they’re working or what project that was.  Did they have the authorization to run the jobs? I’m sure most organizations are running their jobs on the cloud and they’re using the company credit card.

So for an IT department and a finance department, wrangling with those costs and apportioning that to a user, to a project, to a department – it’s an absolute nightmare; so we’ve been working very hard at providing governance. One of the things that we’ve just recently been granted a patent on it, is our ability to profile a job and provide an SLA commitment in time and cost. We think that’s unique and I think that’s going to be the game changer and perhaps the tipping point for broader utilization in the cloud is how long will it take, how much will it cost, and then really apportioning that to the correct department, user, project.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combined peak computing capacity, the new systems will extend the a Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

ASC18: Tough Applications & Tough Luck

May 17, 2018

The applications at the ASC18 Student Cluster Competition were tough. Tougher than the $3.99 steak special at your local greasy spoon restaurant. The apps are so tough that even Chuck Norris backs away from them slowly. Read more…

By Dan Olds

HPE Extreme Performance Solutions

HPC and AI Convergence is Accelerating New Levels of Intelligence

Data analytics is the most valuable tool in the digital marketplace – so much so that organizations are employing high performance computing (HPC) capabilities to rapidly collect, share, and analyze endless streams of data. Read more…

IBM Accelerated Insights

Mastering the Big Data Challenge in Cognitive Healthcare

Patrick Chain, genomics researcher at Los Alamos National Laboratory, posed a question in a recent blog: What if a nurse could swipe a patient’s saliva and run a quick genetic test to determine if the patient’s sore throat was caused by a cold virus or a bacterial infection? Read more…

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and the technology challenges ahead. These discussions happened in Read more…

By Alex R. Larzelere

Japan Meteorological Agency Takes Delivery of Pair of Crays

May 21, 2018

Cray has supplied two identical Cray XC50 supercomputers to the Japan Meteorological Agency (JMA) in northwestern Tokyo. Boasting more than 18 petaflops combine Read more…

By Tiffany Trader

ASC18: Final Results Revealed & Wrapped Up

May 17, 2018

It was an exciting week at ASC18 in Nanyang, China. The student teams braved extreme heat, extremely difficult applications, and extreme competition in order to cross the cluster competition finish line. The gala awards ceremony took place on Wednesday. The auditorium was packed with student teams, various dignitaries, the media, and other interested parties. So what happened? Read more…

By Dan Olds

Spring Meetings Underscore Quantum Computing’s Rise

May 17, 2018

The month of April 2018 saw four very important and interesting meetings to discuss the state of quantum computing technologies, their potential impacts, and th Read more…

By Alex R. Larzelere

Quantum Network Hub Opens in Japan

May 17, 2018

Following on the launch of its Q Commercial quantum network last December with 12 industrial and academic partners, the official Japanese hub at Keio University is now open to facilitate the exploration of quantum applications important to science and business. The news comes a week after IBM announced that North Carolina State University was the first U.S. university to join its Q Network. Read more…

By Tiffany Trader

Democratizing HPC: OSC Releases Version 1.3 of OnDemand

May 16, 2018

Making HPC resources readily available and easier to use for scientists who may have less HPC expertise is an ongoing challenge. Open OnDemand is a project by t Read more…

By John Russell

PRACE 2017 Annual Report: Exascale Aspirations; Industry Collaboration; HPC Training

May 15, 2018

The Partnership for Advanced Computing in Europe (PRACE) today released its annual report showcasing 2017 activities and providing a glimpse into thinking about Read more…

By John Russell

US Forms AI Brain Trust

May 11, 2018

Amid calls for a U.S. strategy for promoting AI development, the Trump administration is forming a senior-level panel to help coordinate government and industry research efforts. The Select Committee on Artificial Intelligence was announced Thursday (May 10) during a White House summit organized by the Office of Science and Technology Policy (OSTP). Read more…

By George Leopold

Emerging Advanced Scale Tech Trends Focus of Annual Tabor Conference

May 9, 2018

At Tabor Communications' annual Advanced Scale Forum (ASF) held this week in Austin, the focus was on enterprise adoption of HPC-class technologies and high performance data analytics (HPDA). It’s a confab that brings together end users (CIOs, IT planners, department heads) and vendors and encourages... Read more…

By the Editorial Team

MLPerf – Will New Machine Learning Benchmark Help Propel AI Forward?

May 2, 2018

Let the AI benchmarking wars begin. Today, a diverse group from academia and industry – Google, Baidu, Intel, AMD, Harvard, and Stanford among them – releas Read more…

By John Russell

How the Cloud Is Falling Short for HPC

March 15, 2018

The last couple of years have seen cloud computing gradually build some legitimacy within the HPC world, but still the HPC industry lies far behind enterprise I Read more…

By Chris Downing

Russian Nuclear Engineers Caught Cryptomining on Lab Supercomputer

February 12, 2018

Nuclear scientists working at the All-Russian Research Institute of Experimental Physics (RFNC-VNIIEF) have been arrested for using lab supercomputing resources to mine crypto-currency, according to a report in Russia’s Interfax News Agency. Read more…

By Tiffany Trader

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Deep Learning at 15 PFlops Enables Training for Extreme Weather Identification at Scale

March 19, 2018

Petaflop per second deep learning training performance on the NERSC (National Energy Research Scientific Computing Center) Cori supercomputer has given climate Read more…

By Rob Farber

AI Cloud Competition Heats Up: Google’s TPUs, Amazon Building AI Chip

February 12, 2018

Competition in the white hot AI (and public cloud) market pits Google against Amazon this week, with Google offering AI hardware on its cloud platform intended Read more…

By Doug Black

US Plans $1.8 Billion Spend on DOE Exascale Supercomputing

April 11, 2018

On Monday, the United States Department of Energy announced its intention to procure up to three exascale supercomputers at a cost of up to $1.8 billion with th Read more…

By Tiffany Trader

Lenovo Unveils Warm Water Cooled ThinkSystem SD650 in Rampup to LRZ Install

February 22, 2018

This week Lenovo took the wraps off the ThinkSystem SD650 high-density server with third-generation direct water cooling technology developed in tandem with par Read more…

By Tiffany Trader

Leading Solution Providers

HPC and AI – Two Communities Same Future

January 25, 2018

According to Al Gara (Intel Fellow, Data Center Group), high performance computing and artificial intelligence will increasingly intertwine as we transition to Read more…

By Rob Farber

Researchers Measure Impact of ‘Meltdown’ and ‘Spectre’ Patches on HPC Workloads

January 17, 2018

Computer scientists from the Center for Computational Research, State University of New York (SUNY), University at Buffalo have examined the effect of Meltdown Read more…

By Tiffany Trader

Google Chases Quantum Supremacy with 72-Qubit Processor

March 7, 2018

Google pulled ahead of the pack this week in the race toward "quantum supremacy," with the introduction of a new 72-qubit quantum processor called Bristlecone. Read more…

By Tiffany Trader

HPE Wins $57 Million DoD Supercomputing Contract

February 20, 2018

Hewlett Packard Enterprise (HPE) today revealed details of its massive $57 million HPC contract with the U.S. Department of Defense (DoD). The deal calls for HP Read more…

By Tiffany Trader

CFO Steps down in Executive Shuffle at Supermicro

January 31, 2018

Supermicro yesterday announced senior management shuffling including prominent departures, the completion of an audit linked to its delayed Nasdaq filings, and Read more…

By John Russell

Deep Learning Portends ‘Sea Change’ for Oil and Gas Sector

February 1, 2018

The billowing compute and data demands that spurred the oil and gas industry to be the largest commercial users of high-performance computing are now propelling Read more…

By Tiffany Trader

Nvidia Ups Hardware Game with 16-GPU DGX-2 Server and 18-Port NVSwitch

March 27, 2018

Nvidia unveiled a raft of new products from its annual technology conference in San Jose today, and despite not offering up a new chip architecture, there were still a few surprises in store for HPC hardware aficionados. Read more…

By Tiffany Trader

Hennessy & Patterson: A New Golden Age for Computer Architecture

April 17, 2018

On Monday June 4, 2018, 2017 A.M. Turing Award Winners John L. Hennessy and David A. Patterson will deliver the Turing Lecture at the 45th International Sympo Read more…

By Staff

  • arrow
  • Click Here for More Headlines
  • arrow
Share This