From Rendering LOTR to Animating HPC Clouds
Much has been written about the incredible animation work that went on behind the scenes of the Lord of the Rings series, but without the rendering horsepower of high performance systems, none of that would have been possible. It took an entire “supercomputing center” and unique cloud build-out for the team to continue—a process that led to the creation of Greenbutton—one of the few companies that identify as HPC cloud oriented.
In this audio interview, HPCwire editor, Nicole Hemsoth, talked with Scott Houston, the former CIO for WETA Digital, which provided the IT power infrastructure that powered the stunning visual effects for the films. Scott is currently the CEO of Greenbutton, which spun out of this work.
HPCwire: Scott, we talked back in 2011 at the Supercomputing conference, about many of your experiences on the Lord of the Rings films. Can you give us a sense of the background, what some of the decision making processes were when they were looking at different IT solutions?
Houston: It’s really been an interesting journey for me. In fact, back in 2003, the cloud really wasn’t available, so we literally had to go out and buy a thousand processors for one particular shot – it’s the battle of Pelennor Fields. I still have shivers up my spine every time I see that shot. It’s the shot that’s actually midway through the Return of the King, and it’s when the 14,000 rides of the Rohirrim (the horse warriors), clash on the battlefield with 83,000 orcs.
And the reality is that even though we had four data centers and clustered two and a half thousand processors, we literally had to go out and buy another thousand processors and build a brand new data center just to get that one shot done right at the end of the production of the movie. It was at that time, at the conclusion of the movie back in 2003, and early 2004 that I realized that there had to be a better way. And in fact, that infrastructure was sitting idle at the end of the movie. The next production for WETA was King Kong, so we didn’t really need those extra thousand processors.
So I formed a consortium called the New Zealand Supercomputer center, and we rented out time in that. We did some biotech work, we did some seismic processing, we helped render the movie Happy Feet. But the problem was for a number of our users, they just wanted to run small jobs, so that’s really where the genesis of green button was born. I saw an opportunity in the market to create some software to automate the process of using this capacity on demand, well before the cloud came around.
And that’s been great and a fantastic story to close this loop is that today we’re working with a company in Mexico that is rendering a full feature animation – a full length movie – that is rendered entirely on the cloud. So in the future CIOs and CTOs that are making movies and doing large computationally intensive projects, may not necessarily have to go out and buy that capacity in the future.
HPCwire: What is the status of the New Zealand Supercomputer center now? Are you running a balance between research/scientific workloads and entertainment/media workloads? Where does that stand?
Houston: That environment was shut down in 2009 and we’ve been cloud only since then. So today, we have jobs running on Windows Azure, Amazon, vCloud environments, and recently, earlier this year we announced support for Open Stack.
HPCwire: Since you’ve been heavy users from the beginning of both Amazon and Azure, what do you think some of the differences are – advantages and disadvantages between those platforms? Where is the advantage of Amazon over Azure, and Azure over Amazon?
Houston: Good question. We have customers, I think at last count in 77 different countries, so in many cases it just comes down to geography – where is the nearest data center and how fat can I have – you know – what is bandwidth of the pipe to get to that datacenter. So a number of those decisions are driven by where the nearest datacenter is.
Almost invariably the technical decisions – so from the processing point of view, from the support point of view…We do have workloads that run on Amazon and primarily one of them is a seismic processing on demand service that we had called Cloud Claritas, and that’s running on Amazon primarily because Amazon supports 10 gigabit Ethernet. The application is MPI-based. So that has driven that decision based on the technical requirements.
Sometimes it’s come down to memory. So there’s new large memory instances on Azure, now. Some of those workloads were running on Amazon, and some of those have been ported over to Azure. There’s a company that has just been setup called ProfitBricks, and they have InfiniBand support, and today not many other cloud providers have InfiniBand, so we’re starting to see some workloads running there.
To answer your question, often it will come down to the geographic location of the customer, and then also the technical performance, or the requirements of the workload.
HPCwire: If you’re talking about seismic processing for instance, not only is that computationally intensive, it’s data intensive. If you’re talking about using public cloud resources, I would imagine that the data movement costs would be pretty significant – how do you balance that out?
Houston: Good question. The reality is – and I’m embarrassed to say this – when we’re talking about moving 20 or 30 terabytes, which is a large scale 3D seismic processing run – we literally still ship the drives. You can’t beat the bandwidth of a Phoenix truck to be honest.
Interestingly enough, we’ve developed a product, which is part of our cloud fabric product called Cloud Sync. It’s a downloadable tool that in the background will move the data to the cloud. We just added an FTP capability that will use UDP protocols and do bulk upload. We’ve just done some recent testing, and we’re getting close to 2.4 gigabits per second. That’s theoretical performance, but that means I could technically, in the right environment, move a terabyte of data in under an hour, with the right connection from the customers site to the cloud. That may be – that will start to be a game changer.
I think we’re starting to see increased bandwidth into the datacenter – there are more cloud datacenters, and so I think that will be overcome. Not today, or tomorrow, or even this year, but over the next couple of years, I think the bandwidth problem will be solved. The whole concept of GreenButton is to push the green button from within your desktop application, and your job will start running. Well, clearly we need to get the data there, so Cloud Sync will manage that data synchronization in the background so that when you push the button, hopefully all of the files are there and we just do a quick synchronization and the job will start running.
HPCwire: That problem you mentioned is one that everyone is trying to tackle – it seems pretty tricky. The theoretical performance you just cited there is pretty impressive. What’s the actual performance based on some use cases?
Houston: Well, we’ll be launching the product later this month, so we’ll probably go back out. The reality is, it will clearly depend on what sort of connection the customer has from their network provider into the datacenter. Many of the datacenters have large pipes into them, so it’s actually going to come down to the performance from the customer to the datacenter. But we’ve just started working with a company that is putting in a dedicated pipe between their facility and their cloud provider, so I would think in a couple of months we will have some real world use cases.
HPCwire: That’s interesting. We’ll keep our eyes open for it. Let’s talk about GreenButton the company for a second. Let me sure I have this straight – you’re a platform as a service company dedicated specifically to HPC workloads. I know today you had an announcement around higher end analytics services. Can you describe the company’s focus in terms of the types of applications and needs that you’re serving specifically, and what is it that makes GreenButton unique and distinctly HPC oriented.
Houston: Often it depends on who we’re talking to. We’re in this growth stage, and HPC hasn’t been sexy. It’s always been sexy to me, of course, but it hasn’t been particularly sexy in the marketplace. The growth in the industry hasn’t been particularly investable, and I know that we have an HPC audience, but – The interesting thing for us is that we don’t just focus on rendering seismic, genomic sequencing, Monte Carlos, and CFD. A number of our customers – and we just had an announcement today – one of our largest customers is in the social media space. So really what we look at at Green Button is any type of workload, but it has to be computationally intensive.
Interestingly enough, we’re doing a lot of work on video indexing. So we’re taking an hour of video content in, and using key algorithms and processing that video and making that indexable. And that’s not a traditional HPC type workload, but it is a big compute workload. And that’s a product that we’ve created called inCus that uses an algorithm called Microsoft called MAVIS. So it’s not just traditional HPC, we’ve got social media customers, we’re processing video, yes we’re rendering movies and we’re doing seismic processing work, but we’ve also been putting some work into running Hadoop workloads and big data analytics through our GreenButton cloud fabric engine as well. So for us, it’s not just a particular vertical market, it’s anything that is computationally intensive, or big data intensive, we can take that workload, run it on GreenButton cloud fabric and deploy that workload to any cloud platform.
HPCwire: That comment that HPC’s definition is expanding beyond the research, scientific computing workloads – that’s getting more and more common as data intensive / computationally intensive sort of merge. But when it comes to cloud computing, especially if you’re using a public cloud resource, the concerns really don’t change – they get more intense actually whether it’s computationally intensive or data intensive – where there’s a performance gap, and so people that require high, high performance, have to really, I would imagine, take a very close look at whatever cloud service they’re going to use because they’re taking a pretty big hit in the virtualization side – data movement side. How do you help customers work through that, and how do you prove the ROI of this over just buying a bunch of infrastructure, which is a big up front cost.
Houston: I think that’s an ongoing process. The reality is, from early on, we’ve talk to customers, because they’ll run a job on their local server environment, then they’ll run it on the cloud and they’ll go, it runs slower. And there’s no denying that. One, we have to get the data up there, and there’s nothing like a dedicated environment that is running in their own datacenter.
That said, we’re seeing significant investment from the cloud providers in their infrastructures. So while there may never be parity in running the job offsite, from a high performance dedicated environment on site – from a financial perspective, it really does make sense. I think that the sort of workloads that we’re talking about, it would be a very brave CIO that’s says “I’m going all into the cloud.” I’ve been in the cloud for 10 years, and I’m not sure that I would say it in their position.
The reality is that the cloud makes a heck of a lot of sense to take specific workloads that either take a long time, or they’re very bursty – they’re only project based – and use the cloud to scale out the business. So I’d encourage IT managers to use the cloud to support either very intensive workloads that they really don’t have the bandwidth or capability to run in-house, and not necessarily just go all into the cloud. We’d love to talk to those folks, but I think we’re still really a couple years away from the cloud being all encompassing and customers not needing datacenters anymore.
To give you an example of that, a number of our customers are using Green Button to render their jobs. And I did say one of them rendered an entire movie, but most of our customers will have really hard shots that the render jobs will run often for a matter of days, and they’re able to take those jobs, run them offsite on the cloud, and continue using their data center more efficiently for smaller jobs or the jobs that they need a quick turnaround on.
HPCwire: To go back to some of the use cases we talked about earlier, you said you’re seeing a lot of growth on the social media side, on the analytics side, big data – whatever that encompasses – it’s very big – what are some of the emerging markets that you think balance computationally intensive workload needs with whatever advantages the cloud can offer. Where is the real hot growth right now?
Houston: it’s a little bit of both. We can’t talk about this customer publically, but they’re a Fortune 500 manufacturing customer – they have a large Monte Carlo simulation that was running in their internal cluster and taking 3 hours to run. We are able to take those workloads using Green Button Cloud Fabric, and spin up 1,100 processors and have that job completed at under 10 minutes. And that’s a traditional HPC workload, and just the power of our spinning up 100 processors and running that job and getting it turned around in 10 minutes is actually quite transformational for this customer.
We’re certainly seeing great growth in traditional HPC workloads – biotech, seismic, rendering. We’re starting to see CFD workloads – we have a customer in Germany that is doing fire dynamics simulation and that’s a traditional HPC MPI based workload and they’re running those jobs in the cloud.
But also balancing that are some really interesting opportunities around video and social media. The press release you’ll see on our website today is with a company called Tout, and effectively they provide 15 second video “tweets” (if you like) through the mobile phone, and they’re using Green Button to provide analytics for those customers – what sort of things they’re talking about, what are the themes, what are the ideas, what are the hot subjects.
So balancing out the traditional HPC workloads are a whole lot of new workloads that are computationally intensive but not viewed as traditional HPC.
HPCwire: In the case like the manufacturing one you cited – cloud pricing is already tricky enough to figure out for a lot of users – how does this layer factor into pricing. How do you work this out?
Houston: The good news is that it’s pretty darn cheap. This particular workload – just think about the economics of it – if I’m spinning up 1,100 processors, most cloud providers are charging by the minute, and I want to run a job, and I’m ten minutes. So that’s $20 or $30 dollars to run the job. So to take that turnaround from under 3 hours to 10 minutes, and it costs $20 or $30 dollars to run a job – and they’ll negotiate that with their cloud provider – that’s a pretty compelling return on investment or business case to make.
I think the challenge is for the cloud provider in that the processing is so darned attractive for the end user. The end user just pays for the compute and storage they use for a particular job. It is becoming more and more compelling for many customers. The challenge for the cloud provider is how will they make money out of that. And if you’ve seen what’s happening in the cloud, you’ve seen the transition of the SAAS type applications, or hosting websites, and you know the cloud makes good sense for that. Increasingly organizations are using the cloud for offsite storage, and that makes sense.
None of those are particularly profitable for the cloud providers. The real money and profit for the cloud providers is in on demand computational work. The challenge for them is around utilization. If they can ensure their cloud is fully utilized, or even 80% utilized, it’s a profitable business, so how do you attract enough customers to keep that utilization. And it’s a win-win if you can do it from a customer point of view – from only using the cloud 10 or 20 or 30 percent of my time – it just makes sense. In fact the analytics that we’re doing right now with current cloud pricing, it’s almost 50%. If I built a datacenter and I’m running at less than 50% of the time, depending on my licensing costs, it may well be cheaper to run those jobs on the cloud with the current pricing model. So that’s an interesting tipping point that I’m sure most CIOs and CFOs are considering at the moment.
HPCwire: Right. So as with everything in high end infrastructure, it all depends.
Houston: I don’t think you want to argue that the pricing is compelling. The biggest problem, and the problem that we’ve been really focused on is governance. Anybody that is running their jobs in the cloud – hopefully they’re having a good experience. What is a really bad experience for people is not the costs, but where to appoint the costs.
If you get a bill from a cloud provider today – and it doesn’t matter who it is, whether it’s Amazon or Microsoft or anybody – it’s like a phone bill. You’ve got no idea who ran the jobs, what department they’re working or what project that was. Did they have the authorization to run the jobs? I’m sure most organizations are running their jobs on the cloud and they’re using the company credit card.
So for an IT department and a finance department, wrangling with those costs and apportioning that to a user, to a project, to a department – it’s an absolute nightmare; so we’ve been working very hard at providing governance. One of the things that we’ve just recently been granted a patent on it, is our ability to profile a job and provide an SLA commitment in time and cost. We think that’s unique and I think that’s going to be the game changer and perhaps the tipping point for broader utilization in the cloud is how long will it take, how much will it cost, and then really apportioning that to the correct department, user, project.
Feeds by Topic
- Developer Tools
Feeds by Industry
October 25, 2016
- Fujitsu Adopts Cadence Palladium Z1 Enterprise Emulation Platform for Post-K Supercomputer Development
- Kitware Plans to Spotlight New VTK and ParaView Releases at SC16
- Seagate Launches New Enterprise HDD
- SES Institutions Awarded £12 Million to Develop University Tier 2, HPC Facilities
October 24, 2016
- Tabor Communications Launches New Corporate Site, Signals Publishing Platform Enhancements Ahead
- Barcelona Supercomputing Center Joins the OpenPOWER Foundation
- SC16 Releases Latest Invited Talk Spotlight: Charlie Catlett
- Intel Capital Announces $38M of New Investments
- Bright Computing to Present on Ceph at OpenStack Summit, Barcelona
October 21, 2016
- National Cancer Institute’s Dr. Warren Kibbe Joins HPC Matters Plenary Panel on Precision Medicine
- ORNL Receives Emu Chick Memory Server From Emu Technology
- SC16 Releases Latest Invited Talk Spotlight: Dr. Thomas Schulthess
- AMD Reports 2016 Third Quarter Results
October 20, 2016
- ISC 2017 Seeking Contributors for Conference Program
- GCS Supports Two German Teams for SC16 Student Cluster Competition
- Kitware Prepares SC16 Presentation on In Situ Simulation Run
- NCSA Collaboration Helps Discover New Dwarf Planet
- U.S. EPA Announced as Newest Member of National Consortium for Data Science
- NSF Awards $5.9M to Broaden Participation in Academic Workforce
October 19, 2016
Most Read Features
- AWS Beats Azure to K80 General Availability
- Vectors: How the Old Became New Again in Supercomputing
- Dell EMC Engineers Strategy to Democratize HPC
- IBM Advances Neuromorphic Computing for Deep Learning
- Container App ‘Singularity’ Eases Scientific Computing
- Power8 with NVLink Coming to the Nimbix Cloud
- OpenCAPI Takes on PCIe, Vows 10X Improvement
- Beyond von Neumann, Neuromorphic Computing Steadily Advances
- Gen-Z Consortium Puts New High Performance Interconnect in Motion
- New Genomics Pipeline Combines AWS, Local HPC, and Supercomputing
- More Features…
Most Read Short Takes
- Intel’s FPGAs Target Datacenters, Networking
- CPU Benchmarking: Haswell Versus POWER8
- Cray KNL-Based XC40 Shines on STAC–A2 Benchmark
- SGI, ANSYS Set New Record for Scaling Commercial CAE Code
- NSF Backs ‘Big Data Spokes’ with $10M in Grants
- Microsoft Eyes AI Supercomputer on Azure
- The Exascale Computing Project Awards $39.8M to 22 Projects
- Researchers Shrink Transistor Gate to One Nanometer
- Micron, Intel Prepare to Launch 3D XPoint Memory
- HPC Career Notes (Oct. 2016)
- More Short Takes…
Most Read Off The Wire
- NYU Using NVIDIA DGX-1 Supercomputer to Push Boundaries of AI
- Nimbix Collaborates With IBM and NVIDIA to Launch Powerful GPU Cloud Offering
- ARM Releases New Interconnect Technology
- SGI Introduces Scale-Out Solution for SAP HANA
- CSIRO Deploys APAC’s First NVIDIA DGX-1 Deep Learning Supercomputers
- NVIDIA Introduces Xavier
- UK Met Office Installs New HPC System With Help From Bright Computing, SGI and DDN
- ALCF Summer Student Projects Tackle Real-World Problems
- SAIC Awarded $575M Contract to Provide Support Services to the HPCMP
- CSRA Expands Supercomputing Cluster for NIH Research
- More Off The Wire…
- Read more…
HPC container platform Singularity is just six months out from its 1.0 release but already is making inroads across the HPC research landscape. Read more…
- Read more…
Here at HPCwire, we aim to keep the HPC community apprised of the most relevant and interesting news items that get tweeted throughout the week. Read more…
Developing energy efficient computing is a high priority worldwide, particularly as the race towards exascale computing heats up. Read more…
Continued jostling among big HPC suppliers, ascent of the industrial segment as a market driver, and a projected total HPC market growth (CAGR) of around 5.2 percent pushing revenues to $36.9 billion by 2020 are a few highlights from the most recent Intersect360 Research report. Read more…
Between the demands of the data deluge and hardware advancements in both CPUs and GPUs alike, it’s no surprise that large HPC clusters are seeing rapid growth as a part of today’s Big Data escalation. Read more…
Storage challenges in the data deluge are nothing new, but as we prepare for unprecedented information overload, the demand innovative storage strategies is at an all-time high. Read more…
Creating the right technology environment is a time-consuming task for researchers who want to focus on science (not servers or long wait times at supercomputing centers). Read more…
The Van Andel Institute (VAI) worked with Silicon Mechanics to not only provide its users a more powerful platform, but a hybrid OpenStack HPC solution with the flexibility to support VAI’s commitment to improve the health and change the lives of current and future generations. Read more…
HPC Job Bank
November 1 - November 2Jakarta Indonesia
November 7 - November 9San Francisco CA United States
November 13 - November 17Salt Lake City UT United States
December 6 - December 7San Jose CA United States
March 15, 2017 - March 16, 2017Houston TX United States