From Rendering LOTR to Animating HPC Clouds
Much has been written about the incredible animation work that went on behind the scenes of the Lord of the Rings series, but without the rendering horsepower of high performance systems, none of that would have been possible. It took an entire “supercomputing center” and unique cloud build-out for the team to continue—a process that led to the creation of Greenbutton—one of the few companies that identify as HPC cloud oriented.
In this audio interview, HPCwire editor, Nicole Hemsoth, talked with Scott Houston, the former CIO for WETA Digital, which provided the IT power infrastructure that powered the stunning visual effects for the films. Scott is currently the CEO of Greenbutton, which spun out of this work.
HPCwire: Scott, we talked back in 2011 at the Supercomputing conference, about many of your experiences on the Lord of the Rings films. Can you give us a sense of the background, what some of the decision making processes were when they were looking at different IT solutions?
Houston: It’s really been an interesting journey for me. In fact, back in 2003, the cloud really wasn’t available, so we literally had to go out and buy a thousand processors for one particular shot – it’s the battle of Pelennor Fields. I still have shivers up my spine every time I see that shot. It’s the shot that’s actually midway through the Return of the King, and it’s when the 14,000 rides of the Rohirrim (the horse warriors), clash on the battlefield with 83,000 orcs.
And the reality is that even though we had four data centers and clustered two and a half thousand processors, we literally had to go out and buy another thousand processors and build a brand new data center just to get that one shot done right at the end of the production of the movie. It was at that time, at the conclusion of the movie back in 2003, and early 2004 that I realized that there had to be a better way. And in fact, that infrastructure was sitting idle at the end of the movie. The next production for WETA was King Kong, so we didn’t really need those extra thousand processors.
So I formed a consortium called the New Zealand Supercomputer center, and we rented out time in that. We did some biotech work, we did some seismic processing, we helped render the movie Happy Feet. But the problem was for a number of our users, they just wanted to run small jobs, so that’s really where the genesis of green button was born. I saw an opportunity in the market to create some software to automate the process of using this capacity on demand, well before the cloud came around.
And that’s been great and a fantastic story to close this loop is that today we’re working with a company in Mexico that is rendering a full feature animation – a full length movie – that is rendered entirely on the cloud. So in the future CIOs and CTOs that are making movies and doing large computationally intensive projects, may not necessarily have to go out and buy that capacity in the future.
HPCwire: What is the status of the New Zealand Supercomputer center now? Are you running a balance between research/scientific workloads and entertainment/media workloads? Where does that stand?
Houston: That environment was shut down in 2009 and we’ve been cloud only since then. So today, we have jobs running on Windows Azure, Amazon, vCloud environments, and recently, earlier this year we announced support for Open Stack.
HPCwire: Since you’ve been heavy users from the beginning of both Amazon and Azure, what do you think some of the differences are – advantages and disadvantages between those platforms? Where is the advantage of Amazon over Azure, and Azure over Amazon?
Houston: Good question. We have customers, I think at last count in 77 different countries, so in many cases it just comes down to geography – where is the nearest data center and how fat can I have – you know – what is bandwidth of the pipe to get to that datacenter. So a number of those decisions are driven by where the nearest datacenter is.
Almost invariably the technical decisions – so from the processing point of view, from the support point of view…We do have workloads that run on Amazon and primarily one of them is a seismic processing on demand service that we had called Cloud Claritas, and that’s running on Amazon primarily because Amazon supports 10 gigabit Ethernet. The application is MPI-based. So that has driven that decision based on the technical requirements.
Sometimes it’s come down to memory. So there’s new large memory instances on Azure, now. Some of those workloads were running on Amazon, and some of those have been ported over to Azure. There’s a company that has just been setup called ProfitBricks, and they have InfiniBand support, and today not many other cloud providers have InfiniBand, so we’re starting to see some workloads running there.
To answer your question, often it will come down to the geographic location of the customer, and then also the technical performance, or the requirements of the workload.
HPCwire: If you’re talking about seismic processing for instance, not only is that computationally intensive, it’s data intensive. If you’re talking about using public cloud resources, I would imagine that the data movement costs would be pretty significant – how do you balance that out?
Houston: Good question. The reality is – and I’m embarrassed to say this – when we’re talking about moving 20 or 30 terabytes, which is a large scale 3D seismic processing run – we literally still ship the drives. You can’t beat the bandwidth of a Phoenix truck to be honest.
Interestingly enough, we’ve developed a product, which is part of our cloud fabric product called Cloud Sync. It’s a downloadable tool that in the background will move the data to the cloud. We just added an FTP capability that will use UDP protocols and do bulk upload. We’ve just done some recent testing, and we’re getting close to 2.4 gigabits per second. That’s theoretical performance, but that means I could technically, in the right environment, move a terabyte of data in under an hour, with the right connection from the customers site to the cloud. That may be – that will start to be a game changer.
I think we’re starting to see increased bandwidth into the datacenter – there are more cloud datacenters, and so I think that will be overcome. Not today, or tomorrow, or even this year, but over the next couple of years, I think the bandwidth problem will be solved. The whole concept of GreenButton is to push the green button from within your desktop application, and your job will start running. Well, clearly we need to get the data there, so Cloud Sync will manage that data synchronization in the background so that when you push the button, hopefully all of the files are there and we just do a quick synchronization and the job will start running.
HPCwire: That problem you mentioned is one that everyone is trying to tackle – it seems pretty tricky. The theoretical performance you just cited there is pretty impressive. What’s the actual performance based on some use cases?
Houston: Well, we’ll be launching the product later this month, so we’ll probably go back out. The reality is, it will clearly depend on what sort of connection the customer has from their network provider into the datacenter. Many of the datacenters have large pipes into them, so it’s actually going to come down to the performance from the customer to the datacenter. But we’ve just started working with a company that is putting in a dedicated pipe between their facility and their cloud provider, so I would think in a couple of months we will have some real world use cases.
HPCwire: That’s interesting. We’ll keep our eyes open for it. Let’s talk about GreenButton the company for a second. Let me sure I have this straight – you’re a platform as a service company dedicated specifically to HPC workloads. I know today you had an announcement around higher end analytics services. Can you describe the company’s focus in terms of the types of applications and needs that you’re serving specifically, and what is it that makes GreenButton unique and distinctly HPC oriented.
Houston: Often it depends on who we’re talking to. We’re in this growth stage, and HPC hasn’t been sexy. It’s always been sexy to me, of course, but it hasn’t been particularly sexy in the marketplace. The growth in the industry hasn’t been particularly investable, and I know that we have an HPC audience, but – The interesting thing for us is that we don’t just focus on rendering seismic, genomic sequencing, Monte Carlos, and CFD. A number of our customers – and we just had an announcement today – one of our largest customers is in the social media space. So really what we look at at Green Button is any type of workload, but it has to be computationally intensive.
Interestingly enough, we’re doing a lot of work on video indexing. So we’re taking an hour of video content in, and using key algorithms and processing that video and making that indexable. And that’s not a traditional HPC type workload, but it is a big compute workload. And that’s a product that we’ve created called inCus that uses an algorithm called Microsoft called MAVIS. So it’s not just traditional HPC, we’ve got social media customers, we’re processing video, yes we’re rendering movies and we’re doing seismic processing work, but we’ve also been putting some work into running Hadoop workloads and big data analytics through our GreenButton cloud fabric engine as well. So for us, it’s not just a particular vertical market, it’s anything that is computationally intensive, or big data intensive, we can take that workload, run it on GreenButton cloud fabric and deploy that workload to any cloud platform.
HPCwire: That comment that HPC’s definition is expanding beyond the research, scientific computing workloads – that’s getting more and more common as data intensive / computationally intensive sort of merge. But when it comes to cloud computing, especially if you’re using a public cloud resource, the concerns really don’t change – they get more intense actually whether it’s computationally intensive or data intensive – where there’s a performance gap, and so people that require high, high performance, have to really, I would imagine, take a very close look at whatever cloud service they’re going to use because they’re taking a pretty big hit in the virtualization side – data movement side. How do you help customers work through that, and how do you prove the ROI of this over just buying a bunch of infrastructure, which is a big up front cost.
Houston: I think that’s an ongoing process. The reality is, from early on, we’ve talk to customers, because they’ll run a job on their local server environment, then they’ll run it on the cloud and they’ll go, it runs slower. And there’s no denying that. One, we have to get the data up there, and there’s nothing like a dedicated environment that is running in their own datacenter.
That said, we’re seeing significant investment from the cloud providers in their infrastructures. So while there may never be parity in running the job offsite, from a high performance dedicated environment on site – from a financial perspective, it really does make sense. I think that the sort of workloads that we’re talking about, it would be a very brave CIO that’s says “I’m going all into the cloud.” I’ve been in the cloud for 10 years, and I’m not sure that I would say it in their position.
The reality is that the cloud makes a heck of a lot of sense to take specific workloads that either take a long time, or they’re very bursty – they’re only project based – and use the cloud to scale out the business. So I’d encourage IT managers to use the cloud to support either very intensive workloads that they really don’t have the bandwidth or capability to run in-house, and not necessarily just go all into the cloud. We’d love to talk to those folks, but I think we’re still really a couple years away from the cloud being all encompassing and customers not needing datacenters anymore.
To give you an example of that, a number of our customers are using Green Button to render their jobs. And I did say one of them rendered an entire movie, but most of our customers will have really hard shots that the render jobs will run often for a matter of days, and they’re able to take those jobs, run them offsite on the cloud, and continue using their data center more efficiently for smaller jobs or the jobs that they need a quick turnaround on.
HPCwire: To go back to some of the use cases we talked about earlier, you said you’re seeing a lot of growth on the social media side, on the analytics side, big data – whatever that encompasses – it’s very big – what are some of the emerging markets that you think balance computationally intensive workload needs with whatever advantages the cloud can offer. Where is the real hot growth right now?
Houston: it’s a little bit of both. We can’t talk about this customer publically, but they’re a Fortune 500 manufacturing customer – they have a large Monte Carlo simulation that was running in their internal cluster and taking 3 hours to run. We are able to take those workloads using Green Button Cloud Fabric, and spin up 1,100 processors and have that job completed at under 10 minutes. And that’s a traditional HPC workload, and just the power of our spinning up 100 processors and running that job and getting it turned around in 10 minutes is actually quite transformational for this customer.
We’re certainly seeing great growth in traditional HPC workloads – biotech, seismic, rendering. We’re starting to see CFD workloads – we have a customer in Germany that is doing fire dynamics simulation and that’s a traditional HPC MPI based workload and they’re running those jobs in the cloud.
But also balancing that are some really interesting opportunities around video and social media. The press release you’ll see on our website today is with a company called Tout, and effectively they provide 15 second video “tweets” (if you like) through the mobile phone, and they’re using Green Button to provide analytics for those customers – what sort of things they’re talking about, what are the themes, what are the ideas, what are the hot subjects.
So balancing out the traditional HPC workloads are a whole lot of new workloads that are computationally intensive but not viewed as traditional HPC.
HPCwire: In the case like the manufacturing one you cited – cloud pricing is already tricky enough to figure out for a lot of users – how does this layer factor into pricing. How do you work this out?
Houston: The good news is that it’s pretty darn cheap. This particular workload – just think about the economics of it – if I’m spinning up 1,100 processors, most cloud providers are charging by the minute, and I want to run a job, and I’m ten minutes. So that’s $20 or $30 dollars to run the job. So to take that turnaround from under 3 hours to 10 minutes, and it costs $20 or $30 dollars to run a job – and they’ll negotiate that with their cloud provider – that’s a pretty compelling return on investment or business case to make.
I think the challenge is for the cloud provider in that the processing is so darned attractive for the end user. The end user just pays for the compute and storage they use for a particular job. It is becoming more and more compelling for many customers. The challenge for the cloud provider is how will they make money out of that. And if you’ve seen what’s happening in the cloud, you’ve seen the transition of the SAAS type applications, or hosting websites, and you know the cloud makes good sense for that. Increasingly organizations are using the cloud for offsite storage, and that makes sense.
None of those are particularly profitable for the cloud providers. The real money and profit for the cloud providers is in on demand computational work. The challenge for them is around utilization. If they can ensure their cloud is fully utilized, or even 80% utilized, it’s a profitable business, so how do you attract enough customers to keep that utilization. And it’s a win-win if you can do it from a customer point of view – from only using the cloud 10 or 20 or 30 percent of my time – it just makes sense. In fact the analytics that we’re doing right now with current cloud pricing, it’s almost 50%. If I built a datacenter and I’m running at less than 50% of the time, depending on my licensing costs, it may well be cheaper to run those jobs on the cloud with the current pricing model. So that’s an interesting tipping point that I’m sure most CIOs and CFOs are considering at the moment.
HPCwire: Right. So as with everything in high end infrastructure, it all depends.
Houston: I don’t think you want to argue that the pricing is compelling. The biggest problem, and the problem that we’ve been really focused on is governance. Anybody that is running their jobs in the cloud – hopefully they’re having a good experience. What is a really bad experience for people is not the costs, but where to appoint the costs.
If you get a bill from a cloud provider today – and it doesn’t matter who it is, whether it’s Amazon or Microsoft or anybody – it’s like a phone bill. You’ve got no idea who ran the jobs, what department they’re working or what project that was. Did they have the authorization to run the jobs? I’m sure most organizations are running their jobs on the cloud and they’re using the company credit card.
So for an IT department and a finance department, wrangling with those costs and apportioning that to a user, to a project, to a department – it’s an absolute nightmare; so we’ve been working very hard at providing governance. One of the things that we’ve just recently been granted a patent on it, is our ability to profile a job and provide an SLA commitment in time and cost. We think that’s unique and I think that’s going to be the game changer and perhaps the tipping point for broader utilization in the cloud is how long will it take, how much will it cost, and then really apportioning that to the correct department, user, project.
Feeds by Topic
- Developer Tools
Feeds by Industry
September 27, 2016
- SC16 Releases Latest Invited Talk Spotlight: Dr. Thomas Sterling
- ARM Releases New Interconnect Technology
- Seagate Showcases Latest Enterprise SSD and HDD Portfolios at Open Data Center Summit 2016
- Internet2 Seeks Community Input on Next Generation R&E Network
- TACC Cloud Computing Testbed Assists PSC
- Mellanox Introduces Windows Server 2016 Storage Solution
September 26, 2016
- Internet2 Announces 2016 Technology Exchange Gender Diversity Award Recipients
- University of Tokyo and JCAHPC to Deploy DDN’s IME14K for New Systems
- Quantum Provides Petascale Data Storage and Management for Major European Research Institutions
- Earlham Institute Partners With Verne Global to Enhance Research
- SC16 On-Site Childcare Registration Now Open
September 23, 2016
- Minimal Metrics Announces Early-Adopter Program for PerfMiner
- NCSA Releases Profile on Blue Waters Graduate Fellow Erin Teich
September 22, 2016
- CENATE Adds Powerful Testbed From NVIDIA
- Volkswagen Innovates Manufacturing With Move to Verne Global
- NASA Career Award Winner Uses Blue Waters Supercomputer to Mine Crop Yield Data
- Supermicro and DataCore Partner to Create Hyper-Converged Solution
- SC16 Releases Latest Invited Talk Spotlight: Dr. Thomas Theis
September 21, 2016
- SC16 Announces Best Paper Nominees
- TYAN HPC Platforms Add Support for NVIDIA Tesla P100, P40 and P4 GPUs
Most Read Features
- IBM Advances Against x86 with Power9
- IBM Debuts Power8 Chip with NVLink and 3 New Systems
- Japan’s Post-K Computer Hits 1-2 Year Speed Bump
- Intel’s Fryman: “It’s not that we love CMOS; it’s the only real choice.”
- Why 2016 Is the Most Important Year in HPC in Over Two Decades
- New Genomics Pipeline Combines AWS, Local HPC, and Supercomputing
- Deep Learning Paves Way for Better Diagnostics
- AMD, GlobalFoundries Renew Vows, Focus on Path to 7nm
- Nvidia Launches Pascal GPUs for Deep Learning Inferencing
- Think Fast – Is Neuromorphic Computing Set to Leap Forward?
- More Features…
Most Read Short Takes
- The Exascale Computing Project Awards $39.8M to 22 Projects
- CPU Benchmarking: Haswell Versus POWER8
- MIT Programmers Attack Big Data Memory Gap
- Micron, Intel Prepare to Launch 3D XPoint Memory
- How Would U.S. Perform in Coding Olympics? Not Great Says Study
- In Search of the Next Circuit Building Method – ORNL has Ideas
- DOE Invests $16M in Supercomputer Technology to Advance Material Sciences
- Jetstream, the NSF-funded “Science” Cloud, Launches Tomorrow
- New Approach to Computationally Designing Drugs for GPCRs
- On-Chip Device Accelerates Core-to-Core Communication
- More Short Takes…
Most Read Off The Wire
- Seven Women in IT Chosen to Help Build and Operate SCinet at SC16
- SC16 Announces Best Paper Nominees
- Rackspace Acquired by Apollo Global for $4.3 Billion
- Juniper Networks to Provide Networking Infrastructure for New NCAR Supercomputer
- Supermicro Unveils NVIDIA Pascal GPU-Enabled Server Offerings
- Berkeley Lab to Lead Two DOE Exascale Computing Proposals, Support Four Others
- Larry Smarr’s “50 Years of Supercomputing” Talk to be Live Streamed
- SDSC Supercomputer Modeling Reveals Acrobatics of CRISPR-Cas9 Technology
- Newly Funded Project Sets Stage for Next Generation of Supercomputers
- Dell Completes Acquisition of EMC
- More Off The Wire…
Vector instructions, once a powerful performance innovation of supercomputing in the 1970s and 1980s became an obsolete technology in the 1990s. Read more…
Declining DNA sequencing costs and the rush to do whole genome sequencing (WGS) of large cohort populations – think 5000 subjects now, but many more thousands soon – presents a formidable computational challenge to researchers attempting to make sense of large cohort datasets. Read more…
- Read more…
SGI, the supercomputing vendor recently acquired by HPE, has teamed with ANSYS, the product engineering and simulation software company, to set a new world record for scaling commercial CAE code. Read more…
Here at HPCwire, we aim to keep the HPC community apprised of the most relevant and interesting news items that get tweeted throughout the week. Read more…
How best to foster academic-industry collaboration is a hot topic. Read more…
Storage challenges in the data deluge are nothing new, but as we prepare for unprecedented information overload, the demand innovative storage strategies is at an all-time high. Read more…
Today’s leading organizations are dealing with larger data sets, higher volume and disparate data sources, and the need for faster insights. Read more…
Creating the right technology environment is a time-consuming task for researchers who want to focus on science (not servers or long wait times at supercomputing centers). Read more…
The Van Andel Institute (VAI) worked with Silicon Mechanics to not only provide its users a more powerful platform, but a hybrid OpenStack HPC solution with the flexibility to support VAI’s commitment to improve the health and change the lives of current and future generations. Read more…