Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderella journey the last three years. It closed out 2015 with record revenue of $725 million and went into 2016 with even higher expectations. However, market sluggishness, chip delays and a mid-year smoke event led Cray to lower 2016 guidance from around $825 million to the $650 million range. Cray still managed to end the year on a high note; it recorded the largest revenue quarter in the company’s history and its second-highest revenue year, just shy of $630 million. This year the company is projecting revenue in the $400 million range and has not yet provided guidance beyond 2017.

During our 45-minute interview, Pete, who has been at the helm of the Seattle-based supercomputing company for the last 12 years, provides a candid perspective on the factors behind the market slowdown, whether Cray can sustain this flat period and why he believes ultimately in the long-term health of the industry that is Cray’s mainstay: high-end supercomputing. We also explore the wave of activities and trends that are creating market momentum, including exascale, commercial HPC, AI and deep learning advances.

When we spoke with Pete, the details of the agreement with Seagate to transfer the ClusterStor assets to Cray were still being ironed out so we did not get into the specifics of that arrangement. That deal has since closed and Cray Storage VP John Howarth briefed us on the details and what it means for Cray – you can read that interview here.

In the interval since we spoke, Cray also announced three big customer wins: a Cray XC40 install at JAIST, an XC50-AC supercomputer at Japan’s Yokohama City University and $48 million contract from the Korea Institute of Science and Technology Information for a CS500 cluster supercomputer. These wins reflect the success Cray is having in the Asian market, which Pete mentions below.

Tiffany Trader: Cray has had quite a journey over the last 12-18 months – there was the smoke event a year ago at one of the Chippewa Falls factories, the fall-off in revenue for 2016 and the overall supercomputing market slowdown — can you give us your perspective on the events and market pressures of the last year, year-and-a-half and their impact on Cray?

Pete Ungaro: It’s been an absolutely crazy last couple of years.  In 2016, we had just come off a run where we had tripled our revenues over the preceding four years, so we were on an amazing growth path in our market and had really built the company up to be profitable on a sustained basis pretty much consistently every year.  Then we got a curve ball thrown at us. As you mentioned, we had the “smoke-event” at one of our two manufacturing facilities in Chippewa Falls, which ended up taking our production down for about three or four weeks and caused smoke damage to a few customer machines, and we had to work hard to get through that. Then about the same time, around the summer of 2016, our primary market, which is the high-end of the supercomputing market, was really starting to go into a significant slowdown. We spent a lot of time looking at what was happening in our market, as we always do, and based on the data we’ve been able to gather, the high-end of the supercomputing market, which we count as systems that are 5 to 10 million dollars and higher, dropped over 25 percent in 2016 and has continued to drop in 2017. At the same time, what’s been really interesting as you look through that market data, is that even though the market is down and our revenues are down, when we look at how Cray did in the market during that time period, we’re doing really well.  And that has continued in 2017.  Our win rates are up; we’re gaining share in the high end, but the market overall is down, so it’s muting our success even though we’re still doing well competitively in the market overall. I think in ‘16, our market share at the high-end of supercomputing was down roughly 16 percent and the market was down by over 25 percent, so we are gaining share in a declining market. So, it’s been an interesting ride for us over the last couple years.

Tom Tabor: So Pete, can Cray sustain this flat period of high-end sales and if so how?

Pete: I’m confident we can. The company is in a very solid financial position — we have a strong balance sheet, and we have no debt on the books. So, we can definitely sustain our business during this slower period. We brought our expenses down a little bit to account for that this year, but we’re also continuing to invest in strategic growth areas, such as artificial intelligence and deep learning. We just did a transaction with Seagate on the ClusterStor line, so we’re still out there fighting and I think doing a great job from a market competitiveness position.

Tiffany: We have heard you emphasize that there is a softer than normal market – what are the factors that you are looking at there? On the Q2 earnings call, you said: “This is the longest and deepest market downturn I have seen in my 25 years of working in this industry.” At the same time, we’re seeing really bullish market forecasts from HPC analysts — Hyperion Research is projecting $30 billion and Intersect360 $44 billion for 2021. Admittedly those figures are for HPC at large, but how do you square this?

Pete: For the overall HPC market, it depends on who you talk to, how they’re measuring it, and what they’re counting. For the core of our business, we focus not just on the supercomputing market, but the really high-end of it. Hyperion, for instance, they count $500,000 machines as supercomputers. While we have systems in the $500,000 range, the vast majority of our revenues are systems that are in the $5 million to $10 million plus range and that’s really our core market. If you look at the worldwide market, a big part of where we don’t operate, by our choice, is in China, and China is by far the fastest growing supercomputing market. I totally agree with Hyperion, as well as Intersect360, that the market overall is going to grow. You see most of that growth in smaller systems, $250,000 to a million or two, and that’s where a huge part of the growth is, but that’s not our strongest part of the market. We have systems that play in that space, but it’s not driving a large amount of revenue for us, so it’s really that subset of the market, our addressable market, that is slow right now. But I do agree that, overall, the HPC market and especially the high-end supercomputing space is going to rebound and grow over time.

There’s all this excitement with the exascale programs going on around the world, and some of those are locked into different countries and different technologies and such that could limit where things are, but mostly those dollars aren’t really flowing into 2017 and 2018. They are out well beyond that, in the 2020-2023 timeframe in funding, so that’s another reason I feel the market is going to come back, but it’s going to be a slower return as it comes back and will take time.

Tom: At a high-level we understand the challenges, but can you can you chop it up for us? Tell us why it’s soft and where is it soft? Is it soft across the board – the federal initiatives, the industrial sector – are they redirecting spending to AI and data analytics? Is cloud encroaching on traditional HPC?

Pete:  If you look at the high-end of the supercomputing market, the slowdown is pretty broad across the market, but for different reasons in different segments, so it really comes down to what’s going on with customer behavior. Generally, we’re seeing customers holding on to their systems longer, really across the board, and this is partly due to the rate of change in processor performance slowing down recently. And so, we’re seeing customers not upgrading systems quite as often as they have been. On the government side of our business, there’s clearly a lot of budget uncertainties in the shorter term and some leadership changes at federal governments around the world – the new administration in the U.S.; Brexit in the UK is a huge change and the UK is one of the top four or five countries from a revenue perspective. There is also a lot of uncertainty in the EU right now. We’re seeing an across-the-board slowdown in spending on the government side as well as on the higher education side of our business. On the commercial side, our biggest market has been the energy segment with oil and gas companies. Clearly, with the price of oil being down, CAPEX spending has slowed as a part of that.

But one of the things you mentioned that I do think is really important is, “is there something else picking up these cycles that high-end machines used to do?” and I don’t believe that there is. I am very certain that cloud is not very competitive with high-end systems today based on architectures that you can get out in the cloud today and the fundamental technology factors around our tightly-integrated systems versus what you can go out and get on the cloud. We also don’t see AI and analytics technologies taking away from what’s going on in high-end supercomputing, in fact we see them as complementary and growing over time.

From a country perspective, we break up our market into four regions: the Americas, EMEA (Europe, Middle East and Africa), we have Japan as a separate region, and then Asia Pacific as a fourth region. We’ve seen the biggest weakness in Americas and EMEA, and we’ve seen that Asia-Pacific is doing great right now, but it’s a smaller segment and a smaller market, but it’s been really strong. It’s different by country, by industry and by customer, so it’s not one thing we can point to, it’s a wide set of different factors all hitting at the same time that is causing the current market weakness.

Tom: When do you think the market will return to a level point where it’s normal in terms of HPC deployment and HPC acquisitions, not just in EMEA and the US but globally?

Pete: Tom, if you can tell me when the market will come back, I will pay you. I will owe you big! It’s tough to know exactly when the market is going to return, but I am confident that the market will return. I come from a sales background and I spend a lot of time with customers around the world. One of the biggest topics of conversation is exactly this, and I remain convinced that because of the unique things that these high-end supercomputers can do, customers are going to continue to rely on high-end supercomputers and that is what is going to make the market return. A number of factors have all hit at one time, and that’s creating a larger than normal dip in the market. The high-end market has always had its ups and downs, but as I have said, it’s the longest and the deepest that I’ve seen in 25 years. I am confident that the market is going to return over time, and because we continue to invest in R&D at Cray and we’re not slowing down on that one iota, I think we’re going to emerge from this stronger than ever.

Tiffany: Fair enough the “crystal ball” question, what about your revenue in terms of traditional HPC versus the commercial sector? How is that shaking down?

Pete: The vast majority of our revenue is at the high-end of the supercomputing market — the systems that are $5 million or $10 million, or even more than that. We also have a growing presence in the storage market, and with our ClusterStor transaction, we expect it to grow even more.  The numbers you were quoting earlier — the market growing to $40 billion — that includes storage, software and services.  In terms of analytics and AI, machine learning, deep learning, those are smaller segment for us.

Commercial has been a big growth area for us. We’ve gone from virtually zero commercial sales when I first came to Cray to being more than 15 percent of our revenue in 2015. It’s taken a step back from that over the past couple of years because of the energy market, but we expect that to grow again over time. Ultimately, our goal is to drive about a third of our revenue from commercial customers and two-thirds from government and higher education customers.

Overall, we have always talked about trying to grow twice as fast as the broader HPC market. So as that market has grown over the past few years, outside of the last couple, we’ve had really nice growth.  We actually grew three times the market rate and we were taking share along the way. I feel like we have the same opportunity. Our biggest investment that we make by far is in R&D, and one of the things I’m most proud of is that we’ve been able to continue to grow our R&D investment over the last several years.  And that has given us a really differentiated product offering and an exciting roadmap. We have a few cards up our sleeve here that we’re going to bring out over the next few years that we are very excited about.

Tiffany: You mentioned the different geographical regions that are important to Cray – we see China, Japan, Europe, possibly India, all pursuing indigenous HPC programs. How will this impact Cray’s sales and what’s the strategy here – as a tag to that, I’m wondering the larger ClusterStor line if you could potentially provide storage to these systems for example?

Pete: We do have a large presence in many of these regions, and we’ve been working hard to expand our presence in those markets, especially in R&D. We have R&D centers in a number of countries now, and as we bring on the ClusterStor line we are going to open a facility in India as part of that transaction. So, we continue to grow R&D not just within the US, but also outside of the US. I mentioned earlier that China is not in play for us, but we’re definitely going to try and play a part in the other programs, whether it’s from a systems perspective or maybe a technology perspective.  Our larger storage line gives us another way to be part of these programs as well. I’m hopeful, but we will have to see how that plays out.

Tiffany: Over the last few years we’ve seen a broadening away from pure-play supercomputing at Cray – with the Appro acquisition, the Urika line, supercomputing-as-a-service plays with the Markley Group and Deloitte – and the recent addition of the ClusterStor line – can you connect these dots for us? Is this all part of your strategy to deal with a constricting and volatile traditional HPC market?

Pete: That’s a great question. Cray isn’t a broad, general IT provider like HPE, IBM or Dell. We’re a focused player who is focused on a very specific part of the market. We have had a single view of this market for quite a few years now about what’s happening in the market and where does Cray want to play, and we believe the traditional supercomputing, the modeling and simulation that’s been done on supercomputers since the start of Cray over 40 years ago, is converging with this high-end big data market that spans analytics, artificial intelligence, (including deep learning and machine learning). Our strategy is to help customers compute, store and manage their data, and also run analytics and data models all within this framework. So, we don’t see it as broadening away from supercomputing; we see all of these elements converging, and virtually everything we’ve done over the past ten years – and you’ve mentioned a few of them – is along that strategy line. So, for us it’s really about how we leverage our technology and knowhow in supercomputing to expand our reach and build a stronger business model for the company, while staying focused on the part of the market where we think we bring the most value.

A great example is the growth of data. Everybody agrees data is continuing to grow like crazy; it’s not going to stop. It’s forcing our customers to confront how they deal with increasing amounts of data, how do they run larger and larger models because now they have to put more data into their models and try to make them more real.  All these things start to play to our strengths of building larger systems, building systems that scale, and getting performance at scale. You mentioned the ClusterStor line, that’s key to our storage and data management part of our strategy; the Appro acquisition helps us on the computing side of our strategy; Urika helps us in analytics. Partnerships like Deloitte and Markley Group, they help us to reach new customers. So, I don’t see it as a broadening away from pure-play supercomputing, I actually see it as this is how the market is evolving over the next few years.

Tom: So it’s not that you’re broadening your portfolio to reach outside of the HPC segment, but you’re expanding your portfolio to sell more to the HPC segment, whether it be industry or public.

Pete: Exactly. I think that in five or ten years, the high-end of the supercomputing market and the high-end of the big data analytics, or the machine learning/deep learning market, is going to be one market. It’s going to look and feel like one big market, and that’s where we want to play. That’s the market that we want to be a part of, so what do we need to put in place across the company from technologies to the solutions to the go-to-market partnerships that are going to enable us to play in that part of the market.

Tiffany: In your last quarterly call you were asked about the health of the pipelines both for the traditional and commercial sectors. We spoke about this a bit, but could you add anything about how you’d characterize the nature and the size of the deals in those pipelines?

Pete: We don’t disclose specific details of our pipeline and our backlog, but as we said on our last earnings call, the pipeline is up slightly compared to last quarter. More customers are now beginning to move their plans forward. So, a lot of the uncertainty that we talked about that has been impacting the market, customers are starting to firm up their plans a little bit more and put more plans into place – the exascale plans are a good example. As a result, we’re starting to see more real, definable opportunities. It’s still to the point though where the movement of opportunities through the pipeline is still very slow, so we still believe that these are some good early signs of the potential market return, but the exact timing is hard to predict.

Tiffany: I’ll push a little bit more; how is your visibility for 2018 and how much of that forecasting process is tied to reading the political tea leaves? I’m thinking specifically about the three budget proposals that are currently on the table for DOE and NSF funding, there’s consensus on boosting exascale funding which has to bode well for tier 1 sales. But lab funding, NSF, NIH are in jeopardy. There were layoffs at Oak Ridge. So how are these unknowns impacting visibility and when will you be able to offer 2018 guidance?

Pete: There’s definitely limited visibility right now in the market because there are these unknowns. We continue to believe that the market will rebound, but it’s hard to say exactly when, or what the slope of that rebound is going to be. We haven’t put out any projections for 2018 yet, but most of what’s been happening on the political funding side of things is really targeted at systems that are out a few years. So, in terms of reading the political tea leaves that you mentioned, the impact to our 2018 revenues aren’t as large as you might think because a lot of those decisions have already been made and are already well under way. A supercomputer is not something you decide to buy on Monday and have a month from now — there’s a pretty long procurement process. I would say that overall – not just in the US but around the world – the political climate and appetite for high-end computing is very positive, and exascale is a great example of that not just in the US but around the world. There are a number of countries that are running exascale programs. I think all of these things taken individually and when you put them together gives us more confidence that the market is going to rebound over time. It’s too early for us to call 2018, but our competitiveness hasn’t changed, so I think when the market returns, we are going to return in a big way.

Tiffany: Do you find encouragement in the numbers for exascale that are in the current budget proposals?

Pete: Yes, I think it’s great. I’m a huge supporter of the program, not just from the perspective that we’re a provider for the program, but in terms of what advanced science and engineering does for competitiveness in the US and around the world.  I’m very pleased to see so many programs that are going after the very high-end of supercomputing because I think it’s going to make a huge difference to industrial competitiveness for a lot of countries.

Tiffany: One challenge I don’t think Cray has that some of its competitors do is the trend for hyperscalers (and their imitators) to get their servers from ODMs in Taiwan and China. This is hurting traditional server vendors; there was a prominent example earlier this year with HPE losing business from a major hyperscale customer, purported to be Microsoft. The Web-scale giants also have the buying power to secure huge discounts. What’s your perspective on the health of the server business and what does that signify for Cray?

Pete: You’re right in that we don’t really play much in that market. But what is clear to us, and it’s been clear for quite a few years now, is that the market is bifurcating.  The bifurcation is from the general-purpose systems that we’ve had in the past, for instance big SMP machines from Sun, IBM and HPE and the like, to really different kinds of purpose built systems.  One is large scale-out systems and the other is tightly integrated high-performance systems, and clearly the part of the market where cloud is predominantly focused is in the scale-out systems space.  That space has been overhauled – think about what happened with Dell and EMC and every major vendor that’s in that general-purpose market – that’s been a very challenging market. The part of the market that we play in is this tightly integrated high-performance market, whether it be for modeling and simulation or large databases or business intelligence or big data analytics or deep learning. We don’t see as much difference and change in that part of the market. In fact, I would argue over the past 10 years, there’s been a reduction in the number of companies that are building those very tightly integrated systems for that part of the market, which is part of the reason why we’ve had such success overall. I’m very pleased we are focused on that part of the market and not the other part of the market, where I think that public and private cloud models will dominate over time.

Tom: Cray is an iconic American supercomputer maker and while the company has undergone some transitions it is really one of a kind in terms of its history and having supercomputing as its core specialty. Seymour Cray, the father of supercomputing and of course the founder of Cray, is universally admired and respected. I imagine that honoring this long shadow is really an honor but could also be quite a burden.

Pete: It is an amazing honor and it’s something I think about quite a bit. Seymour Cray was a pioneer, a true visionary. His passion and vision is something I know all our employees, me included, work very hard to fulfill every day. I often think about how cool it is to be working at the company of the guy who practically invented the market. That’s amazing to me. I really believe at our core we are continuing to build and expand on Seymour’s vision today, by helping our customers solve the toughest challenges, scale to performance levels that people didn’t think were possible, and deliver faster competitive results for companies to be leaders in their industry. That is what we’re trying to do today and it’s fun to think about.

Tiffany: And there is the American connotation. Cray is a global enterprise, but Cray’s roots in Minnesota and Wisconsin, that’s undeniably part of the Cray brand. So let’s talk about Cray’s role in US supercomputing leadership. The Cray Titan supercomputer is the fastest US machine; there’s also Cori (NERSC), Trinity (NNSA), and Theta (Argonne). Cray is part of the DOE’s Fast Forward program to advance the technology needed for the United States’ first crop of exascale systems, on track for 2021-2022. How important is Cray to US supercomputing leadership and strategic competitiveness and isn’t it in the US’ interest to have 2-3 strong US-based companies capable of creating leadership-class systems?

Pete: You probably should ask the government that question, but I think it’s clear that we are a big part of supercomputing leadership in the US, and in many other countries around the world. If you look at the list of the largest supercomputers in the world, we are very fortunate to have a large percentage of those systems. I do think it’s important that there isn’t just one supercomputing company – although I will tell you I often dream about that!  But ultimately, I don’t believe the high-end supercomputing market is large enough to support too many companies, so it’s a balance of making sure customers have options and companies are able to grow and thrive in the market.  I do believe the industry is healthy and can sustain the players that it has. Cray is basically the last pure-play supercomputing company in the industry, and it shows. There’s no other companies like us left — we’re the only ones that are completely dedicated to this market.

Tiffany: Turning a little bit, we touched on trends earlier, the convergence of HPC and big data, but we’re also seeing artificial intelligence, machine learning and deep learning. These are really hot topics in HPC, so what are Cray’s views on AI writ large and what is the company doing from a product standpoint to address this market?

Pete: It’s super exciting. Artificial intelligence has the potential to transform a number of industries, and it’s already making an impact in a number of sectors. I would say virtually every one of our customers around the world are going to explore its potential across every vertical segment that we touch in the market today.

We believe artificial intelligence is really aligned with a number of our core capabilities and strengths because at its heart, AI is much more of a classical high performance computing problem. It leverages a lot of the capabilities that we build into our systems today, and we have seen really exciting results in scaling up deep learning and machine learning problems on our supercomputers. One of our energy customers, PGS, had a great example of using machine learning within their full-wave form inversion model to select a velocity model as part of their seismic simulations. We’ve used deep learning with weather simulation data to do real-time forecasting, what is called “Now-Casting”, which is really exciting. And we’ve done a bunch of work with the largest GPU-based supercomputer in the world at CSCS in Switzerland where we’ve scaled up deep learning models to thousands of GPUs. We have a team at Cray that’s specifically dedicated to this area, so you’re going to see a lot more coming from us over the next years in this space.

Tiffany: We also recently saw NERSC and Stanford scale up a deep learning model to ~15 petaflops on the Cori supercomputer.

Pete: NERSC has been a great partner of ours in working to scale up these kind of models, and looking at how to we take supercomputing technology and apply it to deep learning, machine learning, AI, and big data analytics. They are really pushing the boundaries of that. One of the advantages we have is to get to work with so many smart customers around the world. We really get a lot of insight into what’s going to happen in the market over the next few years.

Tiffany: Do you think that these trends in AI will open up a lot of market opportunities for Cray beyond the scientific sphere?

Pete: We’re hoping. It’s too early to call yet, but we definitely believe that it has a really big potential in terms of what Cray could be, and what Cray could grow into over the next five to ten years.

Tom: One of the things that amazes me about you Pete is how responsive you are to email, text and communications as a whole. I don’t know anyone in our industry, especially someone in your position that is so consistent and quick to respond. Granted a lot of your employees may not, on occasion, appreciate that, but how do you do it?

Pete: I found something that I really love doing, and I’m blessed to have job and a company where I get to work with a lot of smart people and a lot of smart customers, and that gets me energized every day. It’s what you would naturally do when you enjoy what you do, so I don’t think I do anything special and our success as a company has been a heck of a lot more about all the employees of Cray and our customers than it has been about me. But I will tell you, I do really love being here and being part of the industry.

Tom: One final question you’re going to have to think very carefully on: how does your daughter feel about having an HPC rock star as a dad?

Pete: (Laughs) My daughter is as embarrassed about her dad as probably most 20-year-old girls are. I may be somewhat of a recognized name in HPC, but I think she has the same viewpoint as many kids: drop me off a block away from my friends so nobody sees me with you!