With HPC demand ballooning and Moore’s law slowing down, modern supercomputers often undergo exhaustive efficiency efforts aimed at ameliorating exorbitant energy bills and correspondingly large carbon footprints. Others, meanwhile, are asking: is min-maxing the best option, or are there easier paths to reducing the bills and emissions of modern HPC? One such party is Lancium, a Houston-based company that aims to match gigawatts of stranded renewable energy with datacenter workloads like HPC for a fraction of the typical cost.

A massive transformation

“Many people are unaware of this massive transformation going on in the electrical grid system in the United States and around the world,” explained Andrew Grimshaw, president of Lancium Compute, in an interview with HPCwire. Grimshaw was a professor of computer science at the University of Virginia for 34 years until he departed to go full-time at Lancium just a few months ago. The impetus for the move: an opportunity to help Lancium leverage the gigawatts of renewable energy coming online in West Texas (6-7GW of additional capacity per year).

“What you have now is incredibly inexpensive, carbon-free energy that’s available — we’re throwing away terawatt-hours of it every year, and it’s because of two challenges with renewables,” Grimshaw said. Those two problems: intermittency, or the innate fluctuations of renewable energy due to changes in sunlight or wind, and congestion.

“Typically the massive amounts of renewables aren’t where there are people or [demand] load,” Grimshaw said. “They’re oftentimes — I hesitate to use the phrase ‘middle of nowhere,’ but if you’ve ever been to West Texas, you know what ‘middle of nowhere’ means. And you have to get the energy out, [but] oftentimes the wires carrying the energy out are full, and you can’t get any more through them. And when that happens, they have to turn off these renewables, and in essence, when you turn them off, you’re throwing away energy.”

All of this adds up to around 35GW of capacity in West Texas with only 5GW of local demand for electricity and 12-15GW of transmission capacity into the rest of Texas.

The solution: “we need to move into those places industries that initially don’t use a lot of infrastructure and which can be ramped up and down.”

And computing, Grimshaw said, fits that bill.

Lancium’s play

To that end, Lancium has been building datacenters beyond the CREZ lines, which Grimshaw describes as an often-congested set of transmission lines connecting West Texas power generation to the rest of the state. So far, there are two such publicly announced datacenters: one in Fort Stockton, one in Abilene. Currently, the Fort Stockton datacenter (or “Clean Campus,” as Lancium calls them) has 25MW of capacity, but Grimshaw said that the substations are under construction for 300MW of capacity at Fort Stockton and 200MW at Abilene. Both datacenters are expected to have access to those full capacities by early 2023.

At these locations, Grimshaw explained that Lancium is able to access “extremely inexpensive” energy — $10 to $20 per megawatt-hour, which translates to one or two cents per kilowatt-hour. (Compare to an average residential electricity price closer to 15 cents per kWh.) 15 percent of the time, the energy is actually negatively priced — customers are paid to accept the energy.

This energy, by virtue of its origins (predominantly wind) and lack of sufficient demand, is extraordinarily clean. “We can help stabilize the grid by ramping our load up and down and we get incredibly cheap energy, and the vast, vast, vast majority of that energy is renewable,” Grimshaw said, adding that some companies made claims “bordering on the truth” with regard to their clean energy efforts. Not so with Lancium. “It’s not like buying carbon credits,” Grimshaw said.

With energy taken care of, that just leaves one core need. “All we need for a datacenter, when push comes to shove, is electricity and bandwidth,” Grimshaw said. “Fortunately, because of the way these wind farms are controlled, every single wind farm has a fiber bundle going to it, so that part of Texas is lousy with fiber. … We can actually get a hundred gigabits per second for less money in Abilene than we can in Houston.”

Dirt-cheap energy and plentiful networking, Grimshaw said, removes a lot of hardware constraints. “We use older gear, which is slower, but we have more gear — we just go wider,” he said. “We’re not putting new gear in. In fact, the gear we’re using right now comes out of Facebook.”

“One of the things [Lancium CEO Michael McNamara] told me when he brought me on is: ‘consider electricity free.’” “And so we don’t really care so much about whether the nodes have the best flops per watt because we’re basically using renewable power when there’s plenty of it. And on days when it’s not producing and the humans need the electricity, we’ll drop our load.”

Grimshaw said that the datacenters, similarly, resemble “chicken sheds” and don’t use any active cooling. “We bring in lots of external air for cooling and blow it right back out because it turns out you don’t have to keep processors at 70 degrees Fahrenheit,” he said. (On the other hand: “We’re not running InfiniBand at 100 gigabit because InfiniBand doesn’t have the best reputation for being robust under hot temperatures — and our machine rooms are hot.”)

Preparing for real workloads

Of course, not all workloads are suited for Lancium’s offering — and for those that are, a lot of work is going into ensuring that those workloads can stop and resume on a dime while retaining access to stranded power.

“There’s two kinds of loads that we run. We run Bitcoin [mining] and we run HPC cloud computing,” Grimshaw said. “We don’t want applications that interact with any humans or other programs,” he added, since Lancium, of course, needs to be able to spin down workloads in response to energy availability. For suitable workloads, Lancium employs its “Smart Response” control software, which is able to take processors from max energy consumption to standby power in under a second.

“Smart Response allows us to balance the grid and dance with the grid, but that has to be done in an under-15 second time window to be most useful for stabilizing the grid,” Grimshaw said, adding that it was “harder to do that” with cloud computing.

“We can freeze stuff, but the difference between cloud and Bitcoin is really in state,” he continued. “So if you’re running a climate model, you have a ton of state while you’re running, right? … And while we can freeze you, freezing a machine doesn’t drop the power that much compared to the way Bitcoin [does] in terms of relative differences.”

To address this problem, Lancium employed a combination of Singularity containers and a tool called DMTCP (distributed multithreaded checkpointing). “With DMTCP, I could take an arbitrary program that some Joe Scientist wrote and he never wrote it to checkpoint, and I can checkpoint that application to disk,” Grimshaw said. “I can persist it to disk and restart it later and that application will never know that that happened.”

“So we combined DMTCP with Singularity so you can have applications running inside Singularity containers, checkpoint them to disk, persist them, and then restart them either later in the same area or in a different place.”

Lancium developed analogous tools for GPU jobs and MPI applications spanning multiple nodes. Then, they combined these tools and turned them over to an external company called CIQ to manage them.

Using this suite of tools, Lancium is then able to quickly checkpoint these cloud HPC jobs and resume them at a different time or — leveraging the plentiful fiber — place, if there happened to be another site with more available power.

Grimshaw said that all-in, customers can expect to pay around one to four cents per core-hour.

What’s next

So far, with the substations for its clean campuses still being built, Lancium has predominantly engaged with Bitcoin miners and a subset of cloud computing based out of Houston. This Houston-based computing uses renewable energy credits instead of stranded power, but Grimshaw said Lancium was looking to migrate those workloads as soon as possible.

In terms of other workloads, Lancium has been working with the Open Science Grid consortium, donating about a million CPU-hours a week, as well as a handful of other research users. Grimshaw said that so far, the bulk of the research load has been computational fluid dynamics and molecular modeling.

And by all indications, this is just the beginning.

“We’ve done all the paperwork to take a gigawatt off the grid,” Grimshaw said. “And without giving too much away, we know that we can have access to two gigawatts, but we haven’t done all the paperwork. That’s a lot of energy, all in west Texas.” Grimshaw confirmed that additional datacenters were planned, but said that the locations are not yet public.

“Instead of making the grid adapt to the datacenter,” he concluded, “we’re making the datacenter adapt to the grid.”