Accelerating HPC in Formula One

By Tiffany Trader

November 3, 2015

The Formula One racing community has embraced HPC as an essential part of its workflow to build ever-speedier race cars, but suggestions that CFD should obviate the need for wind tunnel testing continue to draw heated commentary. One of the aerodynamics software vendors that supports this move is Exa Corporation. The Burlington, Mass.-headquartered company serves the international automotive and transportation markets and maintains a global customer roster that includes NASA, BMW, Ford, Jaguar and Tesla, and more than one F1 racing team.

Motor sports are specialty for Exa, which positions its PowerFLOW software as uniquely suited to the demands of racing-oriented computation.

“PowerFLOW fits into the rapidly changing environment of racing by providing a broad range of data and flow visualization, which can be used to tune the vehicle design for optimal performance,” the company asserts on its website. “You can compute the dependence on ride heights, vehicle speed, wheel positions, and crosswind or slip angles by simulating key points on the track circuit and evaluate the effects of your aerodynamic tuning and design choices.”

At the heart of Exa’s PowerFLOW software suite is a patented Lattice Boltzmann-based implementation of computational fluid dynamics (CFD) that is known for its high accuracy in low Mach number regimes. Founded in 1991, Exa has spent more than two decades perfecting its use of Lattice Boltzmann-based physics to perform simulations that accurately predict the performance of designs and provide a route to further optimizations.

HPCwire recently spoke with the Exa team about how CFD generally as well as their approach specifically offers benefits over expensive physical testing and reduces the need for costly late-stage changes. Much of our conversation centered on the limitations of wind tunnels with respect to complexities of the testing process and specific shortfalls that arise due to its being “a simulated version of reality that creates wind in a building around a car that’s not moving and a road surface that’s not moving underneath it.”

Exa finds itself having to counter the perception that wind tunnels provide this perfect representation of reality, but the fact is that wind tunnel testing is still a facsimile, a physical simulation as opposed to a digital one. Dr. Brad Duncan, aerodynamics applications director at Exa, explained further.

“The primary problem with a wind tunnel,” Duncan began, “is that you’re trying to model a moving car in a stationary laboratory. In order to make the experience road-like you have to blow the air at the right speed that the car would be driving; you also have to model the fact that the airflow is an open space, but the lab is a confined space; and there’s the fact that everything about the wind tunnel is fixed, but in the real-world the car is on a road and is moving, so you would have belts moving and wind blowing under the car. To capture all of this requires quite a few approximations and complex systems that have a lot of issues.

“One of the main challenges relates to air flow. When blowing air through a duct, trying to get the speed of that air right is a very tricky thing. One of the leading order errors is that the flow speed around the car doesn’t exactly match what you are trying to model, which is the car driving at a certain speed, say 150-200 miles per hour, down the track. The solution is to pick a point of calibration, but if the speed is correct at two lengths in front of the car, it doesn’t mean it’s accurate at the car. In aerodynamics flow speed is very important for accuracy, but in this simulated environment, forces will be too high or too low based on an error in the test speed calibration.

“There’s another problem because when you are testing in the wind tunnel with a Formula One car, you are not able to test at the true speed. A lot of wind tunnels will go faster than highway speed but max out at a speed that is lower than true track speed. Straightaway speed, depending on the particular circuit, can be as high as 200 miles per hour. When these high-speeds are attempted, the tires and the air heat up, which also skews the data. There are a stack of correction factors and adjustments that have to be made to try to address these effects and standardize them, and that creates an error stack up that can add additional tolerance to every number.”

Another factor is the force of the air flow coming off the rear wing, Duncan explained. Very much like how takeoff works on an airplane (but upside down), the flow that goes past the rear car wing is getting pushed up in order to create a tremendous down force on the back of the car. The problem is that the wind tunnel is feeling the presence of this flow that is getting pushed upward far away from the car but starting to hit the walls of the wind tunnel in the collector, the part of the wind tunnel where the air goes back in. That upwash from the wing makes the car look bigger to the air, making an even bigger obstacle to the airflow in the wind tunnel, according to Exa’s Duncan, who adds that all of these individual variances accumulate and will shift up or down the forces that engineering teams are trying to measure.

Sometimes the car itself is also a model at about 60 percent scale (injecting even more approximations into the process), but there is testing where the actual car is used, however even then considerations must be made to how vehicles are attached to the wind tunnel. For example, the car may be attached to a central sting that goes up to the roof (generally used in a scale model) or it can be secured at the struts.

Traditionally, the entire process was based on physical testing where all design was done in a wind tunnel. Now each car company has its own unique workflow and CFD testing can be used in all stages of the process and wind tunnels can also be used throughout. Most automotive companies are using a mix of both; a typical workflow may be heavy in CFD in the beginning and then wind tunnel testing comes in due to compliance and regulations dictated by the EPA (in the US) and the NEDC (in Europe). Even in this area, changes are coming, say Exa, referencing a recent Jaguar model that was completely developed using Exa CFD software and then only the final validation was done in wind tunnel. Discounting the artificial restrictions set down by the Formula One management body, wind tunnel testing is a more limited resource than CFD, but it is still essential for meeting regulatory requirements around fuel emissions. (There is some overlap with safety regulations, but according to Exa, that tends to be its own domain: crash testing and crash simulation.)

“The real value of simulation for aerodynamics is about designing the vehicle so it performs well. What we’re trying to do is make the world a better place — improve fuel economy, reduce emissions, and still make it possible for car companies to sell those cars because their ability to sell is based on the aesthetics and performance of the car and all the other things that consumers are looking for. They’re not necessarily going pay for fuel economy as much as some other things but we want the car companies to be able to deliver a competitive vehicle that performs well in every way, is still beautiful, still has the interior space, still has the safety, and meets all the regulations. We want to give car companies a way to be able to be best in class in aerodynamics and then they can say that this car has better fuel economy than all of its competitors,” said Duncan.

“Where aerodynamics simulation fits in is it allows you to design for fuel economy and emissions. Because you are using a digital test environment you are able to do that in collaboration with the other work that the car company is doing, which is designing the car for style and checking that it meets all of their other engineering requirements.”

“When it comes to engineering simulations, aerodynamics is the one that is most closely coupled to the aesthetics of the styling of the car,” Duncan shared. “Brand identity and what determines if a car sells has so much more to do with styling.

“They say that form follows function, but that’s not really true in the automotive market.

“And for race cars it’s a different story, it’s about winning the next race. For that the simulation has to be accurate and it has to give them a better design solution.

“In automotive racing, you have these engineers that are competitive and then they have to find the best answer to the design problem as fast as possible, which means they need accurate data but they also need something more like what you would call insight simulation can give you that insight because it offers so much more data — visual data and and practical data that helps them make a design decision.

“That’s one of the key limitations to the wind tunnel; you can’t get insight into why something works or doesn’t work.”

Duncan also noted that doing something on a computer allows engineers to optimize the testing process in a way that just isn’t possible with physical testing.

“The workflow is complex and involves adding and removing parts on a car, positioning them differently, combining them in different ways,” he explained. “When you remove material or reposition a part, if you then decide to put it back, if you’re trying to do three different things and find the combinations means you have to keep putting part A back to test part B and C in their positions. That can be challenging unless you have a very well-designed test, designed for efficient modularity. All these steps also involve a lot of time delays and the result is a number on the screen. Automated optimization is on a computer is much more straight-forward.”

Duncan reserved his heartiest endorsement of CFD for F1 design for Exa’s PowerFLOW software, a fully-transient time-accurate solver. PowerFLOW’s transient property means that the software can model how turbulence changes with time. According to Exa, the software provides unmatched simulation accuracy of ±1 count (equivalent to a Cd of 0.001), as opposed to an estimated ±30 counts for the rival steady-state codes. With Lattice Boltzmann physics, fluid is treated as mesoscopic particles (not molecules). “We’re using a very different approach to the physics,” he added, “based on IP and advanced research. You can look up Lattice Boltzmann in a text book, but you can’t look up what’s in our solver.”

“Some of the older CFD codes based on steady-state solvers are providing only one time-averaged flow result and it’s not able to tell you that flow is changing in time, so it’s not getting buffeting and oscillations or the changes that are happening due to a realistic flow effect. A flow around a car is not actually steady, so they are making this giant assumption by using steady-state, one-shot simulations. So they are not able to get as much benefit out of it. This built-in inaccuracy has traditionally been accepted as a tradeoff made to get the answer faster.”

“PowerFLOW doesn’t have a steady-state mode; it only runs in what we call real-mode,” said Duncan. “You see these small vortices and wakes and structures in the airflow that give you that look and feel that the flow is real and also is much more accurate because of that. It’s more similar to the wind tunnel and the track in that sense that the flow is realistically modeled because we have a significant competitive advantage in how we model airflow.

Most CFD codes use Navier-Stokes equations, but Lattice-Boltzman achieves very realistic flow simulation that works for these complex flows that flow around the F1 car. There are other codes based on LBM, inluding XFlow released by Next Limit Technologies, but Exa claims its PowerFLOW software has much more development behind it.

The tradeoff of this additional accuracy is that PowerFLOW is more computationally expensive to run a fully time accurate solver. “It’s so much more accurate that it’s hard to say that that’s actually a problem because if you run a steady-state solver and you get the wrong answer it might cost you more because you’re trying to work around the limitation and you might test something and find that it doesn’t work and then you have expensive delays and rebuilds because the information it gave you wasn’t very helpful. So in the end, you might say that a more accurate solver is priceless.”

In line with these benefits, Exa is not known for being a low-cost solution, Duncan said, but he pointed to Exa’s IP, higher levels of expertise and service and support. “We’re not just providing software in a box, we’re providing the whole engineering solution, so I think the customer is getting a lot more.”

Every simulation typically runs on a compute cluster of 200-400 cores for about one-to-two days, with turnaround time being the critical parameter. “The code scales very well, better than most other CFD solvers,” Duncan said, explaining that because the LB is particle-based there’s a lot of independent operations that make it very condusive to scaling up to thousands of processors. “Most customers are running the code on commodity machines with the individual core performance being the most important parameter along with good networking since overall performance is limited by the data bandwidth between nodes. We try to keep up on the latest Intel racks and interconnect fabrics, InfiniBand and the like,” said Duncan.

The F1 regulations currently enforce limitations on both wind tunnel testing and CFD work with an inverse relationship between the two. Testing is limited no matter how you do it, although workarounds have been rumored to exist. Given this limitation there is an argument to be made that having a more accurate test would make the most use out of the limited time available. As for those who point to poor results with teams who have used a pure-play CFD approach, Duncan suggested that this could be do to those traditional steady-state codes that make accuracy tradeoffs for the sake of speed. Looked at another way, the faster run times mean more runs can be completed within the mandated envelope, but there’s a catch-22 here. If the rules cap CPU hours, teams can do a lot more tests if they have a less accurate method, but a less accurate method is less valuable.

Clarifying an earlier point that he’d made, Duncan noted that he wouldn’t advise that just any CFD solution would be sufficient to completely replace wind tunnels, but an accurate simulation method could replace wind tunnels completely. “There are so many ways that you can get better data from CFD than wind tunnels in terms of predicting the performance of the car on the road. You can make the case that a team would benefit from spending their entire allocation on a more accurate solution.”

One gets the sense that these limitations while intentioned to enhance the competitiveness are hampering the sport and reducing potential innovations that could set the stage for technology transfer.

Duncan responded that automotive companies are very competitive right now to improve aerodynamics. With the CAFE requirements from the Obama administration, car makers are in a tight spot because they have to perform better and still make a car that people want to buy. They can’t give up performance or any other attributes of the car in order to meet these fuel economy mandates. The solution, said Duncan, has to be innovation.

“The only way to get a more aerodynamic car and still meet all these other constraints and make a beautiful car is a truly innovative design process,” he continued. “So that’s where certainly in motor sports and especially Formula One things that they try are always leading-edge. They have to do something else or they are not going to win. The strides that they would make would advance aerodynamics and fuel economy for the rest of the world.”

PowerFLOW is available via annual license or pay-as-you-go on-demand, with the option to run on secure, hosted, high-performance systems as part of its ExaCLOUD offering.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Bill Gropp – Pursuing the Next Big Thing at NCSA

March 28, 2017

About eight months ago Bill Gropp was elevated to acting director of the National Center for Supercomputing Applications (NCSA). Read more…

By John Russell

UK to Launch Six Major HPC Centers

March 27, 2017

Six high performance computing centers will be formally launched in the U.K. later this week intended to provide wider access to HPC resources to U.K. Read more…

By John Russell

AI in the News: Rao in at Intel, Ng out at Baidu, Nvidia on at Tencent Cloud

March 26, 2017

Just as AI has become the leitmotif of the advanced scale computing market, infusing much of the conversation about HPC in commercial and industrial spheres, it also is impacting high-level management changes in the industry. Read more…

By Doug Black

Scalable Informatics Ceases Operations

March 23, 2017

On the same day we reported on the uncertain future for HPC compiler company PathScale, we are sad to learn that another HPC vendor, Scalable Informatics, is closing its doors. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Quants Achieving Maximum Compute Power without the Learning Curve

The financial services industry is a fast-paced and data-intensive environment, and financial firms are realizing that they must modernize their IT infrastructures and invest in high performance computing (HPC) tools in order to survive. Read more…

‘Strategies in Biomedical Data Science’ Advances IT-Research Synergies

March 23, 2017

“Strategies in Biomedical Data Science: Driving Force for Innovation” by Jay A. Etchings is both an introductory text and a field guide for anyone working with biomedical data. Read more…

By Tiffany Trader

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

Google Launches New Machine Learning Journal

March 22, 2017

On Monday, Google announced plans to launch a new peer review journal and “ecosystem” Read more…

By John Russell

Swiss Researchers Peer Inside Chips with Improved X-Ray Imaging

March 22, 2017

Peering inside semiconductor chips using x-ray imaging isn’t new, but the technique hasn’t been especially good or easy to accomplish. Read more…

By John Russell

Bill Gropp – Pursuing the Next Big Thing at NCSA

March 28, 2017

About eight months ago Bill Gropp was elevated to acting director of the National Center for Supercomputing Applications (NCSA). Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the campaign. Read more…

By John Russell

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

New Japanese Supercomputing Project Targets Exascale

March 14, 2017

Another Japanese supercomputing project was revealed this week, this one from emerging supercomputer maker, ExaScaler Inc., and Keio University. The partners are working on an original supercomputer design with exascale aspirations. Read more…

By Tiffany Trader

Nvidia Debuts HGX-1 for Cloud; Announces Fujitsu AI Deal

March 9, 2017

On Monday Nvidia announced a major deal with Fujitsu to help build an AI supercomputer for RIKEN using 24 DGX-1 servers. Read more…

By John Russell

For IBM/OpenPOWER: Success in 2017 = (Volume) Sales

January 11, 2017

To a large degree IBM and the OpenPOWER Foundation have done what they said they would – assembling a substantial and growing ecosystem and bringing Power-based products to market, all in about three years. Read more…

By John Russell

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the campaign. Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

TSUBAME3.0 Points to Future HPE Pascal-NVLink-OPA Server

February 17, 2017

Since our initial coverage of the TSUBAME3.0 supercomputer yesterday, more details have come to light on this innovative project. Of particular interest is a new board design for NVLink-equipped Pascal P100 GPUs that will create another entrant to the space currently occupied by Nvidia's DGX-1 system, IBM's "Minsky" platform and the Supermicro SuperServer (1028GQ-TXR). Read more…

By Tiffany Trader

Tokyo Tech’s TSUBAME3.0 Will Be First HPE-SGI Super

February 16, 2017

In a press event Friday afternoon local time in Japan, Tokyo Institute of Technology (Tokyo Tech) announced its plans for the TSUBAME3.0 supercomputer, which will be Japan’s “fastest AI supercomputer,” Read more…

By Tiffany Trader

IBM Wants to be “Red Hat” of Deep Learning

January 26, 2017

IBM today announced the addition of TensorFlow and Chainer deep learning frameworks to its PowerAI suite of deep learning tools, which already includes popular offerings such as Caffe, Theano, and Torch. Read more…

By John Russell

Lighting up Aurora: Behind the Scenes at the Creation of the DOE’s Upcoming 200 Petaflops Supercomputer

December 1, 2016

In April 2015, U.S. Department of Energy Undersecretary Franklin Orr announced that Intel would be the prime contractor for Aurora: Read more…

By Jan Rowell

Leading Solution Providers

Is Liquid Cooling Ready to Go Mainstream?

February 13, 2017

Lost in the frenzy of SC16 was a substantial rise in the number of vendors showing server oriented liquid cooling technologies. Three decades ago liquid cooling was pretty much the exclusive realm of the Cray-2 and IBM mainframe class products. That’s changing. We are now seeing an emergence of x86 class server products with exotic plumbing technology ranging from Direct-to-Chip to servers and storage completely immersed in a dielectric fluid. Read more…

By Steve Campbell

Enlisting Deep Learning in the War on Cancer

December 7, 2016

Sometime in Q2 2017 the first ‘results’ of the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) will become publicly available according to Rick Stevens. He leads one of three JDACS4C pilot projects pressing deep learning (DL) into service in the War on Cancer. Read more…

By John Russell

BioTeam’s Berman Charts 2017 HPC Trends in Life Sciences

January 4, 2017

Twenty years ago high performance computing was nearly absent from life sciences. Today it’s used throughout life sciences and biomedical research. Genomics and the data deluge from modern lab instruments are the main drivers, but so is the longer-term desire to perform predictive simulation in support of Precision Medicine (PM). There’s even a specialized life sciences supercomputer, ‘Anton’ from D.E. Shaw Research, and the Pittsburgh Supercomputing Center is standing up its second Anton 2 and actively soliciting project proposals. There’s a lot going on. Read more…

By John Russell

HPC Startup Advances Auto-Parallelization’s Promise

January 23, 2017

The shift from single core to multicore hardware has made finding parallelism in codes more important than ever, but that hasn’t made the task of parallel programming any easier. Read more…

By Tiffany Trader

HPC Technique Propels Deep Learning at Scale

February 21, 2017

Researchers from Baidu’s Silicon Valley AI Lab (SVAIL) have adapted a well-known HPC communication technique to boost the speed and scale of their neural network training and now they are sharing their implementation with the larger deep learning community. Read more…

By Tiffany Trader

CPU Benchmarking: Haswell Versus POWER8

June 2, 2015

With OpenPOWER activity ramping up and IBM’s prominent role in the upcoming DOE machines Summit and Sierra, it’s a good time to look at how the IBM POWER CPU stacks up against the x86 Xeon Haswell CPU from Intel. Read more…

By Tiffany Trader

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

IDG to Be Bought by Chinese Investors; IDC to Spin Out HPC Group

January 19, 2017

US-based publishing and investment firm International Data Group, Inc. (IDG) will be acquired by a pair of Chinese investors, China Oceanwide Holdings Group Co., Ltd. Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This