In a recent paper,  the authors suggest that AI models collapse when trained on recursively generated data points from previous LLM models. Basically, the snake is eating its tail. Generative-AI has been successful in mining the Internet to create large LLM models that provide human-like answers to many problems. In particular, creating new text has been a forte of many LLMs. LLMs create a large amount of text from document summation to report generation. Inevitably, the LLM text will show up on the Internet and be scooped up for the next generation of LLM. The researchers in the paper suggest that this recursive training will lead to degraded models and, eventually, model collapse.

One suggested solution is to use synthetic data to train models. Another is to use LLMs to create synthetic data when access to large, diverse labeled datasets is limited. Synthetic data can mimic real-world data characteristics, which can improve data quality and increase the performance of custom LLMs (not the foundation models).

Plenty of HPC Data

While mainstream GenAI development has begun to scrounge for original data, HPC has no such issue. The practice of HPC has been creating numeric models that predict or mirror physical systems. From galaxies to proteins to F100, HPC has created clean synthetic solutions (data) since the beginning. As computing speeds have increased, so has model fidelity.

The ability of LLMs to find hidden relationships (or a “feature-ness”) in data was described in a previous HPCwire article.

A good example of using traditional HPC data for a foundational model is the Microsoft Aurora weather project. (Note: the Microsoft Aurora model is different than the Argonne Aurora Exascale system) Compared to the traditional numerical forecasting Integrated Forecasting Systems, the Aurora model provided a 5,000-fold increase in computational speed. Microsoft explains the Aurora approach and states (emphasis HPCwire):

“Aurora’s effectiveness lies in its training on more than a million hours of diverse weather and climate simulations, which enables it to develop a comprehensive understanding of atmospheric dynamics.“

Basically, they teach the LLM by using the outcomes of “weather physics” and then using what it learned to predict the weather. The success of foundational models like Aurora, point the way toward a sea change in HPC.

The traditional HPC approach is shown in Figure 2, where a model is often developed using the physics that describe the system of interest. Once developed, the model is run for a specific set of initial conditions. These models can take days, weeks, and even months to produce a solution. If a different set of initial conditions is used, the model must be re-run from the beginning.

With AI-augmented HPC, an LLM-like model is trained using existing data (e.g., simulated weather data or real data). The creation or collection of the training data can take weeks or months. In many cases, the “synthetic” HPC training data may already exist. Once trained, however, the model can quickly supply a solution to a set of initial conditions (“a query”) through inference instead of numeric computation. This process is shown in Figure 3.

The advantage of an AI-augmented approach is the flexibility of the model. When the initial conditions change, the model can infer the result instead of calculating from scratch. In the case of the Aurora foundational weather model, a new weather prediction can be found quickly without the need to grind through the mathematics of the traditional model. In addition, the foundational models can be fine-tuned to more prefigured solutions.

For example, the Microsoft Aurora Foundation weather model provides both accuracy and computational speed but offers remarkable versatility that is not available in traditional weather models. Aurora can forecast a broad range of atmospheric variables, from temperature and wind speed to air pollution levels and concentrations of greenhouse gases. In addition, Aurora’s architecture is designed to handle heterogeneous inputs and generate predictions at different resolutions and levels of fidelity.

No Free HPC Lunch

The capabilities of AI-augmented HPC models represent a new vista for simulating our physical world. There is no free lunch, however. With LLM models, the computation is up-front; it takes huge resources to train a model, but the resulting generalized inference can work for many problems variations. As mentioned, once you have your LLM-based weather model, you don’t need to compute tomorrow’s weather from scratch using traditional HPC models; you can ask the model to infer the weather, which can be just as accurate as “computing out” the traditional model. The more the model is used, the more the upfront computation is amortized. AI-augmented models may become more computationally efficient in the long run than the traditional approach. The HPC lunch should get cheaper and perhaps a bit more tasty. (And we need to make sure our tail stays off the menu.)

Importance of Good HPC Data

Training LLMs requires large amounts of data. In Data Science, clean data is essential for model generation and use. The often stated rule is, “Data scientists and engineers spend 80% of their time finding, cleaning, and organizing the data. The rest of the time is actual developing and running modes.”

The same is true for collecting data from the Internet for LLM models. Garbage-In Garbage-Out (GIGO) holds at all levels. In almost all cases, however, the synthetic HPC model data used to train LLMs does not suffer from such issues. By design traditional HPC model output data is well managed and often designed for further use, e.g. visualization.

Looking forward, the HPC community knows how to create good, clean synthetic data to train foundational LLMs for science and engineering. In fact, HPC has been creating such data for decades. It is somewhat fortuitous that many of the GPU-based systems that create the training data need AI-augmented HPC are the same systems that can create the foundational model.

Finally, there has been some concern that relying too much on foundational models may produce biased results. There is the issue of hallucinations and improper training. As AI methods and technology develop, this can be addressed. Essentially, we are showing the best models of reality we can create to an LLM and asking, “What do you see?” As our models and questions get better, we can expect the LLM to find relationships and insights that are invisible to us.

Folding Proteins the Smart Way

The recent Microsoft Aurora model is not the only AI-augmented success story. Google has essentially solved the thorny protein folding problem using AI in an application called AlphaFold. The protein folding problem involves computing the conformation of proteins to form biologically active structures. Solving the protein folding problem using traditional molecular modeling is a compute-intensive operation.

AlphaFold does not use traditional molecular dynamics to determine a solution. Instead, AlphaFold employs a deep learning algorithm that uses machine learning to predict the 3D structure of a protein from its amino acid sequence. AlphaFold used over 170,000 proteins from a public repository of protein sequences and structures to train AlphaFold. The program uses an approach similar to the GenerativeAI transformer but is not strictly an LLM. The AlphaFold AI model combines the desired folding sequence with multiple sequence alignment (MSA) statistics as inputs and outputs of a structure prediction. The initial training was reportedly conducted using between 100 and 200GPUs.

Work on AlphaFold began in 2018. By then, the structure of 17% of human presence was known. Thanks to AlphaFold, the 3-D structures for virtually all (98.5%) of the human proteins are currently known.

Chasing the Wrong Dog

With AI taking a bigger and possibly leading role in HPC, chasing a double-precision Top500 benchmark may not be the best way to gauge future HPC performance.

According to Rick Stevens, who leads the Aurora exascale efforts at Argonne National Laboratory, “It was a deliberate design decision to not use silicon for a matrix unit for double precision. We put that extra silicon into accelerating lower precision. In bfloat16, for example, we have a lot more performance. So that’s the technical reason”

Measuring AI performance has fallen to the MLCommons, an Artificial Intelligence engineering consortium, and their MLPerf suite of AI benchmarks.

For HPC, both traditional floating point (Top500) and low precision AI (MLPerf) will be important. At this time, the aims of these two benchmarks are somewhat orthogonal, and a new way of defining HPC performance is probably needed.

The path to scientific and engineering insight is about to change in a very big way.