There has been a bit of delay after our initial launch of Under the Wire. We decided to create some cutting-edge graphics, produce a concurrent video blog, and squeeze in some vacation time since the first installment. Now that all this is in place, the UTW installments should show up at least twice a month.

The accompanying video blog will highlight the topics in each installment and show up within a week of the written article (or sooner).

On to the topics that may have slipped under the wire.

Free as in AI Beer

We all know the free software movement has dramatically affected HPC. From the Linux plumbing to the MPI-based applications, open source continues to fuel HPC growth. Early on, the concept of open-source software was confusing to many users. The “free as in speech” is a matter of liberty, not price, and it was the underlying “freedom” that was emphasized. Support and packaging notwithstanding, freely available source code also means the software (actually source code) is available for free, as in “free beer.”

For the most part, we had this all worked out, but when it comes to GenAI, the confusion is back. First, let’s assume I create a company and call it “OpenPizza” and produce some really good pizza. I decided to give free samples, but you must pay me if you want to serve my pizza in your restaurant.

Since I called it “open,” you would think the recipe would be openly available so people could see what was in my culinary masterpiece. Of course, someone could copy my creation, and I might trademark the name or realize that the pizza is also a gateway to upsell other items on the menu. But that is not going to happen. Not Free as in speech, not free as in beer, just free as in sample.

Moving on to the “open” LLMs, consider Llama3 from Meta. This LLM is available in various sizes, from huge GPU clusters to many low-power devices (even cell phones). Llama and thousands of other models are available from HuggingFace. These models are often called “open source” because they provide the model weights (the data that directs the model), but in reality, the data used to train the model is not available. A better name is probably “open weights” model. Open-weight models have some freedom because you have access to the full set of weights and can fine-tune or continue the pre-training run. Releasing the weights is a good start, but it does not give someone the freedom to fully recreate your model.

There are many practical issues with training data access. First, the data set is huge and will not fit on my new notebook, and the sysadmin will not like me downloading over 15 trillion tokens and requesting all the GPUs in the cluster for six months. That is not to say some organizations would not be interested in such an opportunity.

There are openly available training data sets and full instructions on creating your own LLM. The Olmo project provides models with full model weights, training code, training logs, training metrics in the form, and inference code. The training data is available as Dolma, an open dataset from a diverse mix of web content, academic publications, code, books, and comprehensive materials.

The Allen Institute for AI (AI2) recently released OLMo 7B, a truly open, state-of-the-art large language model that includes access to the pre-training data and training code. This project empowers researchers and developers to use the best and most open models to advance the open science of language models. (i.e., someone has to figure out what is going on in those GPUs)

Finally, I want to emphasize that I’m not advocating that anyone open or close their LLM; that is your choice; what I am asking is for better use of the terms “Open” and “Open Source.” We took 20 years to kind of figure those out. Let’s move forward in a good way. I am also available to discuss this further (free as in speech) over a beverage (free as in beer) and some OpenPizza.

[Update (23-Aug): Right after this article was posted, I found this news in my mailbox: We finally have a definition for open-source AI. A good start.]

A Good Thing™

Another story that may have slipped under the wire comes from super-site Phoronix. In a July story, Michael Larabel discusses “Device Memory TCP Nears The Finish Line For More Efficient Networked Accelerators“. Of course not something most people discuss at the summer picnic, yet it is important for many HPC users.

As mentioned in the article, Google engineers posted experimental Linux code for Device Memory TCP for more efficient transferring of data from GPUs/accelerators to network devices without going through a host CPU memory buffer.

Device Memory TCP (Devmem TCP) is a Linux kernel feature that transfers data to/from device memory without copying data through a host memory buffer. Many highly interconnected HPC systems that use TPUs, GPUs, NPUs, and other accelerator device types must use system memory to buffer data. Devmem TCP aims to avoid memory copies through the host system memory when sending or receiving data from those discrete devices across the network.

Suppose a GPU on one server needs to communicate with a GPU on another. In that case, the communication over standard TCP (meaning it is designed for Ethernet-based clusters) will bypass the kernel and allow direct reads and writes to the device. This process is very similar to what InfiniBand does so well — direct memory transfer from application to applicant without introducing any kernel overhead. Expect things to get faster.

We are going to give this feature a Martha Stewart “Good Thing™” acknowledgment. Nice work.

Fortran, Why Yes Fortran

I recently came across a paper about the Portability of Fortran’s DO CONCURRENT on GPUs. For those who may not know (or have forgotten), the DO CONCURRENT construct was introduced in ISO Fortran 2008. It tells the compiler that the iterations of the DO CONCURRENT loop are independent and can be executed in parallel (that means speed-up). It would also be nice if the compiler could target GPUs as parallel acceleration devices.

As the paper points out, there is a continuing interest in using standard languages like Fortran for accelerated computing to avoid hardware-specific vendor APIs. The DO CONCURRENT (DC) loop has been successfully demonstrated for Fortran codes on the Nvidia platforms. However, support for DC on other hardware platforms has taken longer to implement. Recently, both HPE (for AMD APIs) and Intel (for Intel APIs) have added DC GPU offload support to their compilers.

The paper explores the current portability of using DC across GPU vendors using a production solar surface flux evolution code, HipFT. The authors discuss implementation and compilation details, including when/where using directive APIs for data movement is needed/desired compared to using a unified memory system. Performance results for both data center and consumer platforms are provided.

A few points to consider. First, hiding hardware APIs has earned today’s second “Good Thing™” award. Hiding APIs means codes become more portable, and we all like to move our codes from one hardware platform to another.

Second, looking at the authors, the following companies contributed: Predictive Science Inc., Nvidia, AMD, Intel, and HPE. Talk about playing nice. This kind of cooperation is notable, and it benefits the HPC community in a big way.

Rust Never Sleeps

The Rust language has been in the news lately, and I thought I would be one of the many tech writers who would drop the Neil Young Album as a sub-title. (Note: I also want to point out that, you know, I was there, man, back in the 70s, so from what I remember, it was all pretty cool, so I figure that gives me some kind of special license to use that sub-tile, but I digress)

The reason I am bringing up Rust is the recent announcement “DARPA: Translating All C to Rust (TRACTOR)“. Gulp. As stated on the page:

“After more than two decades of grappling with memory safety issues in C and C++, the software engineering community has reached a consensus. It’s not enough to rely on bug-finding tools. The preferred approach is to use “safe” programming languages that can reject unsafe programs at compile time, thereby preventing the emergence of memory safety issues.

The TRACTOR program aims to automate the translation of legacy C code to Rust. The goal is to achieve the same quality and style that a skilled Rust developer would produce, thereby eliminating the entire class of memory safety security vulnerabilities present in C programs. This program may involve novel combinations of software analysis, such as static analysis and dynamic analysis, and machine learning techniques like large language models.”

Breathe, Doug, Breathe. Let’s break this down. First, memory-safe Rust is a good idea, C is great, and it takes some skill. I consider it a universal assembly language because it is low-level and portable across most hardware. It can also be like juggling chainsaws at times. Some people can juggle software chainsaws, but many security issues we encounter result from one of these chainsaws still flipping around in the air due to mistakes or inexperience.

Rust is now used in the Linux kernel, and it is growing fast. It provides a way to create “memory-safe programs (those that can’t be caused to let the bad guys in, no chainsaws). DARPA has good reason to want better security; the D stands for “defense.”

Going forward, using Rust is a great idea and should be considered for any new project. The announcement, however, was for tools to translate existing C code to Rust. The “existing” part is what has me hyperventilating.

I spent part of my career working on a tool that converted Fortran 77 to Fortran 77 with MPI calls. The project worked and taught us some interesting ideas about software conversion.

1. Parsing Fortran 77 code is hard. You would not believe how some people write code. (I take that back; you probably would.) Of course, it would compile, but we were looking beyond basic compiler structures.
2. Once parsed, there is no guarantee that what you discovered makes any sense, does not have bugs (bugs can manifest differently in different languages), or is efficient.
3. We were asked about translating C code. The conversation was short and usually started and ended with one word, “pointers” (Fortran 77 has no such “feature.”).

A mandate to convert all DARPA-managed HPC C-based codes into Rust is a scary proposition. At this point, I don’t think there is a magic bullet even with the current level of GenAI; my Rust Never Sleeps hallucinations are going to be pretty tame compared to a code conversion LLM trained on all that 70’s C code scraped from the Internet.

Converting everything to Rust is a noble goal, but it may need to be thought through a bit more over some free AI beer.