IDAS: ‘Automagic’ HPC With Training Wheels

By Elizabeth Leake

September 12, 2019

High-performance computing (HPC) for research is notorious for having steep barriers to entry. For this reason, high-tech disciplines were early adopters, have used the most cycles and typically drove hardware and software purchasing decisions. But as more fields engage with data-intensive research and artificial intelligence (AI) workflows, data-center hardware and software landscapes are changing to meet a larger and more diverse community of practice.

The University of Iowa’s (UI) Interactive Data Analytics Service (IDAS) is an HPC resource that supports large-scale and collaborative data analytics workflows involving RStudio for R and Jupyter Notebook for Python (but not limited to Python). Applications via the IDAS interface look and feel like they do on a regular workstation, but with access to thousands of times the computing power. IDAS has its own HPC and graphics processing units (GPUs) and allows users to perform interactive data analysis tasks with applications used for machine learning and AI. In the future, the UI Research Services development team will assist with custom environments. There are plans to add a feature that allows researchers to access Iowa’s Argon supercomputer if more power is needed, or to purchase commercial cloud if locally-hosted HPC isn’t enough.

While RStudio for classroom use is available, RStudio commercial licensing terms changed and pricing increased midstream as IDAS was being developed. Therefore, until a more affordable and dynamic license distribution method is available, IDAS employs the free RStudio classroom license. In the future, a remote desktop feature will be added.

IDAS especially appeals to those who may not have used HPC in the past, but whose data-intensive research would benefit from advanced computational capabilities. Such disciplines are often referred to as, “the long tail of science,” and many have begun to engage with AI methodologies which make their work more computationally-intensive than ever. While they aren’t always the biggest compute consumers, their collective needs represent critical mass that IDAS accommodates quite well.

Experimentation within IDAS is safe with associated storage options that are appropriate for most classifications of data. Sensitive data should be evaluated for IDAS consideration on a case-by-case basis. “If you aren’t sure about your data’s classification, you can arrange a consultation with our compliance specialist,” said Research Services Director Danny Tang.

IDAS Jupyter Notebook serves both general research and classroom needs which were optimized during its pilot phase. A dozen faculty and staff from six departments test-drove the research platform for several months, and classroom attributes were trialed by students in Data Scientist Kang Lee’s (UI Research Services; College of Business) summer business analytics class.

Paul Gowder, UI College of Law

At the time of the mid-August launch, eight classroom instances were registered, including one by Professor Paul Gowder (UI College of Law). Gowder’s research involves the impact of machine learning and predictive modeling on both the practice and concept of law. “I’m planning to use IDAS in my ‘Introduction to Quantitative and Computational Legal Reasoning’ course, and may also utilize it for my research involving AI to understand contracts and records from publicly-available court documents,” said Gowder.

Teaching Assistant James Kent (UI Interdisciplinary Graduate Program in Neuroscience) aides Professor Michelle Voss with Iowa’s structural/functional Magnetic Resonance Imaging (MRI) methods course. “Our two biggest challenges were installing software on heterogeneous hardware, and random-access memory (RAM) restrictions when running a virtual machine,” said Kent. “With Jupyter Notebook on IDAS, the HPC is handled in the background; this provides students with enough computational power to complete full analyses on real MRI data, not the toy datasets we always curated for this class,” he added.

Kent explained how useful he imagines IDAS will be for workshops, too. Since most are one to three hours in length, less instructional time will be spent installing software and/or learning how to master the hardware. “Some students get discouraged if the preparation phase is too tedious; it presents an unnecessary hurdle,” he said. “If students feel this time is wasted, some may skip the workshop, leave early or walk away with the impression that programming is inaccessible to them,” he added.

Outside of the formal classroom setting, Kent thinks IDAS will be perfect for hackathons. “They’re often about prototyping and Jupyter Notebooks provides a great platform on which to interact with code and receive immediate feedback,” said Kent. In the past, a downside of hackathons was that machine learning models can take a lot of RAM and/or use significant GPU power. HPC is the answer for increased RAM and GPU usage, but mastery of Jupyter Notebooks was difficult with the traditional HPC interface. Kent expressed appreciation for IDAS’ development, and sustained back-end management when he said, “IDAS combines Jupyter Notebooks’ ease of prototyping with the power of HPC that runs and scales ‘automagically’ in the background!”

Prof. Michelle Voss and TA James Kent from the Interdisciplinary Graduate Program in Neuroscience leading Brainhack 2019
Brandon LeBeau

Brandon LeBeau (UI College of Education) uses IDAS in an introductory statistics course that allows future educators to explore statistical concepts, remove assumption barriers in the statistical analysis, and introduces them to best practices surrounding script-based analyses. “The goal for this course is for students to learn how to reason with data and make appropriate statistical/data arguments, vs. learning how to program,” he said, and added, “This hands-on approach will help them on their journey to becoming stronger data connoisseurs.”

The computation component in IDAS’ Jupyter Notebooks instance, with an R kernel, gives students access to a modern computing environment without requiring an instructor who doubles as an information technology (IT) professional. “The user experience is also oblivious to compute functions managed by Research Services on the back-end; otherwise the need for programming skills would become a barrier to entry for many students and faculty alike,” said LeBeau.

IDAS was the brainchild of former UI Research Services Director Ben Rogers who aligned the necessary collaborations and allocated funds to make his vision a reality. It was developed by Lee and Interactive Systems Lead Cody B. Johnson. Research Services would like to thank Chris Wilkins and Kevin Zhu of the UI Core IT Facilities Group for their help with deploying IDAS.

“While ten other universities that we know of have similar environments, Iowa is one of the first to host a large-scale, cloud-enabled and expanded HPC resource,” said Lee. “We’re excited about the number of Iowa colleges and departments that engaged in our pilot, and some of the novel applications they’re pursuing on IDAS,” he added.

For more information, visit the U-Iowa ITS-Research Services website.

About the Author

HPCwire Contributing Editor Elizabeth Leake is a consultant, correspondent and advocate who serves the global high performance computing (HPC) and data science industries. In 2012, she founded STEM-Trek, a global, grassroots nonprofit organization that supports workforce development opportunities for science, technology, engineering and mathematics (STEM) scholars from underserved regions and underrepresented groups.

As a program director, Leake has mentored hundreds of early-career professionals who are breaking cultural barriers in an effort to accelerate scientific and engineering discoveries. Her multinational programs have specific themes that resonate with global stakeholders, such as food security data science, blockchain for social good, cybersecurity/risk mitigation, and more. As a conference blogger and communicator, her work drew recognition when STEM-Trek received the 2016 and 2017 HPCwire Editors’ Choice Awards for Workforce Diversity Leadership.

An earlier version of this article appears on the University of Iowa website.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industry updates delivered to you every week!

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion XL — were added to the benchmark suite as MLPerf continues Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing power it brings to artificial intelligence.  Nvidia's DGX Read more…

Call for Participation in Workshop on Potential NSF CISE Quantum Initiative

March 26, 2024

Editor’s Note: Next month there will be a workshop to discuss what a quantum initiative led by NSF’s Computer, Information Science and Engineering (CISE) directorate could entail. The details are posted below in a Ca Read more…

Waseda U. Researchers Reports New Quantum Algorithm for Speeding Optimization

March 25, 2024

Optimization problems cover a wide range of applications and are often cited as good candidates for quantum computing. However, the execution time for constrained combinatorial optimization applications on quantum device Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at the network layer threatens to make bigger and brawnier pro Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HBM3E memory as well as the the ability to train 1 trillion pa Read more…

MLPerf Inference 4.0 Results Showcase GenAI; Nvidia Still Dominates

March 28, 2024

There were no startling surprises in the latest MLPerf Inference benchmark (4.0) results released yesterday. Two new workloads — Llama 2 and Stable Diffusion Read more…

Q&A with Nvidia’s Chief of DGX Systems on the DGX-GB200 Rack-scale System

March 27, 2024

Pictures of Nvidia's new flagship mega-server, the DGX GB200, on the GTC show floor got favorable reactions on social media for the sheer amount of computing po Read more…

NVLink: Faster Interconnects and Switches to Help Relieve Data Bottlenecks

March 25, 2024

Nvidia’s new Blackwell architecture may have stolen the show this week at the GPU Technology Conference in San Jose, California. But an emerging bottleneck at Read more…

Who is David Blackwell?

March 22, 2024

During GTC24, co-founder and president of NVIDIA Jensen Huang unveiled the Blackwell GPU. This GPU itself is heavily optimized for AI work, boasting 192GB of HB Read more…

Nvidia Looks to Accelerate GenAI Adoption with NIM

March 19, 2024

Today at the GPU Technology Conference, Nvidia launched a new offering aimed at helping customers quickly deploy their generative AI applications in a secure, s Read more…

The Generative AI Future Is Now, Nvidia’s Huang Says

March 19, 2024

We are in the early days of a transformative shift in how business gets done thanks to the advent of generative AI, according to Nvidia CEO and cofounder Jensen Read more…

Nvidia’s New Blackwell GPU Can Train AI Models with Trillions of Parameters

March 18, 2024

Nvidia's latest and fastest GPU, codenamed Blackwell, is here and will underpin the company's AI plans this year. The chip offers performance improvements from Read more…

Nvidia Showcases Quantum Cloud, Expanding Quantum Portfolio at GTC24

March 18, 2024

Nvidia’s barrage of quantum news at GTC24 this week includes new products, signature collaborations, and a new Nvidia Quantum Cloud for quantum developers. Wh Read more…

Alibaba Shuts Down its Quantum Computing Effort

November 30, 2023

In case you missed it, China’s e-commerce giant Alibaba has shut down its quantum computing research effort. It’s not entirely clear what drove the change. Read more…

Nvidia H100: Are 550,000 GPUs Enough for This Year?

August 17, 2023

The GPU Squeeze continues to place a premium on Nvidia H100 GPUs. In a recent Financial Times article, Nvidia reports that it expects to ship 550,000 of its lat Read more…

Shutterstock 1285747942

AMD’s Horsepower-packed MI300X GPU Beats Nvidia’s Upcoming H200

December 7, 2023

AMD and Nvidia are locked in an AI performance battle – much like the gaming GPU performance clash the companies have waged for decades. AMD has claimed it Read more…

DoD Takes a Long View of Quantum Computing

December 19, 2023

Given the large sums tied to expensive weapon systems – think $100-million-plus per F-35 fighter – it’s easy to forget the U.S. Department of Defense is a Read more…

Synopsys Eats Ansys: Does HPC Get Indigestion?

February 8, 2024

Recently, it was announced that Synopsys is buying HPC tool developer Ansys. Started in Pittsburgh, Pa., in 1970 as Swanson Analysis Systems, Inc. (SASI) by John Swanson (and eventually renamed), Ansys serves the CAE (Computer Aided Engineering)/multiphysics engineering simulation market. Read more…

Choosing the Right GPU for LLM Inference and Training

December 11, 2023

Accelerating the training and inference processes of deep learning models is crucial for unleashing their true potential and NVIDIA GPUs have emerged as a game- Read more…

Intel’s Server and PC Chip Development Will Blur After 2025

January 15, 2024

Intel's dealing with much more than chip rivals breathing down its neck; it is simultaneously integrating a bevy of new technologies such as chiplets, artificia Read more…

Baidu Exits Quantum, Closely Following Alibaba’s Earlier Move

January 5, 2024

Reuters reported this week that Baidu, China’s giant e-commerce and services provider, is exiting the quantum computing development arena. Reuters reported � Read more…

Leading Solution Providers

Contributors

Comparing NVIDIA A100 and NVIDIA L40S: Which GPU is Ideal for AI and Graphics-Intensive Workloads?

October 30, 2023

With long lead times for the NVIDIA H100 and A100 GPUs, many organizations are looking at the new NVIDIA L40S GPU, which it’s a new GPU optimized for AI and g Read more…

Shutterstock 1179408610

Google Addresses the Mysteries of Its Hypercomputer 

December 28, 2023

When Google launched its Hypercomputer earlier this month (December 2023), the first reaction was, "Say what?" It turns out that the Hypercomputer is Google's t Read more…

AMD MI3000A

How AMD May Get Across the CUDA Moat

October 5, 2023

When discussing GenAI, the term "GPU" almost always enters the conversation and the topic often moves toward performance and access. Interestingly, the word "GPU" is assumed to mean "Nvidia" products. (As an aside, the popular Nvidia hardware used in GenAI are not technically... Read more…

Shutterstock 1606064203

Meta’s Zuckerberg Puts Its AI Future in the Hands of 600,000 GPUs

January 25, 2024

In under two minutes, Meta's CEO, Mark Zuckerberg, laid out the company's AI plans, which included a plan to build an artificial intelligence system with the eq Read more…

Google Introduces ‘Hypercomputer’ to Its AI Infrastructure

December 11, 2023

Google ran out of monikers to describe its new AI system released on December 7. Supercomputer perhaps wasn't an apt description, so it settled on Hypercomputer Read more…

China Is All In on a RISC-V Future

January 8, 2024

The state of RISC-V in China was discussed in a recent report released by the Jamestown Foundation, a Washington, D.C.-based think tank. The report, entitled "E Read more…

Intel Won’t Have a Xeon Max Chip with New Emerald Rapids CPU

December 14, 2023

As expected, Intel officially announced its 5th generation Xeon server chips codenamed Emerald Rapids at an event in New York City, where the focus was really o Read more…

IBM Quantum Summit: Two New QPUs, Upgraded Qiskit, 10-year Roadmap and More

December 4, 2023

IBM kicks off its annual Quantum Summit today and will announce a broad range of advances including its much-anticipated 1121-qubit Condor QPU, a smaller 133-qu Read more…

  • arrow
  • Click Here for More Headlines
  • arrow
HPCwire