Ubiquitous Parallelism and the Classroom

By Tom Murphy of Contra Costa College, Paul Gray of the University of Northern Iowa, Charlie Peck of Earlham College, and Dave Joiner of Kean University

November 20, 2009

The oft-contended best simple statement is that we need ubiquitous parallelism in the classroom. Once upon a time, it was solely the lunatic fringe, programming esoteric architectures squirreled away in very special corners of the globe that cared about parallelism. In the near future, most electronic devices will have multiple cores which would benefit greatly from parallel programming. The low hanging fruit is, of course, the student’s laptop, and aiding the student to make full use of that laptop.

So how do we get there?

Our perception of next steps comes from close to a decade of collaboration pushing parallel and distributed computing education. This doesn’t mean we are right, just that we have been walking the walk. Three of the four of us are computer scientists and Dave, our physicist, is essentially also one (of course he claims that we’re all physicists). The bulk of our time together, outside of our respective day jobs teaching, is spent leading week-long workshops for faculty – largely focused on the teaching of parallel and distributed programming and computational thinking. Our assertion is this: As computer architectures evolve from single core to multicore to manycore, the computer science curriculum must experience a commensurate single-course to multi-course to many-course evolution in terms of where parallelism is studied.

Thus, you’re probably not surprised we’re saying faculty education is the key way to get from here to there, using as many modes of conveyance as possible. For teaching parallelism in our courses, few of us CS educators have learned what we have needed from our own formal education. We possess a self-taught science/art crafted via the hands-on hard-knock cycles of design, debugging, and despair which provided us with rich learning opportunities. This highlights the goals we have for our students: theory tightly coupled with the pragmatic skills of the practiced practitioner, learned via the cycles of design, debugging, and despair. Note that performance programming is wonderfully resurfacing in importance, for if you don’t need performance, why bother with the complexity of a parallel solution? Just run on your friendly neighborhood SMP or NUMA architecture, which will suffice as a first order solution for many problems. It was performance parallel programming that put the ‘L’ in lunatic fringe, and to raise ‘L’, we will ultimately need to examine the isolated graduate and undergraduate courses and weave the key components of parallelism into the fabric of all computer science courses beginning at the earliest level.

So let’s get specific on possibilities for the first courses at the undergraduate level. The core of CS1 typically starts with the nomenclature, theory, and components of a simple algorithm and a basic block of execution. Flow of control is our next extension: branches, loops, and functions. Parallelism is easily a natural next layer. When we invoke parallelism, we might demonstrate by conjuring with threads and shared memory, since the use of shared memory will not perturb the student’s simple notion of array-like memory. Additionally, the most frequently used shared memory mechanism, OpenMP, allows a gradual move from pure von-Neumann towards “pure” shared memory parallelism. This will cover fine-grain parallelism. A hunger for a different course of studies leads to the course-grained approach of distributed memory parallelism with MPI. Larger scale parallelism is naturally necessarily discovered by students as the problems of interest continue to grow.

The legal battlefield of Amdahl and Gustafson is a good next stop, guiding us into the study of data structures and algorithms via a perilous path littered with algorithms which scale poorly. Unchecked and unplanned parallelism will lead us to throttled resources whether Von Neumann’s bottleneck or the more insidious communication costs incurred when trying to tame a parallel algorithm. Students can learn of dwarvish parallel patterns and associated phenomena such as a sequentially elegant quicksort quickly foundering in the presence of unamortized distributed memory costs.

This is a good time to consider how to squeeze weeks and weeks of new material on parallelism into a semester. Something has to give and something will give, but this is not a new dilemma. It is something we each faced when first crafting what we will cover in a course. It is something we face to a greater or lesser extent every time we re-teach a course given the pace of change in our discipline.

Now it is time for an anecdote. Tom interviewed Dave Paterson as part of the “Teach Parallel” series of interviews. The interview ranged over many topics, one of which was Dave’s fourth edition of “Computer Organization and Design”, which gloriously has parallel topics woven into each chapter. This led to talking with Dave’s publisher about targeting an adaptation of the book towards community colleges, such as Contra Costa College where Tom teaches. The publisher was surprised to learn no dilution of the 703 pages was desired. Tom plans to cherry pick the material to use in his Computer Architecture course, which is a continuation of an experiment he’s been running in all his courses, which allows the entire book is covered, just at varying depths. It is important for Tom to convey how to be a good student, part of which is being able to self-learn from practitioners’ resources. This raises a good point: more textbook support for parallelism is going to make this whole process a heck of a lot easier. Unfortunately, it takes awhile to prime the curricular pump.

Computer architecture has traditionally incorporated elements of parallelism and concurrency; via semaphores and atomic operations, pipelines and multiple functional units, SMP architectures, and instruction and data paths. It has always been the place where the key hardware issues of the current architectures inform the software designed to run on it.

There are no easy answers, but there really are clear steps. We need to help students get to a place where they think of a single processing unit as just a special case of multiple processing units, much like they now learn to view a single variable as a special case of an array.

About the Authors

Thomas Murphy is a professor of Computer Science at Contra Costa College (CCC). He is chair of the CCC Computer Science program and is director of the CCC High Performance Computing Center, which has supported both the Linux cluster administration program and the computational science education program. Thomas has worked with the National Computational Science Institute (NCSI) since 2002. He is one of four members of the NCSI Parallel and Distributed Working group, which presents several three to seven day workshops each year, and helps develop the Bootable Cluster CD software platform, the LittleFe hardware platform, and the CSERD (Computational Science Education Reference Desk) curricular platform.

Paul Gray is an Associate Professor of Computer Science at the University of Northern Iowa. He created the Bootable Cluster CD project (http://bccd.net/) and provides instructional support for the National Computational Sciences Institute summer workshops on Cluster and Parallel Computing. He was SC08 Education Program Chair and serves on the executive committee for the SC07-11 Education Program.

Charlie Peck is the leader of the The Cluster Computing Group (CCG) at Earlham College, a student/faculty research group in the Computer Science department. The CCG is the primary design and engineering team for LittleFe, developers of computational science software, e.g., [email protected], and technical contributors to Paul Gray’s Bootable Cluster CD project. Additionally, Charlie is the primary developer on the LittleFe project.

Dave Joiner is an assistant professor of Computational Mathematics in the New Jersey Center for Science, Technology, and Mathematics Education. The NJCSTME focuses on the training of science and math teachers with an integrated view of modern math, science, and computing. Additionally, Dave has collaborated since 1999 with the efforts of the Shodor Education Foundation, Inc., and the National Computational Science Institute.  He currently serves as a Co-PI on the Computational Science Education Reference Desk, the Pathway of the National Science Digital Library devoted to computational science education.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

NSF Extends Access to Its Leadership Systems Blue Waters & Frontera

December 14, 2018

The National Science Foundation is seeking supplemental requests for access on its leadership-class computers Blue Waters and Frontera to enable "fundamental science and engineering research that would otherwise not be p Read more…

By Staff

CFD on ORNL’s Titan Simulates Cleaner, Low-MPG ‘Opposed Piston’ Engine

December 13, 2018

Pinnacle Engines is out to substantially improve vehicle gasoline efficiency and cut greenhouse gas emissions with a new motor based on an “opposed piston” design that the company hopes will be widely adopted while t Read more…

By Doug Black

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC) is procuring from Atos in two phases over the next year-an Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

AI Can Be Scary. But Choosing the Wrong Partners Can Be Mortifying!

As you continue to dive deeper into AI, you will discover it is more than just deep learning. AI is an extremely complex set of machine learning, deep learning, reinforcement, and analytics algorithms with varying compute, storage, memory, and communications needs. Read more…

IBM Accelerated Insights

4 Ways AI Analytics Projects Fail — and How to Succeed

“How do I de-risk my AI-driven analytics projects?” This is a common question for organizations ready to modernize their analytics portfolio. Here are four ways AI analytics projects fail—and how you can ensure success. Read more…

Nvidia Leads Alpha MLPerf Benchmarking Round

December 12, 2018

Seven months after the launch of its AI benchmarking suite, the MLPerf consortium is releasing the first round of results based on submissions from Nvidia, Google and Intel. Of the seven benchmarks encompassed in version Read more…

By Tiffany Trader

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Nvidia Leads Alpha MLPerf Benchmarking Round

December 12, 2018

Seven months after the launch of its AI benchmarking suite, the MLPerf consortium is releasing the first round of results based on submissions from Nvidia, Goog Read more…

By Tiffany Trader

IBM, Nvidia in AI Data Pipeline, Processing, Storage Union

December 11, 2018

IBM and Nvidia today announced a new turnkey AI solution that combines IBM Spectrum Scale scale-out file storage with Nvidia’s GPU-based DGX-1 AI server to pr Read more…

By Doug Black

Mellanox Uses Univa to Extend Silicon Design HPC Operation to Azure

December 11, 2018

Call it a corollary to Murphy’s Law: When a system is most in demand, when end users are most dependent on the system performing as required, when it’s crunch time – that’s when the system is most likely to blow up. Or make you wait in line to use it. Read more…

By Doug Black

Topology Can Help Us Find Patterns in Weather

December 6, 2018

Topology--the study of shapes--seems to be all the rage. You could even say that data has shape, and shape matters. Shapes are comfortable and familiar concepts, so it is intriguing to see that many applications are being recast to use topology. For instance, looking for weather and climate patterns. Read more…

By James Reinders

Zettascale by 2035? China Thinks So

December 6, 2018

Exascale machines (of at least a 1 exaflops peak) are anticipated to arrive by around 2020, a few years behind original predictions; and given extreme-scale performance challenges are not getting any easier, it makes sense that researchers are already looking ahead to the next big 1,000x performance goal post: zettascale computing. Read more…

By Tiffany Trader

Robust Quantum Computers Still a Decade Away, Says Nat’l Academies Report

December 5, 2018

The National Academies of Science, Engineering, and Medicine yesterday released a report – Quantum Computing: Progress and Prospects – whose optimism about Read more…

By John Russell

Revisiting the 2008 Exascale Computing Study at SC18

November 29, 2018

A report published a decade ago conveyed the results of a study aimed at determining if it were possible to achieve 1000X the computational power of the the Read more…

By Scott Gibson

Quantum Computing Will Never Work

November 27, 2018

Amid the gush of money and enthusiastic predictions being thrown at quantum computing comes a proposed cold shower in the form of an essay by physicist Mikhail Read more…

By John Russell

Cray Unveils Shasta, Lands NERSC-9 Contract

October 30, 2018

Cray revealed today the details of its next-gen supercomputing architecture, Shasta, selected to be the next flagship system at NERSC. We've known of the code-name "Shasta" since the Argonne slice of the CORAL project was announced in 2015 and although the details of that plan have changed considerably, Cray didn't slow down its timeline for Shasta. Read more…

By Tiffany Trader

IBM at Hot Chips: What’s Next for Power

August 23, 2018

With processor, memory and networking technologies all racing to fill in for an ailing Moore’s law, the era of the heterogeneous datacenter is well underway, Read more…

By Tiffany Trader

House Passes $1.275B National Quantum Initiative

September 17, 2018

Last Thursday the U.S. House of Representatives passed the National Quantum Initiative Act (NQIA) intended to accelerate quantum computing research and developm Read more…

By John Russell

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

AMD Sets Up for Epyc Epoch

November 16, 2018

It’s been a good two weeks, AMD’s Gary Silcott and Andy Parma told me on the last day of SC18 in Dallas at the restaurant where we met to discuss their show news and recent successes. Heck, it’s been a good year. Read more…

By Tiffany Trader

US Leads Supercomputing with #1, #2 Systems & Petascale Arm

November 12, 2018

The 31st Supercomputing Conference (SC) - commemorating 30 years since the first Supercomputing in 1988 - kicked off in Dallas yesterday, taking over the Kay Ba Read more…

By Tiffany Trader

TACC’s ‘Frontera’ Supercomputer Expands Horizon for Extreme-Scale Science

August 29, 2018

The National Science Foundation and the Texas Advanced Computing Center announced today that a new system, called Frontera, will overtake Stampede 2 as the fast Read more…

By Tiffany Trader

Leading Solution Providers

SC 18 Virtual Booth Video Tour

Advania @ SC18 AMD @ SC18
ASRock Rack @ SC18
DDN Storage @ SC18
HPE @ SC18
IBM @ SC18
Lenovo @ SC18 Mellanox Technologies @ SC18
NVIDIA @ SC18
One Stop Systems @ SC18
Oracle @ SC18 Panasas @ SC18
Supermicro @ SC18 SUSE @ SC18 TYAN @ SC18
Verne Global @ SC18

HPE No. 1, IBM Surges, in ‘Bucking Bronco’ High Performance Server Market

September 27, 2018

Riding healthy U.S. and global economies, strong demand for AI-capable hardware and other tailwind trends, the high performance computing server market jumped 28 percent in the second quarter 2018 to $3.7 billion, up from $2.9 billion for the same period last year, according to industry analyst firm Hyperion Research. Read more…

By Doug Black

Contract Signed for New Finnish Supercomputer

December 13, 2018

After the official contract signing yesterday, configuration details were made public for the new BullSequana system that the Finnish IT Center for Science (CSC Read more…

By Tiffany Trader

Nvidia’s Jensen Huang Delivers Vision for the New HPC

November 14, 2018

For nearly two hours on Monday at SC18, Jensen Huang, CEO of Nvidia, presented his expansive view of the future of HPC (and computing in general) as only he can do. Animated. Backstopped by a stream of data charts, product photos, and even a beautiful image of supernovae... Read more…

By John Russell

Germany Celebrates Launch of Two Fastest Supercomputers

September 26, 2018

The new high-performance computer SuperMUC-NG at the Leibniz Supercomputing Center (LRZ) in Garching is the fastest computer in Germany and one of the fastest i Read more…

By Tiffany Trader

Houston to Field Massive, ‘Geophysically Configured’ Cloud Supercomputer

October 11, 2018

Based on some news stories out today, one might get the impression that the next system to crack number one on the Top500 would be an industrial oil and gas mon Read more…

By Tiffany Trader

Intel Confirms 48-Core Cascade Lake-AP for 2019

November 4, 2018

As part of the run-up to SC18, taking place in Dallas next week (Nov. 11-16), Intel is doling out info on its next-gen Cascade Lake family of Xeon processors, specifically the “Advanced Processor” version (Cascade Lake-AP), architected for high-performance computing, artificial intelligence and infrastructure-as-a-service workloads. Read more…

By Tiffany Trader

Google Releases Machine Learning “What-If” Analysis Tool

September 12, 2018

Training machine learning models has long been time-consuming process. Yesterday, Google released a “What-If Tool” for probing how data point changes affect a model’s prediction. The new tool is being launched as a new feature of the open source TensorBoard web application... Read more…

By John Russell

The Convergence of Big Data and Extreme-Scale HPC

August 31, 2018

As we are heading towards extreme-scale HPC coupled with data intensive analytics like machine learning, the necessary integration of big data and HPC is a curr Read more…

By Rob Farber

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This