Lessons learned from Practice & Experience in Advanced Research Computing

If you’ve spent much time cruising employment ads lately, you’ve probably noticed that certain research computing specializations are in high demand. Some university-based centers have had positions open for months; others years. It’s the same in densely-populated communities that compete with regional industries as it is elsewhere. The culture has forced managers and human resource professionals to explore novel ways to fill the prospect pipeline.

Few academic programs provide the practical knowledge necessary to support research computing, so some universities have begun to incorporate advanced skills training into the curriculum. But what do you call the course, and where should it live? If you’re just starting out, there are additional, even more critical things to consider than what to call the course; how you approach this effort could make or break your program.

In smaller schools, if you do manage to get a for-credit course approved, at some point in your future, it’s possible that an administrator with no background in computer science (CS) will judge it, unfairly, based on economics, alone. How many students were served, and could that classroom be better utilized with a course that draws more paying customers? It may inspire scrutiny over the return on investment of your computational cluster, itself. Some are unlikely to prioritize something with such immense power, network and personnel operational costs—especially when budgets are tight and an athletic program might be on the chopping block!

If you envision the course to be structured around the use of a cluster; in other words, if you want to train advanced computer science students on how to run, maintain and optimize workflows for its use, you might draw 5-15 students in a 400-level CS program. Frankly, depending on the size of your resources and data center, that’s about as many as you’d want in a hands-on lab. If you’re only communicating with CS students, it could be called “Distributed and Parallel Computing.” Uber-geeks will understand what they’re in for. But if you do this, don’t encumber a full classroom. Occupy a conference room and pull them into the data center when it’s appropriate. That’ll keep the space auditors happy.

But if you want your program to grow, you should call it something that denotes employment potential and economic prosperity, for example, “Performance Computing for Research and Industry.” That title will resonate favorably with a broader range of prospective stakeholders (and advocates). At the master’s level, plan to train 5-15 the first year, with the goal of doubling that number after two years. A worthy goal for the future would be to attract interdisciplinary students—that’s where the magic happens in terms of scientific and engineering discoveries.

And, if you’re all-in for the academic approach, you might want to create an undergraduate-level, general education course with the same title. This might convene in a small classroom the first year, and move to an auditorium as the number of registrants grows. Open that course to all majors, targeting computationally-curious students, with the lion’s share in CS, engineering, physics, bio, and business (in terms of allocations awarded—those are big users). That could serve as a prerequisite for those who would ultimately pursue any computationally-intensive graduate program.

If this type of course is not established as an undergraduate, interdisciplinary gen-ed course from the beginning, it will invariably get political. Each college will begin to sponsor their own as the demand for computational knowledge increases across disciplines, and departments will make a power grab for seats that would be gained by CS, if that’s where it’s initially housed.

A gen-ed survey course should present applications for HPC in the full spectrum of industries so that more can envision the economic outcomes, in terms of research advances, startups that employ regional workers, collaborating industrial partners, and grant awards. You might incorporate a lecture about cloud computing, and how to determine if it’s a good fit for the workflow. I visited a company recently (1,000 employees; mostly technical) that has a data-intensive mission for which AWS Lambda plus cloud-based GPU computing is performing quite well. They have no interest in supporting HPC. They pay as they go—much like you’d pay for a utility—and they don’t have the capital burden. That wouldn’t work for universities whose mission is to prepare the workforce for a range of occupations, however. Those that do this well support a diverse portfolio of systems and services, including cloud. But understanding when it’s appropriate would prepare students for a cloud-exclusive scenario upon graduation, especially those who take jobs in industries where it’s normalized and they don’t need to employ people who can spell HPC.

There’s a lot to be said for vocational training. In that case, you can bypass academic credit hurdles and politics all together. Most of the senior sysadmins I know—especially generalists capable of handling a range of tasks well—earned their stripes as student employees at one point. But plan to focus on quantity in anticipation of attrition—even among student employees. Students with LinkedIn profiles showing two or three years of in-house experience are getting noticed by talent scouts from the 14 big tech companies that recently waived degree requirements. While the starting salary is tempting to an undergrad who thinks that dropping out would reduce student debt, they need sound advice when it comes to assessing community cost of living comparisons. Also, it’s difficult to return to school once departing; you must soon begin to repay student debt. Many tech companies offer a combination of salary and stock, but that grass isn’t always greener.

Someone recently explained to me that a year after joining a big tech company, he realized that it wasn’t all he had hoped it would be. While doubling his salary, one-third was in the form of stocks that aren’t currently performing well. At the same time, he had to move to a region that costs three times as much as the one he left behind. When comparing lifestyles, he said, “I can’t eat stocks; I live in cramped quarters and there’s nothing left at the end of the month.”

By 2025, three-fourths of the world’s workers will have been born between 1977 and 1995. According to BVK Marketing research, this demographic is impatient; 91 percent expect to change jobs every three years. The gig culture which gained popularity after the 2008 economic downturn affects both employee and employer loyalty. A 2017 Intuit (company behind TurboTax) study found that by 2020, 43 percent of the workforce will be temporary. While I devoted 22 years of service to Illinois’ public university system, and STEM-Trek’s Vice President, David Stack, recently retired after a long career in Wisconsin’s, such commitment and loyalty will be extremely rare in the future.

Quality of life is important to this demographic; if the stars aren’t in alignment where they land, they won’t stick around for long. While a competitive salary is important, university employers who can’t compete with industry would do well to focus on fringes that they may have more control over, such as professional development and related travel, and the ability to work-from-home. In many cases, this is institutionally frowned upon, so it’s incumbent upon technical leadership to drive positive change on their campuses, and offer peer support for such changes to others through professional organizations.

Do you have experience to share, or suggestions that I haven’t thought of?

Many would love to hear from you. Please continue the dialogue during a Practice & Experience in Advanced Research Computing (PEARC19) panel titled, “Stop Chasing Unicorns in the Global Gig Economy,” Wednesday, July 31, 2019 in Chicago. In this panel, five senior research computing center directors will share lessons learned, and road-tested recruitment and retention strategies. PEARC19 is July 28-August 1, 2019 in Chicago, Illinois; early registration ends June 23.

About the Author

HPCwire Contributing Editor Elizabeth Leake is a consultant, correspondent and advocate who serves the global high performance computing (HPC) and data science industries. In 2012, she founded STEM-Trek, a global, grassroots nonprofit organization that supports workforce development opportunities for science, technology, engineering and mathematics (STEM) scholars from underserved regions and underrepresented groups.

As a program director, Leake has mentored hundreds of early-career professionals who are breaking cultural barriers in an effort to accelerate scientific and engineering discoveries. Her multinational programs have specific themes that resonate with global stakeholders, such as food security data science, blockchain for social good, cybersecurity/risk mitigation, and more. As a conference blogger and communicator, her work drew recognition when STEM-Trek received the 2016 and 2017 HPCwire Editors’ Choice Awards for Workforce Diversity Leadership.