HPCwire: Software Carpentry for Scientists and Engineers

The Leading Source for Global News and Information Covering the Ecosystem of High Productivity Computing

Leading HPC
Solution Providers

HPCwire >> Features

August 04, 2006

Software Carpentry for Scientists and Engineers

HPCwire recently had the opportunity to talk with Greg Wilson, author of the book "Data Crunching." In this interview, he describes his "Software Carpentry" course, a primer for scientists and engineers who are not professional software developers. This audience often spends a lot of time developing, debugging and maintaining programs, but doesn't have the computer science background to do those tasks effectively or efficiently. Wilson teaches them about the software development process as well as basic proficiency in higher-level programming languages like MATLAB and Python. This allows scientists and engineers to increase quality and productivity, while at the same time taking advantage of the latest advancements in hardware.

HPCwire: What is the history behind the "Software Carpentry" course you've developed and who is the course targeted for?

Wilson: The idea for the course goes all the way back to work I did at the Edinburgh Parallel Computing Centre in the late 1980s and early 1990s. My job was to help parallelize scientific software, but after a while, I realized the real problem was that most scientists and engineers simply didn't know how to develop code efficiently: they'd never heard of version control, they didn't know how to test things, they did repetitive tasks by hand instead of writing little scripts for them, and so on.

Starting in about 1997, Brent Gorda (now at Lawrence Livermore National Laboratory) and I began teaching short courses on basic software development skills at Los Alamos National Laboratory. Response was mixed: some participants, particularly younger ones, ate it up, while others felt that what they'd been doing for the last twenty years was good enough. Over the next three or four years, we figured out which parts of standard industrial software development models actually made sense for HPC, and which didn't. I then taught the course at a few other places (most recently the University of Toronto), then received a grant from an open source organization called the Python Software Foundation to tidy up the notes and make them freely available over the Web.

In its current form, the course is aimed at people who have backgrounds in science or engineering, but not computer science, and who now find themselves having to write or maintain complex programs. It covers basic tools, including the Unix shell, make and version control, introduces students to the Python scripting language, then goes on to look at data crunching, integration, web programming, security, best practices for small teams and other practical issues. All the material is freely available at our website, http://swc.scipy.org, and like most open source projects, we'd welcome more contributions.

HPCwire: How much time would it take for the average non-professional programmer to take the entire course?

Wilson: It takes me about an hour to deliver each of the two dozen or so lectures, and I expect that students will spend two to three hours on exercises for each. When I teach the course again this coming winter, I'll be recording the lectures and posting the MP3s to the web. Then people outside Toronto who want to follow along would be very welcome to -- we had about 70 out-of-towners downloading the lectures and chipping in on the mailing list last fall.

HPCwire: What is the difference between "Software Carpentry" and software engineering?

Wilson: Scale. This course doesn't try to teach people how to build the electronic equivalent of the Channel Tunnel; it's more about drywalling the basement. If you're developing code on your own for your thesis, or working in a team of half a dozen people for a year or two, most of what you need to know is in this course.

HPCwire: As software developers, what do scientists and engineers seem to lack most?

Wilson: A mental picture of how they ought to be working. Once you've seen a project that has version control, bug tracking, and a few other odds and ends integrated together, and whose members are actually writing unit tests, it's obvious that you should be working that way. The problem is most scientists and engineers never see a project that's being run that way. There are a few exceptions -- high-energy physics, for example, has a much better track record than most disciplines -- but those are exceptions.

HPCwire: Can you give us some of the highlights of the course you've developed?

Wilson: The highlight for me is the last two lectures, which cover the development processes and tools that small teams should use. In a way, everything before that is plowing the field -- those lectures are the seeds we really want to plant in students' minds.

What students seem to appreciate most, though, is the discussion of the interplay between quality and productivity. A lot of computational science these days isn't really science: few computational "experiments" are reproducible in any sense, and only a handful of scientists and engineers have any idea how reliable their programs are. I'm convinced that sooner or later this is going to lead to some sort of "computational thalidomide," but as long as journal and conference editors don't ask questions about the software used to produce results, people who invest time in QA are at a disadvantage compared to people who don't. I've therefore learned to emphasize that investing in quality up front makes programmers more productive: if you do more testing, you'll have to do less debugging, and you'll get more results published per unit of time. That is something that grad students understand.

HPCwire: In the HPC space, there's a great deal of reliance on C and Fortran (with MPI). Yet most HPC end-users are not professional software developers and would probably be more productive by using higher-level programming environments like MATLAB, Python, etc. What's your perspective on this?

Wilson: I completely agree, and I think that people like Paul Dubois (formerly at LLNL) have demonstrated how much more science you can get done if you write most of your application in a high-level language, then rewrite the performance-critical bits in something like C once you're sure they're actually performance-critical.

You can actually put some math behind this if you want to. Everyone in HPC knows Amdahl's Law: if the serial fraction of your code is 's', and the sequential runtime is 't', then the runtime on 'p' processors is st + (1-s)t/p, and the speedup as 'p' goes to infinity is limited by '1/s'. What most people fail to recognize is that Amdahl was only thinking of a single run of the program -- which isn't surprising, since he was a hardware engineer. In the real world, programs have to be developed, debugged and maintained, and none of these processes is sped up very much by parallel hardware.

So, if you let 'T' be the program's total runtime over its life, and 'D' be its development time, the lifetime speedup on 'p' processors is sT + (1-s)T/p + D, and the speedup with infinite hardware is limited to 1/(s + D/T). This says that the more time you spend developing rather than running, the less advantage you're actually taking of all that hardware you spent so much money on. If students come out of the course understanding this, and knowing how to reduce 'D', I'll be happy.

HPCwire: How do you think computer hardware advances -- especially the rise of multi-core processors -- will affect programming tools and the way programming is done?

Wilson: As things stand, they're going to increase development time for all but embarrassingly-parallel applications. For example, I've been writing parallel programs for twenty-one years now, and still haven't found a parallel debugger that actually helps. New debugging and analysis techniques, like those discussed in Andreas Zeller's recent book [1], are going to be essential, but unless scientists and engineers master the tools and techniques we already have, all we're going to see is more and more wasted cycles.

[1] Andreas Zeller: "Why Programs Fail". Morgan Kaufmann, 2005.

-----

Greg Wilson holds a Ph.D. in Computer Science from the University of Edinburgh, and has worked on high performance scientific computing, data visualization, and computer security. His most recent book is "Data Crunching" (Pragmatic Bookshelf, 2005), and he is now developing a course on basic software development skills for scientists and engineers. Greg is a freelance consultant, a contributing editor with "Doctor Dobb's Journal" and an adjunct professor in Computer Science at the University of Toronto.

Article Tools

Share & Save Options

Discussion

There are 0 discussion items posted.

Join the Discussion

You must be a registered user in order to comment on HPCWire stories.

Members enjoy:

	Access to Special Content, including Podcasts, Webcasts, and Whitepapers and more!
	Ability to take an active role in the discussions that are shaping the HPC technology discipline!
	Special offers available only to HPCWire members!

Become a Registered User Today!

Registered Users Log in Here to Comment

Email Address:	Password (case sensitive)

Remember me	Forgot Password?

RSS Feeds

Feeds by Topic

Feeds by Industry

Feeds by Content Type

Top Headlines

25 Years of Conventional Evaluation of Data Analysis Proves Worthless In Practice

Sep 05 | Uppsala University | Swedish researchers are revealing that "intelligent" computer-based methods for classifying patient samples is worthless when it comes to practical problems. Read more...

Three Dimensional Scans Could Revolutionise Brain Surgery

Sep 03 | Telegraph.co.uk | A new form of three dimensional scans could revolutionise brain surgery within a year, doctors claim. Read more...

Marching Penguins: Monitoring Your HPC Cluster

Sep 03 | Linux Magazine | In HPC, most attention is paid to utilization and performance, rather than service availability and problem notification. This article focuses on the latter Read more...

Big Data: Welcome to the Petacentre

Sep 03 | Nature News | What does it take to store bytes by the tens of thousands of trillions? Read more...

UD Scientist Finds Quicker Computer On-Ramp

Sep 01 | Delaware Online | In the world of supercomputer-powered science, speed is everything, and an open road can lead to the promised land. Read more...

Read more headlines...

Featured Whitepapers

SUSE® Linux Enterprise Server for High Performance Computing

Sep 05 | | The excellent scalability features of Linux, in addition to robust security and performance makes it an excellent choice for server systems, especially in the high performance computing area.

“Going Parallel - An Implementation Guide”

Sep 01 | | The paper outlines the basic steps and tools involved in the process of migrating a desktop application to a parallel environment.

Improving Performance and Manageability for Seismic Processing and Imaging Applications with Parallel Storage

Jun 05 | | As pressure increases on the upstream seismic processing community to deliver ever-higher levels of productivity and efficiency, a new generation of storage solutions will be required that allow the maximum utilisation of high-performance computing (HPC) Linux cluster resources, together with the minimum of management overhead.

View the White Paper Library

Multimedia

Video White Paper: Architecting a Better Network Storage Solution

BlueArc's Titan architecture represents an evolutionary step in file servers by creating a hardware-based file system that can scale bandwidth, IOPS, and overall data capacity well beyond conventional software-based devices. With its ability to virtualize a massive storage pool of up to four usable petabytes of tiered storage, Titan can scale with growing data requirements, offering a competitive advantage for businesses, researchers, or other enterprises seeking to better manage data growth while still ensuring optimal performance.

Podcast: Interview with Ben Bennett of ClearSpeed Technology

Today, HPC organizations are requiring substantially more floating point performance to solve real-world problems. In this podcast, Ben Bennett, ClearSpeed General Manager, discusses how acceleration technology can improve the overall performance of standard x86-based systems...