HPCwire recently had the opportunity to talk with Greg Wilson, author of the book “Data Crunching.” In this interview, he describes his “Software Carpentry” course, a primer for scientists and engineers who are not professional software developers. This audience often spends a lot of time developing, debugging and maintaining programs, but doesn't have the computer science background to do those tasks effectively or efficiently. Wilson teaches them about the software development process as well as basic proficiency in higher-level programming languages like MATLAB and Python. This allows scientists and engineers to increase quality and productivity, while at the same time taking advantage of the latest advancements in hardware.

HPCwire: What is the history behind the “Software Carpentry” course you've developed and who is the course targeted for?

Wilson: The idea for the course goes all the way back to work I did at the Edinburgh Parallel Computing Centre in the late 1980s and early 1990s. My job was to help parallelize scientific software, but after a while, I realized the real problem was that most scientists and engineers simply didn't know how to develop code efficiently: they'd never heard of version control, they didn't know how to test things, they did repetitive tasks by hand instead of writing little scripts for them, and so on.

Starting in about 1997, Brent Gorda (now at Lawrence Livermore National Laboratory) and I began teaching short courses on basic software development skills at Los Alamos National Laboratory. Response was mixed: some participants, particularly younger ones, ate it up, while others felt that what they'd been doing for the last twenty years was good enough. Over the next three or four years, we figured out which parts of standard industrial software development models actually made sense for HPC, and which didn't. I then taught the course at a few other places (most recently the University of Toronto), then received a grant from an open source organization called the Python Software Foundation to tidy up the notes and make them freely available over the Web.

In its current form, the course is aimed at people who have backgrounds in science or engineering, but not computer science, and who now find themselves having to write or maintain complex programs. It covers basic tools, including the Unix shell, make and version control, introduces students to the Python scripting language, then goes on to look at data crunching, integration, web programming, security, best practices for small teams and other practical issues. All the material is freely available at our website, http://swc.scipy.org, and like most open source projects, we'd welcome more contributions.

HPCwire: How much time would it take for the average non-professional programmer to take the entire course?

Wilson: It takes me about an hour to deliver each of the two dozen or so lectures, and I expect that students will spend two to three hours on exercises for each. When I teach the course again this coming winter, I'll be recording the lectures and posting the MP3s to the web. Then people outside Toronto who want to follow along would be very welcome to — we had about 70 out-of-towners downloading the lectures and chipping in on the mailing list last fall.

HPCwire: What is the difference between “Software Carpentry” and software engineering?

Wilson: Scale. This course doesn't try to teach people how to build the electronic equivalent of the Channel Tunnel; it's more about drywalling the basement. If you're developing code on your own for your thesis, or working in a team of half a dozen people for a year or two, most of what you need to know is in this course.

HPCwire: As software developers, what do scientists and engineers seem to lack most?

Wilson: A mental picture of how they ought to be working. Once you've seen a project that has version control, bug tracking, and a few other odds and ends integrated together, and whose members are actually writing unit tests, it's obvious that you should be working that way. The problem is most scientists and engineers never see a project that's being run that way. There are a few exceptions — high-energy physics, for example, has a much better track record than most disciplines — but those are exceptions.

HPCwire: Can you give us some of the highlights of the course you've developed?

Wilson: The highlight for me is the last two lectures, which cover the development processes and tools that small teams should use. In a way, everything before that is plowing the field — those lectures are the seeds we really want to plant in students' minds.

What students seem to appreciate most, though, is the discussion of the interplay between quality and productivity. A lot of computational science these days isn't really science: few computational “experiments” are reproducible in any sense, and only a handful of scientists and engineers have any idea how reliable their programs are. I'm convinced that sooner or later this is going to lead to some sort of “computational thalidomide,” but as long as journal and conference editors don't ask questions about the software used to produce results, people who invest time in QA are at a disadvantage compared to people who don't. I've therefore learned to emphasize that investing in quality up front makes programmers more productive: if you do more testing, you'll have to do less debugging, and you'll get more results published per unit of time. That is something that grad students understand.

HPCwire: In the HPC space, there's a great deal of reliance on C and Fortran (with MPI). Yet most HPC end-users are not professional software developers and would probably be more productive by using higher-level programming environments like MATLAB, Python, etc. What's your perspective on this?

Wilson: I completely agree, and I think that people like Paul Dubois (formerly at LLNL) have demonstrated how much more science you can get done if you write most of your application in a high-level language, then rewrite the performance-critical bits in something like C once you're sure they're actually performance-critical.

You can actually put some math behind this if you want to. Everyone in HPC knows Amdahl's Law: if the serial fraction of your code is 's', and the sequential runtime is 't', then the runtime on 'p' processors is st + (1-s)t/p, and the speedup as 'p' goes to infinity is limited by '1/s'. What most people fail to recognize is that Amdahl was only thinking of a single run of the program — which isn't surprising, since he was a hardware engineer. In the real world, programs have to be developed, debugged and maintained, and none of these processes is sped up very much by parallel hardware.

So, if you let 'T' be the program's total runtime over its life, and 'D' be its development time, the lifetime speedup on 'p' processors is sT + (1-s)T/p + D, and the speedup with infinite hardware is limited to 1/(s + D/T). This says that the more time you spend developing rather than running, the less advantage you're actually taking of all that hardware you spent so much money on. If students come out of the course understanding this, and knowing how to reduce 'D', I'll be happy.

HPCwire: How do you think computer hardware advances — especially the rise of multi-core processors — will affect programming tools and the way programming is done?

Wilson: As things stand, they're going to increase development time for all but embarrassingly-parallel applications. For example, I've been writing parallel programs for twenty-one years now, and still haven't found a parallel debugger that actually helps. New debugging and analysis techniques, like those discussed in Andreas Zeller's recent book [1], are going to be essential, but unless scientists and engineers master the tools and techniques we already have, all we're going to see is more and more wasted cycles.

[1] Andreas Zeller: “Why Programs Fail”. Morgan Kaufmann, 2005.

—–

Greg Wilson holds a Ph.D. in Computer Science from the University of Edinburgh, and has worked on high performance scientific computing, data visualization, and computer security. His most recent book is “Data Crunching” (Pragmatic Bookshelf, 2005), and he is now developing a course on basic software development skills for scientists and engineers. Greg is a freelance consultant, a contributing editor with “Doctor Dobb's Journal” and an adjunct professor in Computer Science at the University of Toronto.