The Leading Source for Global News and Information Covering the Ecosystem of High Productivity Computing
From the Editor | Main Blog Index
July 08, 2009
There were a couple of stories floating around the Intertubes in the past week or so that reminded me of how little we know about large classes of HPC applications. That's not a good thing.
The first story is about the arrest of former Goldman Sachs computer programmer, Sergey Aleynikov, who allegedly made off with proprietary trading software the firm used to execute high volume, low latency automated trading. The news was covered in Bloomberg and elsewhere, and is a compelling tale of espionage in the financial services industry. Here's the money quote, so to speak, from the Bloomberg report:
The proprietary code lets the firm do "sophisticated, high-speed and high-volume trades on various stock and commodities markets," prosecutors said in court papers. The trades generate "many millions of dollars" each year.
The second story concerns espionage too -- in this case, the more old-fashioned kind. Apparently the National Security Agency (NSA) is planning to construct a $2 billion dollar datacenter in Utah that will eventually consume 65 megawatts to power new supercomputers. The agency's current facility in Fort Meade, Maryland, is already power-constrained, preventing the NSA from installing any more supers at that site. The Salt Lake City Tribune reported that the new Utah datacenter will be used to support the NSA's intelligence-gathering mission. The Tribune used congressional budget documents to squeeze a bit more detail from the story:
The supercomputers in the center will be part of the NSA's signal intelligence program, which seeks to "gain a decisive information advantage for the nation and our allies under all circumstances" according to the documents.
Of course, the speculation is that the NSA is using these supers to perform domestic spying -- a program begun under the Bush administration and now being continued under the Obama regime. The Fort Meade machine is a Cray "Black Widow" super, which is reportedly sifting through emails and phone conversations to find out who's been naughty and nice.
The nexus of these two stories is that secrecy prevents the public from knowing the full breadth of HPC applications. In the case of the government, it's for national security reasons; for financial institutions, it's to maintain competitive advantage. But it's not just the financial industry and government intelligence domains (although these two areas are probably the most discrete in regard to technology transparency). You'll notice, for example, that the energy industry, commercial biotech companies, and automobile/aerospace manufacturers aren't giving public tours of their HPC datacenters either.
Even the TOP500 list reflects this secretive nature. The list is quickly becoming an anonymous record of supercomputing, where many of the users are only listed generically (for example, Financial Institution, Government, IT Provider, Semiconductor Company, and so on). This is especially true as you move toward the end of the list, where there are fewer public institutions. On the June 2009 TOP500 list, 78 of the bottom 100 supercomputers are listed anonymously.
By contrast, HPC being performed by national labs, supercomputing centers, and academic institutions is highly publicized, and tends to be very visible on the TOP500 list. These organizations are constantly on the prowl for grants, funding and other sorts of collaboration, so it pays for them to advertise what they're up to. Plus researchers tend to be a talkative bunch anyway. I suspect this is the reason that the public generally only associates supercomputing with applications like climate modeling and looking for galactic black holes.
What's the result of all this secrecy? Besides giving the public a skewed view of the industry, it also makes the technology invisible to a larger number of developers. Consider that most HPC apps are still implemented in legacy languages like Fortran and C, while "public" applications for personal computers or the Web are using more modern software frameworks, like Java, .NET, Python, etc. Even though HPC is not a volume industry in terms of software licenses, if more codes were public, you'd probably see a much more rapid development of libraries and tools (which is one reason why CUDA software has developed so quickly). Keeping software in silos makes for a lousy ecosystem.
The other aspect to secrecy is that it encourages the kind of bad behavior that Aleynikov and the US government are being accused of. There's nothing inherently wrong with protecting state secrets and proprietary IP, but eventually the whole model can become self defeating. Consider this: arguably, two of the biggest catastrophes of this young 21st century are that of US intelligence regarding 9/11 and Iraq and the collapse of the financial industry. Both institutions relied on keeping information siloed to such an extent that even the institutions themselves couldn't explain the data. And when the secrets are lies, nobody will know until it's too late.
Posted by Michael Feldman - July 8 @ 5:37PM
(Digg, Technorati, more)
Platform HPC Workgroup Manager
Platform HPC Workgroup Manager integrates all the cluster productivity tools you need to deploy, run and manage your HPC environment.
Michael Feldman is the editor of HPCwire.
More Michael Feldman
HPC = servers but a lot more by lawries
Lena by Nastyanna
Lena by Nastyanna
Multi core deployment becomes a memory game by truly64
Re: Venture Capital Drought? Not So Much. by Ron Van Holst
Re: AMD Confirms 12-Core Opteron Production by Nastyanna
Re: Cray Corrals Big Defense Deal by Nastyanna
Re: Podcast: Cray Awarded Defense Deal; SGI Makes Storage Buy; IBM Invents New Algorithm by Nastyanna
Painful Truth by jeffrey.mcallister
SGI = graphics + HPC by johnbarr
HPC = servers but a lot more by truly64
Oracle SPARC != Fujitsu SPARC by Alan M. Feldstein
Sun & HPC != Oracle & HPC by Merblich
a third vendor for lossless low latency 10GbE fabric by lee.fisher@hp.com
Response to GAH by KevinButerbaugh
Response to KevinButerbaugh by GAH
Response to KevinButerbaugh by GAH
Response to GAH by KevinButerbaugh
Response to bdrupp by KevinButerbaugh
Climate Crisis and Exaflops by bdrupp
Climate Crisis and Exaflops by John Hules
Climate Crisis and Exaflops by GAH
Climate Crisis by KevinButerbaugh
IBM "Brain Simulation" article is not properly presented. by Merritt
563 out of 1206 by vvolkov
Little Iron by gadunk
At least it's not "cloud" by KevinButerbaugh
Native QPI Interface? by commike
Mmmmmm by hellcats
New transistorized IC chip scales. by symmecon
Itanium at IDF by Alan M. Feldstein
Communication time by jnapper
"The financial meltdown and computing" by donpellegrino
Human Models by mdgabriel
High-End SPARC Chip for Scientific Applications by Alan M. Feldstein
RapidMind by Mr LolO
Rapidmind by dminor
Longer run times by JohnWest
re: Algo trading Angst by jshore
Results of Testing by in_the_crease
For the first time in 62 years, the four-man Olympics bobsled team from the US captured the gold medal, setting a course world record in the process. The winning bobsled had some state-of-the-art engineering behind it, including CFD software from Exa Corporation. As it turned out, that software may have proved to be the margin of difference in the race.
Read More...
Cray and Microsoft Research partner on cloud computing project; IBM donates a POWER7-based supercomputer to Rice University; and the Kavli Foundation hosts a dialogue on the convergence of nanoscience and neuroscience. We recap those stories and more in our weekly wrapup.
Read More...
Multicore software specialist Fixstars Corporation has released Yellow Dog Enterprise Linux (YDEL) for CUDA, the first commercial Linux distribution for GPU computing. The OS is aimed at HPC customers using NVIDIA GPU hardware to accelerate their vanilla Linux clusters, and is designed to lower the overall cost of system deployment, the idea being to bring these still-exotic systems into the mainstream.
Read More...
Mar 09 | Free Software Magazine | Data-driven computing will need open software. Read more...
Mar 09 | Bio-IT World | Tahoe Informatics founder eyes GPUs, CUDA software. Read more...
Mar 08 | Sporting Life | Formula One engineers differ on benefits of CFD. Read more...
Mar 08 | InfoWorld | AMD offers up 48-core server prize. Read more...
Mar 04 | Linux Magazine | The new x86 multicore offerings could portend big changes for HPC platforms. Read more...
Jan 12 | | In-depth look at vSMP Foundation server virtualization technology, technical implementation, use cases and capabilities. The technical whitepaper provides an architectural overview and details on the three vSMP Foundation products: vSMP Foundation for SMP, vSMP Foundation for Cluster and vSMP Foundation for Cloud.
Jan 18 | | This white paper discusses Gore’s copper cable assemblies, and how they continue to exceed the standards for providing reliable, cost-effective solutions for high-performance computer applications.
Join this online panel discussion for live Q&A with leading industry experts, analysts, and end-users to discuss the latest innovations, best practices, barriers to implementation, and measurable benefits of server virtualization with a particular focus on today's real world solutions.
Learn about scalable fault-tolerant architectures and examples of energy efficient and scalable supercomputing clusters using dual QDR InfiniBand to combine capacity computing with network failover capabilities with the help of programming languages such as MPI and a robust Linux cluster management package.
LIVE@SCO9: The IBM team discusses new innovations in hardware, software and services that help clients better understand their workloads and get insight from their R&D efforts. Technology demonstrations include the soon-to-be-released Power7 HPC processor, the DCS990 system with 2.4 petabytes of storage, the xCAT management tool, secure HPC cloud computing and more. Winners of two HPCwire Readers' and Editors’ Choice Awards! Take the IBM virtual tour at SC09 or more information go online to: http://www-03.ibm.com/systems/deepcomputing/sc09.html