Visit additional Tabor Communication Publications
December 15, 2011
The data deluge in the life sciences is no where more acute than at Chinese genomics powerhouse BGI, which probably sequences more DNA than any other organization in the world. To turn that data into something meaningful for genomic researchers, the institute has begun to employ GPU-accelerated HPC to greatly reduce processing times. In doing so, BGI was able to increase computational throughput by an order of magnitude or more.
At the GPU Technology Conference in Beijing this week, Dr. BingQiang Wang, who heads the HPC group at BGI, described the daunting task of keeping the computational analysis in line with the rapid accumulation of genomic data. At BGI, he says, they are currently able to sequence 6 trillion base pairs per day and have a stored database totaling 20 PB.
The data deluge problem stems from an imbalance between the DNA sequencing technology and computer technology. According to Dr. Wang, using second-generation sequencing machines, genomes can now be mapped 50,000 times faster than just a decade ago. The technology on track to increase approximately 10-fold every 18 months. That is 5 times the rate of Moore's Law, and therein lies the problem.
Obviously it would be impractical to upgrade one's computational infrastructure at that rate, so BGI has turned to NVIDIA GPUs to accelerate the analytics end of the workflow. The architecture of the GPU is particularly suitable for DNA data crunching, thanks to its many simple cores and its high memory bandwidth. Encouraged by the speedup results from similar types of data-parallel programs, BGI developed three genome analysis applications (SOAP3, GSNP, and GAMA) to take advantage of the manycore processing power of the graphics processor.
Developed in 2011, SOAP3 is a GPU-enabled short oligonucleotide alignment program, which aligns short reads against a reference DNA sequence. Thanks to GPU acceleration and some additional memory capacity on the CPU side, SOAP3 is 10 to 30 times faster than its CPU-only predecessor, SOAP2. Running SOAP3 on a Xeon cluster using NVIDIA's Tesla C2070 module, BGI was able to increase performance by 10X for a human genome -- 16X if you discount data loading times from the CPU to the GPU.
The SNP detection tool, GSNP, is GPU-enabled version of SOAPsnp, which find differences of a single nucleotide polymorphism (SNP) in the DNA. SNPs represent genetic variations that can be associated with traits such as disease resistance and drug response. Using GSNP, BGI was able to reduce processing times by about 7X on a typical run -- from 4 days to 14 hours.
GAMA is another genetic variation code, used to estimate the frequencies of gene variants. To compute the frequencies of 1,000 individuals, the original version of GAMA could take a year or more. The GPU-accelerated version could do the same in just two days.
Using GPU-accelerated tools, BGI has been able to reduce the computational part of their standard workflow from about 11 days to 23 hours. But data manipulation times for the storage only decreased at a more modest 50 percent. As a result the data manipulation component of the workflow went from 8 percent of the total time to 25 percent.
The solution was data compression, which was accomplished inside the GPU. BGI implements a Hoffman-based compression algorithm that delivers a compression ration of around 24 percent. The compression rate is a respectable 1 GB/second, with decompression at 1.5 GB/second.
Most of the work described here was done on a BGI-owned 20-GPU server cluster, employing NVIDIA Tesla parts. According to Wang, they have two such clusters, one at their main facility in Shenzhen and another at their Hong Kong office. Storage is provided by Isilon and is made up of multiple systems to house their 20 PB (and growing) database.
Although the workhorse HPC clusters are used for the majority of the BGI analysis tools, more challenging genomics requires a great deal more processing power that a 20-GPU machine. For example, BGI has an application that estimates minor allele frequency (MAF) across a population. MAF refers to the frequency at which the less common alleles occurs in a given population. It's useful for studying genetic variations on a geographical scale.
To estimate the MAF results for even a modest size population is very computationally expensive. For example, using just 1,024 human genomes, it would take 10 years on a single CPU and 0.5 years on a single GPU to generate the MAF results. To make such an application run feasible, one would need thousands or CPUs or hundreds of GPUs.
To achieve an MAF population estimation, BGI teamed up with the Tianjin Supercomputing Center to use their GPU-equipped Tianhe-1A, the top supercomputer in China, and the second most powerful system in the world. Using 256 of the machine's 7,168 GPUs, and employing MPI to communicate between the nodes, the MAF run for that 1,024 population took just 13 hours.
To date, all of GPU porting effort for the analytics applications has been done under CUDA, requiring some serious development effort on the part of the BGI team. Dr. Wang would like to make the development effort more productive, and is intrigued by the recently announced OpenACC programming model, which would allow developers to insert OpenMP-like directives into non-CUDA serial code to expose the parallelism.
Dr. Wang says the GPU application coding has necessitated both computer engineers with GPU expertise as well as biologists who have an intimate knowledge of the genomics application domain. To develop these kinds of applications, the scientists have to work in tandem. Ideally, a developer would encompass both areas of expertise, says Wang. But, he adds, "there is no such person."
May 23, 2013 |
The study of climate change is one of those scientific problems where it is almost essential to model the entire Earth to attain accurate results and make worthwhile predictions. In an attempt to make climate science more accessible to smaller research facilities, NASA introduced what they call ‘Climate in a Box,’ a system they note acts as a desktop supercomputer.
May 22, 2013 |
At some point in the not-too-distant future, building powerful, miniature computing systems will be considered a hobby for high schoolers, just as robotics or even Lego-building are today. That could be made possible through recent advancements made with the Raspberry Pi computers.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.