Accelerating bioinformatics with hybrid-core computing

By Nicole Hemsoth

April 2, 2012

Advances in sequencing technology have significantly increased data generation and require commensurate computational advances for bioinformatics analysis. Advanced architectures based on reconfigurable computing can reduce application run times from hours to minutes and address problem sizes unattainable with commodity servers. The increased capability also fundamentally improves research quality by allowing more accurate, previously impractical approaches. The use of a hybrid-core computing architecture can be used to solve data-intensive problems of next-generation sequencing analysis like de novo assembly and reference mapping of short-read sequences.

Two important steps in next-generation sequencing analysis are de novo assembly and reference mapping of short-read sequences. Both of these lend themselves to high levels of acceleration with the FPGA-based coprocessor on the Convey systems. Convey’s bioinformatics applications Graph Constructor and BWA can be used in conjunction with, or replace, workflows using standard Velvet[1] and BWA[2], respectively. Graph Constructor reduces not only run time for Velvet, but also reduces memory requirements, making it capable of larger assemblies. Additional performance and workflow optimization includes a fast kmer counting tool that allows quick identification of optimal kmer length and coverage cutoffs for de novo assembly.

Convey’s Hybrid Core Architecture: Fast Compute, Faster Memory

The Convey Hybrid-Core (HC) architecture pairs Intel® x86 microprocessors with a coprocessor comprised of reconfigurable hardware (FPGAs) (Figure 1). Algorithms are implemented as instructions, called personalities, which are loaded onto the FPGAs at runtime to accelerate the applications that use them. Complementing the high performance of the reconfigurable compute elements, Convey’s hybrid-core system also has a highly parallel memory subsystem that is optimized for random accesses. Hybrid-Core Globally Shared Memory (HCGSM) provides a single coherent view of memory to the cache based x86 cores and the high throughput word optimized processing elements on the coprocessor. Bioinformatics applications that experience memory performance limitations on cache-based x86 servers greatly benefit from Convey’s memory architecture.

 The Convey Hybrid-Core Architecture

Figure 1. The Convey Hybrid-Core Architecture. The architecture pairs an Intel x86 host system tightly integrated with a reconfigurable FPGA based coprocessor. Hybrid-Core Globally Shared Memory (HCGSM) provides a single coherent view of memory to the x86 cores and the coprocessor’s highly parallel memory subsystem.

Burrows-Wheeler and de Bruijn Graph Personalities

de Bruijn graph-based assemblers such as Velvet consist of large numbers of relatively simple operations on large randomly accessed data structures. Conventional architectures lack sufficient parallelism in the core processing elements and the memory subsystem to efficiently execute these algorithms. The Convey Graph Constructor implements a high speed de Bruijn graph generator that can reduce the runtime and memory footprint for graph-based genome assembly. It can be run by itself or in conjunction with the Velvet application.

Other algorithms also benefit from massively parallel implementations of application-appropriate-data-type operations, which use logic gates more efficiently than commodity servers. In Burrows-Wheeler mapping applications, significant gains are made in the population bit count required to traverse the compressed reference suffix trees in memory. Convey has developed a personality that improves the performance of the aln step of the BWA processing pipeline, and a version of the open-source BWA application with thread parallelized single- and paired-end processing. The BWA personality has 64 alignment units which each operate on 32 sequences simultaneously, for a total of 2,048 simultaneous alignment operations.

Align and Paired End Performance for Human Genome

For these tests (Figure 1) we aligned paired-end sequence data from the 1000 Genomes project to a human reference (human_g1k_v37), consisting of 84 sequences and a total of 3.1 billion bases. SRR189815_1 and SRR189815_2 are paired-end Illumina reads from individual HG00124 containing a total of 242 million reads, average length 101. The aln steps were run using Convey accelerated BWA on HC-1 and HC-1ex systems, and the paired end step was run on a commodity x86 system using a parallelized version of bwa sampe. The results are compared to BWA 0.5.9 running on the commodity server.

 Align and Paired End Performance for Human Genome

Figure 1. Align and Paired End Performance for Human Genome. The addition of an HC-1ex and the Convey accelerated BWA pipeline to a commodity x86 system delivers 14.7x the throughput of the x86 system alone, processing 120 K reads/sec.

Results

  • Convey’s hardware accelerated aln is 7.5x (HC-1) and 9x (HC-1ex) over a 12-core x86.
  • Thread parallel sampe is 7.3x faster than the standard bwa implementation on the same hardware.
  • The addition of an HC-1ex and the Convey accelerated BWA pipeline to a commodity x86 system delivers 14.7x the throughput of the x86 system alone, processing 120 K reads/sec.

de novo Assembly Parameter Optimization

A feature of the Convey Bioinformatics Suite is the Kmer Counter. The Convey Kmer Counter generates a histogram of kmer coverage counts by hashing kmers in each read sequence. As shown in Figure 2, analysis of the read data aids in selecting optimal kmer length and coverage cutoff values for de novo assembly.

 Histogram of kmer coverage for the Assemblathon data set, as produced by Convey’s Kmer Counter for kmer length 21.

Figure 2. Histogram of kmer coverage for the Assemblathon data set, as produced by Convey’s Kmer Counter for kmer length 21. Statistics for assembly results using the selected coverage cutoffs show the impact of parameter selection on assembly quality, as compared with Velvet’s default setting. Convey’s Kmer Counter can analyze multiple kmer lengths in the same run as shown in the blue overlay.

Results:

  • Higher quality assemblies by using optimal parameters
  • Reduced run time and memory by avoiding poor kmers
  • Extremely efficient compared with VelvetOptimiser
  • Handles longer kmer lengths than Jellyfish
  • Analyzes multiple kmer lengths in a single job

Summary

Convey has developed a personality that improves the performance of the aln step of the BWA processing pipeline, and a parallelized version of the samse and sampe processing steps, that allow Convey systems to dramatically reduce time to solution and increase throughput 15x for a full BWA paired-end pipeline, processing 120 K reads/sec.

We have developed a GraphConstructor personality that interfaces to Velvet and Oases that reduces memory requirements by about 75% and accelerates throughput by an order of magnitude, making it possible to tackle previously impractical genomes with higher quality results. In addition to this work, there are several other projects recently submitted or in progress comparing the performance and accuracy of Convey’s Graph Constructor for genome and transcriptome assemblies, comparing with a range of popular assembly programs.

We are working on additional performance and workflow optimization for these applications, as well as accelerating additional applications.

References and Acknowledgements

  1. “Velvet: Algorithms for de novo Short Read Assembly Using de Bruijn Graphs”, Daniel R. Zerbino and Ewan Birney, EMBL-European Bioinformatics Institute, Genome Res. 18 (2008) 821.
  2. “Fast and Accurate Short Read Alignment with Burrows-Wheeler Transform”, Heng Li and Richard Durban, Wellcome Trust Sanger Institute, Bioinformatics 25 (2009) 1754.
  3. “Metagenomic discovery of biomass-degrading genes and genomes from cow rumen”, Hess ,et al, Science 331 (2011) 463.
  4. “Efficient Graph Based Assembly of Short-Read Sequences on a Hybrid Core Architecture“ Alex Sczyrba, Abhishek Pratap, Shane Canon, James Han, Alex Copeland, Zhong Wang, DOE Joint Genome Institute User Meeting, March 2011.

For more information go to Convey Computer.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visitors to the Colorado Convention Center in Denver for the larg Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some cases, city managers didn’t even know existed. Speaking Read more…

By Doug Black

HPE Extreme Performance Solutions

Harness Scalable Petabyte Storage with HPE Apollo 4510 and HPE StoreEver

As a growing number of connected devices challenges IT departments to rapidly collect, manage, and store troves of data, organizations must adopt a new generation of IT to help them operate quickly and intelligently. Read more…

SC17 Student Cluster Competition Configurations: Fewer Nodes, Way More Accelerators

November 16, 2017

The final configurations for each of the SC17 “Donnybrook in Denver” Student Cluster Competition have been released. Fortunately, each team received their equipment shipments on time and undamaged, so the teams are r Read more…

By Dan Olds

SC Bids Farewell to Denver, Heads to Dallas for 30th

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visit Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some Read more…

By Doug Black

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

HPE Launches ARM-based Apollo System for HPC, AI

November 14, 2017

HPE doubled down on its memory-driven computing vision while expanding its processor portfolio with the announcement yesterday of the company’s first ARM-base Read more…

By Doug Black

OpenACC Shines in Global Climate/Weather Codes

November 14, 2017

OpenACC, the directive-based parallel programming model used mostly for porting codes to GPUs for use on heterogeneous systems, came to SC17 touting impressive Read more…

By John Russell

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This