TACC-Intel Symposium Highlights MIC Architecture Developments

By Aaron Dubrow

May 1, 2012

Leading researchers, hardware and software engineers, and high performance computing specialists from around the country attended the TACC-Intel Highly Parallel Computing Symposium on April 10 and 11 at the Texas Advanced Computing Center (TACC) in Austin, Texas. The meeting showcased the experiences of researchers who had ported their scientific computing codes to Intel’s Knights Ferry (KNF) software development platform, — the prototype hardware and software development package for Intel’s Many Integrated Core (MIC) architecture — as well as those working on the single-chip cloud (SCC). 

Over 100 participants from all sectors of the science and technology community attended the symposium and more than 30 stayed on to attend a tutorial on “Preparing for Many Core Programming,” taught by Kent Milfeld, research associate at TACC.  By and large, participants expressed enthusiasm for the forthcoming chips based on their early experiences. The symposium followed a series of Intel-led discussions and training sessions held around the world over the last six months in anticipation of the release of Knights Corner, the first product to use the Intel MIC architecture.

Day One

Two long-time Intel engineers, James Reinders and Tim Mattson, anchored day one of the symposium. Both researchers had worked on ASCI Red, the first supercomputer to break a teraflop of sustained performance in 1996. In addition, Reinders had worked on Knights Corner, the first commercial supercomputing chip to achieve one teraflop, which debuted at the Supercomputing 2011 conference.

The invited speakers steered clear of discussions about the raw performance or architectural details of the Intel MIC chip itself, the ostensible topic of the workshop, but still being closely held by Intel. Instead, they focused on the goals and research processes that led to the development of Intel’s MIC architecture, and the ecosystem of libraries, kernels and programming paradigms that Intel hopes will make its new coprocessors a long-term success in the HPC community.

In his talk, Reinders described the suite of tools and libraries that Intel has developed in anticipation of their manycore chips, as well as their efforts to improve the open standards that enable parallel programming. He expressed the oft-repeated goal, voiced by many Intel staffers, of making the Intel MIC coprocessor usable to researchers right out of the gate. 

Reinders spoke at length about the toolkit developed by Intel called Threading Building Blocks, which assists in the implementation of threading processes, while providing capabilities for task scheduling, synchronization and memory allocation. A thread is the smallest unit of processing that can be scheduled by an operating system. On a multicore system, many threads can run at the same time with each core running a particular thread or task.

“If your program can scale, Intel Threading Building Blocks won’t get in your way,” Reinders said. “There’s little penalty to using it.”

He also noted Intel’s acquisition and subsequent development of Intel Cilk Plus, a sister to Intel Thread Building Blocks for the C and C++ languages. According to Reinders, Intel Cilk Plus offers an easy, reliable way to improve the performance of programs on multicore processors.

Vectorization was a buzzword throughout the symposium. In parallel computing parlance it refers to software programs that are modified to perform the same operation many times simultaneously on a large number of operands. Vectorizing a code allows it to achieve significantly better performance, as much as four times as many operations per clock period on a CPU, and even more on a manycore chip like the future Knights Corner. Reinders was the first, but not the last of the Intel speakers to highlight the importance and value of vectorizing and threading one’s code.

Intel researcher, Tim Mattson, the second invited speaker, works on parallel system design, parallel programming environments and application software that are sometimes “far off the roadmap.” He pushed the date of the first teraflop chip back a few years to his time working on Intel’s 80-core Teraflops Research Chip, a “flop monster” on which the company tested its manycore ideas and architectures.

More than anything related to Intel MIC architecture, Mattson’s talk gave a compelling picture of the way Intel operates, and the concern, care and angst shared by its engineers as they create a new product line for the multi-billion dollar company. According to Mattson, the power problem looms largest in his group’s research, overshadowing even performance gains.

To minimize energy consumption, Mattson and his colleagues conducted a component-by-component power analysis of the experimental chips on which they worked. On the 80-core chip, they found that 28 percent of the power was going into networking capabilities. After tweaking the design, energy consumption was down to 10 percent on the 48-core, version. These two development chips, along with Larrabee, shaped the theoretical basis for Intel’s MIC architecture, and researchers from the three projects comprise the core Intel MIC team.

More than a dozen research groups at Intel are working on the manycore problem from a variety of angles. How many cores can you fit on die? What architecture should many-core chips have? Can such chips be manufactured at reasonable cost? How do you implement a clock infrastructure that can scale?  “It’s the small things that will kill you,” Mattson said.

He also spoke of future-looking ideas such as fine-grained management of the voltage, frequency and power consumption of compute cores that can be controlled by the programmer. Many challenges exist with such a solution, but Mattson’s description of the project provided a glimpse of the kinds of ideas Intel is exploring in their experimental research labs. “The architecture for many-core is still being determined,” he said.

Along the road to debugging and developing software for its manycore offerings, Intel went in an unusual direction. Instead of hiring a large internal software development group, Intel partnered with hundreds of institutions worldwide — among them TACC, many U.S. national laboratories, and top research institutions in the U.S., Europe and Asia — to test-drive the prototype MIC equipment.

“We built the silicon and said, ‘Let’s look to the academic community to do the research on it,’” Mattson explained in his talk.

Since late 2010, this elite group, the Many-Core Application Research Community (MARC), has been experimenting with the Single-Chip Cloud Computer (SCC) experimental processor, a 48-core ‘concept vehicle’ created by Intel Labs as a platform for manycore software research. More recently, researchers have been experimenting with the Knights Ferry hardware as well. Through these initiatives, researchers have implemented solutions that Intel never would have thought of, according to Mattson. “The smartest people in the world don’t necessarily work at your company,” he admitted.

This sentiment served as an apt segue to the paper and poster sections of the symposium, where early users presented results from code porting, testing and performance characterization.

The two-day event pointed to the promise and challenge of implementing existing codes on Intel’s new coprocessor. No absolute performance results were offered, yet scalability plots, comparative studies, and anecdotal experiences provided an overall positive impression of the group’s experiences to date. Most impressively, several groups reported independently that their applications ran right away on Knights Ferry with no or very minor code modifications.

Symposium papers and presentations are available online. A sampling of presentation briefs follows:

Robert Harkness: “Experiences with ENZO on the Intel MIC Co-Processor”

Harness (National Institute for Computational Sciences/Oak Ridge National Laboratory) is one of the creators of ENZO, a large cosmological community code used to study the early ages of the universe. Harkness estimated that it had taken 25 person-years to develop ENZO. The code can’t be abandoned, he said, but no one will pay to rewrite it either. The question he posed was: how to move forward with a large, working science code?

Harkness tested the application on three KNF systems at NICS, which cover most of the choices in basic cluster configurations:

  • An Intel development workstation with two Intel  Xeon  5600 processors and 2 KNF cards.
  • A Cray CX-1 system consisting of a single head node with two Intel Xeon 5600 processors
  • And two compute nodes each containing two Intel Xeon 5600 processors plus one KNF card.
  • An Appro cluster consisting of four compute nodes, each of which contains two Intel Xeon 5600 processors and two KNF cards, for a total of eight KNF cards

He tested the Knights Ferry in both native and offload modes and presented relative scaling results for both.

Harkness said he was personally able to port his code to KNF in one week. In his paper he wrote: “With all these components in place for offload and native builds, the compilation of ENZO-R itself is straightforward. No ENZO-R source code modifications are required to build a functional ENZO-R binary in native mode although, clearly, some source code changes will be necessary to achieve full optimization on the Intel MIC co-processor.”

After the experience, Harkness said he is “bullish on Intel MIC co-processors.” However, he warned the chip community to not waste time on vendor-specific interfaces.

Karl W. Schulz: “Early Experiences Porting Scientific Applications to the Many Integrated Core (MIC) Platform”

Schulz is the director of the scientific applications group at TACC and also chief software architect for The Center for Predictive Engineering and COmputational Sciences (PECOS), a DOE-funded Center of Excellence at The University of Texas at Austin. The group was funded four years ago to develop new uncertainty quantification (UQ) methodologies and apply them to multiscale simulations of atmospheric reentry vehicles.

He and his collaborators tried several codes from their portfolio on Maverick, a Knights Ferry test system with eight nodes and one Knights Ferry per Westmere node, and also built 3rd party libraries to facilitate runs. He presented relative scaling results and anecdotal experiences from several application tests.

Included among these applications was the porting of a quantum chemistry application that had 1.4 million lines of Fortran code.  They also tested the kernel of a Direct Numerical Simulations (DNS) code for turbulent fluid flow, a multi-threaded FFT kernel that exhibited good strong-scaling properties on the Knights Ferry. In the future, the group is targeting very large turbulent flow simulations (10 billion grid points) and proper utilization of the Intel MIC coprocessor will be crucial for achieving that goal.

Schulz who won best paper at the symposium, gave the most comprehensive insights into the process that MARC members have been experiencing over the past few months. To the assembled application developers in the room he counseled that “moving forward, we must take vectorization more seriously.” Overall, he remarked, “It was a positive experience for all of us.”

Ashay Rane: “PerfExpert and MACPO: Which code segments should (not) be ported to MIC?”

Rane is one of the developers of the tools PerfExpert and MACPO, which are designed to analyze the performance and performance characteristics of multi-threaded applications on multicore chips. In his spirited talk he presented a process by which application developers can reliably determine the parts of a code that are good candidates for execution on Intel MIC co-processors. Guidance in this area is certainly needed. Focusing on suitable candidates for porting (either whole functions or loops nests) has the potential to not only greatly reduce the level of human effort, but also to increase the achievable speed-up.

In Rane’s two-step approach, PerfExpert is used to identify the hot spots in an application. MACPO, currently under heavy development by Rane and others, analyzes data access patterns and TLB misses within the hot spots to eliminate those that exhibit irregular access strides, etc. Interestingly, the analysis is not done on the Intel MIC co-processor itself, but on the host, which enables users today, who do not have access to any Intel MIC platform yet, to prepare using this tool.

Three examples were presented as a proof of concept. Those code regions that were recommended by the tools did indeed perform best on the Knights Ferry Software Development Package (SDP).

Sreeram Potluri: “Intra-MIC MPI Communication using MVAPICH2: Early Experience

Sreeram Potluri is a Ph.D. student at The Ohio State University who has been researching and developing the MVAPICH2 library for MPI on InfiniBand and other communications platforms. Potluri describe his group’s work implementing MVAPICH2 for communications between cores on the Knights Ferry platform.

Their initial experiments ported MVAPICH2 directly to the Intel MIC card with no changes. Subsequently, they developed two kinds of optimizations of the library. The first tuned its parameters to take maximal advantage of the memory hierarchy and cache structure on the Knights Ferry card. The second set of optimization extended the work done in the first to use Intel MIC-specific communications protocols to further improve the performance.

The results showed impressive performance gains relative to the baseline case for all varieties of communications operations that Potluri tested. These included direct, point-to-point communications between pairs of cores, and collective communications operations that gather information from many cores together or spread information from one core to all the others.

Future work will take advantage of protocols for communication between Intel MIC cards in the same host, between tasks on the Intel MIC card and tasks on the hosts, and between Intel MIC cards running in different host computers. Potluri received the award for best student paper at the symposium.

Day Two

The day two invited talks focused more explicitly on the Intel MIC coprocessors, the programming models that will drive usage, and the market conditions that inspired Intel to develop the coprocessor.

Scott McMillan, a staff software engineer at Intel, spoke on “Programming Models for Intel Xeon processors and Intel MIC Architecture.” His presentation highlighted an important point: that Xeon and Intel MIC co-processor will work together in most installations.

TACC’s upcoming Stampede system, which will be the first large-scale deployment of the Intel MIC co-processors, has a two-petaflop base system built on the Intel Xeon E5 2600 processors (formerly codenamed Sandy Bridge). The Knights Corner coprocessors will add eight petaflops of expected peak performance. Many of the discussions about parallelism, threading and vectorization apply equally to Intel’s multicore Xeon processor and to the Intel MIC architecture.

The combination of these two architectures lend themselves to a spectrum of programming models and mindsets, which McMillan outlined in his talk, from all multicore, to offload, to all manycore, to symmetric usage of both the Intel Xeon and the Intel MIC architectures.“Which to use depends on the shape and structure of one’s problem,” he said.

The keys to productive performance on the Intel MIC architecture are a matter of common sense, McMillan insisted, and could begin immediately. He offered a few expert tips:

  • Pick the right model for your application.
  • Vectorize your application (today).
  • Parallelize your application (today).
  • Go asynchronous to overlap computation and communication

“Hybrid programming today prepares you for Intel MIC architecture,” he concluded.

Last up was Nash Palaniswamy from Intel’s Technical Computing Group, who offered some perspective on the strategy behind Intel’s manycore architecture.  For more than 15 years, Palaniswamy has been working in the accelerator and coprocessor space. Throughout that time, technologically-driven companies in the oil and gas, financial services and pharmaceutical industries, as well as academic and government researchers, have explored the use of accelerators to find answers quickly, driving strong research in this market.

Intel’s development of manycore chips over several years shows a unique level of commitment to enhancing performance, Palaniswamy said. In 2006, Intel began its accelerator and coprocessor effort under the Intel QuickAssist Technology program, and in 2010 after a thorough analysis of 80 to 90 workloads from customers, they announced a roadmap for manycore. According to Palaniswamy’s assessment, manycore processors are useful for 10 to15 percent of applications today, and this number will continue to grow as programmers increasingly think parallel.

In the road-map, “little cores” — smaller, low-power cores with reduced single thread performance that rely on a high degree of parallelism — co-exist and supplement “big cores” – higher-power, higher-performance cores. This change, Palaniswamy insisted, could be achieved in a way that maximizes code reuse while requiring minimal developer re-training.

“We want to make sure the software that runs today will run 10 years from now with minimal changes,” Palaniswamy said. “Architecture, models and applications evolve. Revolution is unnecessary and unaffordable.”

With scientists continuing to port, test and optimize their software for Knights Ferry, and ultimately Knights Corner, conversations about the Intel MIC architecture will be ongoing for many months to come.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

Thus week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high perform Read more…

By John Russell

SC17 Student Cluster Competition Configurations: Fewer Nodes, Way More Accelerators

November 16, 2017

The final configurations for each of the SC17 “Donnybrook in Denver” Student Cluster Competition have been released. Fortunately, each team received their equipment shipments on time and undamaged, so the teams are r Read more…

By Dan Olds

Student Clusterers Demolish HPCG Record! Nanyang Sweeps Benchmarks

November 16, 2017

Nanyang pulled off the always difficult double-play at this year’s SC Student Cluster Competition. The plucky team from Singapore posted a world record LINPACK, thus taking the Highest LINPACK Award, but also managed t Read more…

By Dan Olds

HPE Extreme Performance Solutions

Harness Scalable Petabyte Storage with HPE Apollo 4510 and HPE StoreEver

As a growing number of connected devices challenges IT departments to rapidly collect, manage, and store troves of data, organizations must adopt a new generation of IT to help them operate quickly and intelligently. Read more…

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s at SC17 in Denver. The previous record, established by German Read more…

By Dan Olds

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

Thus week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projec Read more…

By John Russell

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

HPE Launches ARM-based Apollo System for HPC, AI

November 14, 2017

HPE doubled down on its memory-driven computing vision while expanding its processor portfolio with the announcement yesterday of the company’s first ARM-base Read more…

By Doug Black

OpenACC Shines in Global Climate/Weather Codes

November 14, 2017

OpenACC, the directive-based parallel programming model used mostly for porting codes to GPUs for use on heterogeneous systems, came to SC17 touting impressive Read more…

By John Russell

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

Tensors Come of Age: Why the AI Revolution Will Help HPC

November 13, 2017

Thirty years ago, parallel computing was coming of age. A bitter battle began between stalwart vector computing supporters and advocates of various approaches to parallel computing. IBM skeptic Alan Karp, reacting to announcements of nCUBE’s 1024-microprocessor system and Thinking Machines’ 65,536-element array, made a public $100 wager that no one could get a parallel speedup of over 200 on real HPC workloads. Read more…

By John Gustafson & Lenore Mullin

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Leading Solution Providers

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

How ‘Knights Mill’ Gets Its Deep Learning Flops

June 22, 2017

Intel, the subject of much speculation regarding the delayed, rewritten or potentially canceled “Aurora” contract (the Argonne Lab part of the CORAL “ Read more…

By Tiffany Trader

Graphcore Readies Launch of 16nm Colossus-IPU Chip

July 20, 2017

A second $30 million funding round for U.K. AI chip developer Graphcore sets up the company to go to market with its “intelligent processing unit” (IPU) in Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This