TACC-Intel Symposium Highlights MIC Architecture Developments

By Aaron Dubrow

May 1, 2012

Leading researchers, hardware and software engineers, and high performance computing specialists from around the country attended the TACC-Intel Highly Parallel Computing Symposium on April 10 and 11 at the Texas Advanced Computing Center (TACC) in Austin, Texas. The meeting showcased the experiences of researchers who had ported their scientific computing codes to Intel’s Knights Ferry (KNF) software development platform, — the prototype hardware and software development package for Intel’s Many Integrated Core (MIC) architecture — as well as those working on the single-chip cloud (SCC). 

Over 100 participants from all sectors of the science and technology community attended the symposium and more than 30 stayed on to attend a tutorial on “Preparing for Many Core Programming,” taught by Kent Milfeld, research associate at TACC.  By and large, participants expressed enthusiasm for the forthcoming chips based on their early experiences. The symposium followed a series of Intel-led discussions and training sessions held around the world over the last six months in anticipation of the release of Knights Corner, the first product to use the Intel MIC architecture.

Day One

Two long-time Intel engineers, James Reinders and Tim Mattson, anchored day one of the symposium. Both researchers had worked on ASCI Red, the first supercomputer to break a teraflop of sustained performance in 1996. In addition, Reinders had worked on Knights Corner, the first commercial supercomputing chip to achieve one teraflop, which debuted at the Supercomputing 2011 conference.

The invited speakers steered clear of discussions about the raw performance or architectural details of the Intel MIC chip itself, the ostensible topic of the workshop, but still being closely held by Intel. Instead, they focused on the goals and research processes that led to the development of Intel’s MIC architecture, and the ecosystem of libraries, kernels and programming paradigms that Intel hopes will make its new coprocessors a long-term success in the HPC community.

In his talk, Reinders described the suite of tools and libraries that Intel has developed in anticipation of their manycore chips, as well as their efforts to improve the open standards that enable parallel programming. He expressed the oft-repeated goal, voiced by many Intel staffers, of making the Intel MIC coprocessor usable to researchers right out of the gate. 

Reinders spoke at length about the toolkit developed by Intel called Threading Building Blocks, which assists in the implementation of threading processes, while providing capabilities for task scheduling, synchronization and memory allocation. A thread is the smallest unit of processing that can be scheduled by an operating system. On a multicore system, many threads can run at the same time with each core running a particular thread or task.

“If your program can scale, Intel Threading Building Blocks won’t get in your way,” Reinders said. “There’s little penalty to using it.”

He also noted Intel’s acquisition and subsequent development of Intel Cilk Plus, a sister to Intel Thread Building Blocks for the C and C++ languages. According to Reinders, Intel Cilk Plus offers an easy, reliable way to improve the performance of programs on multicore processors.

Vectorization was a buzzword throughout the symposium. In parallel computing parlance it refers to software programs that are modified to perform the same operation many times simultaneously on a large number of operands. Vectorizing a code allows it to achieve significantly better performance, as much as four times as many operations per clock period on a CPU, and even more on a manycore chip like the future Knights Corner. Reinders was the first, but not the last of the Intel speakers to highlight the importance and value of vectorizing and threading one’s code.

Intel researcher, Tim Mattson, the second invited speaker, works on parallel system design, parallel programming environments and application software that are sometimes “far off the roadmap.” He pushed the date of the first teraflop chip back a few years to his time working on Intel’s 80-core Teraflops Research Chip, a “flop monster” on which the company tested its manycore ideas and architectures.

More than anything related to Intel MIC architecture, Mattson’s talk gave a compelling picture of the way Intel operates, and the concern, care and angst shared by its engineers as they create a new product line for the multi-billion dollar company. According to Mattson, the power problem looms largest in his group’s research, overshadowing even performance gains.

To minimize energy consumption, Mattson and his colleagues conducted a component-by-component power analysis of the experimental chips on which they worked. On the 80-core chip, they found that 28 percent of the power was going into networking capabilities. After tweaking the design, energy consumption was down to 10 percent on the 48-core, version. These two development chips, along with Larrabee, shaped the theoretical basis for Intel’s MIC architecture, and researchers from the three projects comprise the core Intel MIC team.

More than a dozen research groups at Intel are working on the manycore problem from a variety of angles. How many cores can you fit on die? What architecture should many-core chips have? Can such chips be manufactured at reasonable cost? How do you implement a clock infrastructure that can scale?  “It’s the small things that will kill you,” Mattson said.

He also spoke of future-looking ideas such as fine-grained management of the voltage, frequency and power consumption of compute cores that can be controlled by the programmer. Many challenges exist with such a solution, but Mattson’s description of the project provided a glimpse of the kinds of ideas Intel is exploring in their experimental research labs. “The architecture for many-core is still being determined,” he said.

Along the road to debugging and developing software for its manycore offerings, Intel went in an unusual direction. Instead of hiring a large internal software development group, Intel partnered with hundreds of institutions worldwide — among them TACC, many U.S. national laboratories, and top research institutions in the U.S., Europe and Asia — to test-drive the prototype MIC equipment.

“We built the silicon and said, ‘Let’s look to the academic community to do the research on it,’” Mattson explained in his talk.

Since late 2010, this elite group, the Many-Core Application Research Community (MARC), has been experimenting with the Single-Chip Cloud Computer (SCC) experimental processor, a 48-core ‘concept vehicle’ created by Intel Labs as a platform for manycore software research. More recently, researchers have been experimenting with the Knights Ferry hardware as well. Through these initiatives, researchers have implemented solutions that Intel never would have thought of, according to Mattson. “The smartest people in the world don’t necessarily work at your company,” he admitted.

This sentiment served as an apt segue to the paper and poster sections of the symposium, where early users presented results from code porting, testing and performance characterization.

The two-day event pointed to the promise and challenge of implementing existing codes on Intel’s new coprocessor. No absolute performance results were offered, yet scalability plots, comparative studies, and anecdotal experiences provided an overall positive impression of the group’s experiences to date. Most impressively, several groups reported independently that their applications ran right away on Knights Ferry with no or very minor code modifications.

Symposium papers and presentations are available online. A sampling of presentation briefs follows:

Robert Harkness: “Experiences with ENZO on the Intel MIC Co-Processor”

Harness (National Institute for Computational Sciences/Oak Ridge National Laboratory) is one of the creators of ENZO, a large cosmological community code used to study the early ages of the universe. Harkness estimated that it had taken 25 person-years to develop ENZO. The code can’t be abandoned, he said, but no one will pay to rewrite it either. The question he posed was: how to move forward with a large, working science code?

Harkness tested the application on three KNF systems at NICS, which cover most of the choices in basic cluster configurations:

  • An Intel development workstation with two Intel  Xeon  5600 processors and 2 KNF cards.
  • A Cray CX-1 system consisting of a single head node with two Intel Xeon 5600 processors
  • And two compute nodes each containing two Intel Xeon 5600 processors plus one KNF card.
  • An Appro cluster consisting of four compute nodes, each of which contains two Intel Xeon 5600 processors and two KNF cards, for a total of eight KNF cards

He tested the Knights Ferry in both native and offload modes and presented relative scaling results for both.

Harkness said he was personally able to port his code to KNF in one week. In his paper he wrote: “With all these components in place for offload and native builds, the compilation of ENZO-R itself is straightforward. No ENZO-R source code modifications are required to build a functional ENZO-R binary in native mode although, clearly, some source code changes will be necessary to achieve full optimization on the Intel MIC co-processor.”

After the experience, Harkness said he is “bullish on Intel MIC co-processors.” However, he warned the chip community to not waste time on vendor-specific interfaces.

Karl W. Schulz: “Early Experiences Porting Scientific Applications to the Many Integrated Core (MIC) Platform”

Schulz is the director of the scientific applications group at TACC and also chief software architect for The Center for Predictive Engineering and COmputational Sciences (PECOS), a DOE-funded Center of Excellence at The University of Texas at Austin. The group was funded four years ago to develop new uncertainty quantification (UQ) methodologies and apply them to multiscale simulations of atmospheric reentry vehicles.

He and his collaborators tried several codes from their portfolio on Maverick, a Knights Ferry test system with eight nodes and one Knights Ferry per Westmere node, and also built 3rd party libraries to facilitate runs. He presented relative scaling results and anecdotal experiences from several application tests.

Included among these applications was the porting of a quantum chemistry application that had 1.4 million lines of Fortran code.  They also tested the kernel of a Direct Numerical Simulations (DNS) code for turbulent fluid flow, a multi-threaded FFT kernel that exhibited good strong-scaling properties on the Knights Ferry. In the future, the group is targeting very large turbulent flow simulations (10 billion grid points) and proper utilization of the Intel MIC coprocessor will be crucial for achieving that goal.

Schulz who won best paper at the symposium, gave the most comprehensive insights into the process that MARC members have been experiencing over the past few months. To the assembled application developers in the room he counseled that “moving forward, we must take vectorization more seriously.” Overall, he remarked, “It was a positive experience for all of us.”

Ashay Rane: “PerfExpert and MACPO: Which code segments should (not) be ported to MIC?”

Rane is one of the developers of the tools PerfExpert and MACPO, which are designed to analyze the performance and performance characteristics of multi-threaded applications on multicore chips. In his spirited talk he presented a process by which application developers can reliably determine the parts of a code that are good candidates for execution on Intel MIC co-processors. Guidance in this area is certainly needed. Focusing on suitable candidates for porting (either whole functions or loops nests) has the potential to not only greatly reduce the level of human effort, but also to increase the achievable speed-up.

In Rane’s two-step approach, PerfExpert is used to identify the hot spots in an application. MACPO, currently under heavy development by Rane and others, analyzes data access patterns and TLB misses within the hot spots to eliminate those that exhibit irregular access strides, etc. Interestingly, the analysis is not done on the Intel MIC co-processor itself, but on the host, which enables users today, who do not have access to any Intel MIC platform yet, to prepare using this tool.

Three examples were presented as a proof of concept. Those code regions that were recommended by the tools did indeed perform best on the Knights Ferry Software Development Package (SDP).

Sreeram Potluri: “Intra-MIC MPI Communication using MVAPICH2: Early Experience

Sreeram Potluri is a Ph.D. student at The Ohio State University who has been researching and developing the MVAPICH2 library for MPI on InfiniBand and other communications platforms. Potluri describe his group’s work implementing MVAPICH2 for communications between cores on the Knights Ferry platform.

Their initial experiments ported MVAPICH2 directly to the Intel MIC card with no changes. Subsequently, they developed two kinds of optimizations of the library. The first tuned its parameters to take maximal advantage of the memory hierarchy and cache structure on the Knights Ferry card. The second set of optimization extended the work done in the first to use Intel MIC-specific communications protocols to further improve the performance.

The results showed impressive performance gains relative to the baseline case for all varieties of communications operations that Potluri tested. These included direct, point-to-point communications between pairs of cores, and collective communications operations that gather information from many cores together or spread information from one core to all the others.

Future work will take advantage of protocols for communication between Intel MIC cards in the same host, between tasks on the Intel MIC card and tasks on the hosts, and between Intel MIC cards running in different host computers. Potluri received the award for best student paper at the symposium.

Day Two

The day two invited talks focused more explicitly on the Intel MIC coprocessors, the programming models that will drive usage, and the market conditions that inspired Intel to develop the coprocessor.

Scott McMillan, a staff software engineer at Intel, spoke on “Programming Models for Intel Xeon processors and Intel MIC Architecture.” His presentation highlighted an important point: that Xeon and Intel MIC co-processor will work together in most installations.

TACC’s upcoming Stampede system, which will be the first large-scale deployment of the Intel MIC co-processors, has a two-petaflop base system built on the Intel Xeon E5 2600 processors (formerly codenamed Sandy Bridge). The Knights Corner coprocessors will add eight petaflops of expected peak performance. Many of the discussions about parallelism, threading and vectorization apply equally to Intel’s multicore Xeon processor and to the Intel MIC architecture.

The combination of these two architectures lend themselves to a spectrum of programming models and mindsets, which McMillan outlined in his talk, from all multicore, to offload, to all manycore, to symmetric usage of both the Intel Xeon and the Intel MIC architectures.“Which to use depends on the shape and structure of one’s problem,” he said.

The keys to productive performance on the Intel MIC architecture are a matter of common sense, McMillan insisted, and could begin immediately. He offered a few expert tips:

  • Pick the right model for your application.
  • Vectorize your application (today).
  • Parallelize your application (today).
  • Go asynchronous to overlap computation and communication

“Hybrid programming today prepares you for Intel MIC architecture,” he concluded.

Last up was Nash Palaniswamy from Intel’s Technical Computing Group, who offered some perspective on the strategy behind Intel’s manycore architecture.  For more than 15 years, Palaniswamy has been working in the accelerator and coprocessor space. Throughout that time, technologically-driven companies in the oil and gas, financial services and pharmaceutical industries, as well as academic and government researchers, have explored the use of accelerators to find answers quickly, driving strong research in this market.

Intel’s development of manycore chips over several years shows a unique level of commitment to enhancing performance, Palaniswamy said. In 2006, Intel began its accelerator and coprocessor effort under the Intel QuickAssist Technology program, and in 2010 after a thorough analysis of 80 to 90 workloads from customers, they announced a roadmap for manycore. According to Palaniswamy’s assessment, manycore processors are useful for 10 to15 percent of applications today, and this number will continue to grow as programmers increasingly think parallel.

In the road-map, “little cores” — smaller, low-power cores with reduced single thread performance that rely on a high degree of parallelism — co-exist and supplement “big cores” – higher-power, higher-performance cores. This change, Palaniswamy insisted, could be achieved in a way that maximizes code reuse while requiring minimal developer re-training.

“We want to make sure the software that runs today will run 10 years from now with minimal changes,” Palaniswamy said. “Architecture, models and applications evolve. Revolution is unnecessary and unaffordable.”

With scientists continuing to port, test and optimize their software for Knights Ferry, and ultimately Knights Corner, conversations about the Intel MIC architecture will be ongoing for many months to come.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Bill Gropp – Pursuing the Next Big Thing at NCSA

March 28, 2017

About eight months ago Bill Gropp was elevated to acting director of the National Center for Supercomputing Applications (NCSA). Read more…

By John Russell

UK to Launch Six Major HPC Centers

March 27, 2017

Six high performance computing centers will be formally launched in the U.K. later this week intended to provide wider access to HPC resources to U.K. Read more…

By John Russell

AI in the News: Rao in at Intel, Ng out at Baidu, Nvidia on at Tencent Cloud

March 26, 2017

Just as AI has become the leitmotif of the advanced scale computing market, infusing much of the conversation about HPC in commercial and industrial spheres, it also is impacting high-level management changes in the industry. Read more…

By Doug Black

Scalable Informatics Ceases Operations

March 23, 2017

On the same day we reported on the uncertain future for HPC compiler company PathScale, we are sad to learn that another HPC vendor, Scalable Informatics, is closing its doors. Read more…

By Tiffany Trader

HPE Extreme Performance Solutions

Quants Achieving Maximum Compute Power without the Learning Curve

The financial services industry is a fast-paced and data-intensive environment, and financial firms are realizing that they must modernize their IT infrastructures and invest in high performance computing (HPC) tools in order to survive. Read more…

‘Strategies in Biomedical Data Science’ Advances IT-Research Synergies

March 23, 2017

“Strategies in Biomedical Data Science: Driving Force for Innovation” by Jay A. Etchings is both an introductory text and a field guide for anyone working with biomedical data. Read more…

By Tiffany Trader

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

Google Launches New Machine Learning Journal

March 22, 2017

On Monday, Google announced plans to launch a new peer review journal and “ecosystem” Read more…

By John Russell

Swiss Researchers Peer Inside Chips with Improved X-Ray Imaging

March 22, 2017

Peering inside semiconductor chips using x-ray imaging isn’t new, but the technique hasn’t been especially good or easy to accomplish. Read more…

By John Russell

Bill Gropp – Pursuing the Next Big Thing at NCSA

March 28, 2017

About eight months ago Bill Gropp was elevated to acting director of the National Center for Supercomputing Applications (NCSA). Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the campaign. Read more…

By John Russell

CPU-based Visualization Positions for Exascale Supercomputing

March 16, 2017

In this contributed perspective piece, Intel’s Jim Jeffers makes the case that CPU-based visualization is now widely adopted and as such is no longer a contrarian view, but is rather an exascale requirement. Read more…

By Jim Jeffers, Principal Engineer and Engineering Leader, Intel

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

New Japanese Supercomputing Project Targets Exascale

March 14, 2017

Another Japanese supercomputing project was revealed this week, this one from emerging supercomputer maker, ExaScaler Inc., and Keio University. The partners are working on an original supercomputer design with exascale aspirations. Read more…

By Tiffany Trader

Nvidia Debuts HGX-1 for Cloud; Announces Fujitsu AI Deal

March 9, 2017

On Monday Nvidia announced a major deal with Fujitsu to help build an AI supercomputer for RIKEN using 24 DGX-1 servers. Read more…

By John Russell

For IBM/OpenPOWER: Success in 2017 = (Volume) Sales

January 11, 2017

To a large degree IBM and the OpenPOWER Foundation have done what they said they would – assembling a substantial and growing ecosystem and bringing Power-based products to market, all in about three years. Read more…

By John Russell

Quantum Bits: D-Wave and VW; Google Quantum Lab; IBM Expands Access

March 21, 2017

For a technology that’s usually characterized as far off and in a distant galaxy, quantum computing has been steadily picking up steam. Read more…

By John Russell

Trump Budget Targets NIH, DOE, and EPA; No Mention of NSF

March 16, 2017

President Trump’s proposed U.S. fiscal 2018 budget issued today sharply cuts science spending while bolstering military spending as he promised during the campaign. Read more…

By John Russell

HPC Compiler Company PathScale Seeks Life Raft

March 23, 2017

HPCwire has learned that HPC compiler company PathScale has fallen on difficult times and is asking the community for help or actively seeking a buyer for its assets. Read more…

By Tiffany Trader

TSUBAME3.0 Points to Future HPE Pascal-NVLink-OPA Server

February 17, 2017

Since our initial coverage of the TSUBAME3.0 supercomputer yesterday, more details have come to light on this innovative project. Of particular interest is a new board design for NVLink-equipped Pascal P100 GPUs that will create another entrant to the space currently occupied by Nvidia's DGX-1 system, IBM's "Minsky" platform and the Supermicro SuperServer (1028GQ-TXR). Read more…

By Tiffany Trader

Tokyo Tech’s TSUBAME3.0 Will Be First HPE-SGI Super

February 16, 2017

In a press event Friday afternoon local time in Japan, Tokyo Institute of Technology (Tokyo Tech) announced its plans for the TSUBAME3.0 supercomputer, which will be Japan’s “fastest AI supercomputer,” Read more…

By Tiffany Trader

IBM Wants to be “Red Hat” of Deep Learning

January 26, 2017

IBM today announced the addition of TensorFlow and Chainer deep learning frameworks to its PowerAI suite of deep learning tools, which already includes popular offerings such as Caffe, Theano, and Torch. Read more…

By John Russell

Lighting up Aurora: Behind the Scenes at the Creation of the DOE’s Upcoming 200 Petaflops Supercomputer

December 1, 2016

In April 2015, U.S. Department of Energy Undersecretary Franklin Orr announced that Intel would be the prime contractor for Aurora: Read more…

By Jan Rowell

Leading Solution Providers

Is Liquid Cooling Ready to Go Mainstream?

February 13, 2017

Lost in the frenzy of SC16 was a substantial rise in the number of vendors showing server oriented liquid cooling technologies. Three decades ago liquid cooling was pretty much the exclusive realm of the Cray-2 and IBM mainframe class products. That’s changing. We are now seeing an emergence of x86 class server products with exotic plumbing technology ranging from Direct-to-Chip to servers and storage completely immersed in a dielectric fluid. Read more…

By Steve Campbell

Enlisting Deep Learning in the War on Cancer

December 7, 2016

Sometime in Q2 2017 the first ‘results’ of the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) will become publicly available according to Rick Stevens. He leads one of three JDACS4C pilot projects pressing deep learning (DL) into service in the War on Cancer. Read more…

By John Russell

BioTeam’s Berman Charts 2017 HPC Trends in Life Sciences

January 4, 2017

Twenty years ago high performance computing was nearly absent from life sciences. Today it’s used throughout life sciences and biomedical research. Genomics and the data deluge from modern lab instruments are the main drivers, but so is the longer-term desire to perform predictive simulation in support of Precision Medicine (PM). There’s even a specialized life sciences supercomputer, ‘Anton’ from D.E. Shaw Research, and the Pittsburgh Supercomputing Center is standing up its second Anton 2 and actively soliciting project proposals. There’s a lot going on. Read more…

By John Russell

HPC Startup Advances Auto-Parallelization’s Promise

January 23, 2017

The shift from single core to multicore hardware has made finding parallelism in codes more important than ever, but that hasn’t made the task of parallel programming any easier. Read more…

By Tiffany Trader

HPC Technique Propels Deep Learning at Scale

February 21, 2017

Researchers from Baidu’s Silicon Valley AI Lab (SVAIL) have adapted a well-known HPC communication technique to boost the speed and scale of their neural network training and now they are sharing their implementation with the larger deep learning community. Read more…

By Tiffany Trader

CPU Benchmarking: Haswell Versus POWER8

June 2, 2015

With OpenPOWER activity ramping up and IBM’s prominent role in the upcoming DOE machines Summit and Sierra, it’s a good time to look at how the IBM POWER CPU stacks up against the x86 Xeon Haswell CPU from Intel. Read more…

By Tiffany Trader

US Supercomputing Leaders Tackle the China Question

March 15, 2017

Joint DOE-NSA report responds to the increased global pressures impacting the competitiveness of U.S. supercomputing. Read more…

By Tiffany Trader

IDG to Be Bought by Chinese Investors; IDC to Spin Out HPC Group

January 19, 2017

US-based publishing and investment firm International Data Group, Inc. (IDG) will be acquired by a pair of Chinese investors, China Oceanwide Holdings Group Co., Ltd. Read more…

By Tiffany Trader

  • arrow
  • Click Here for More Headlines
  • arrow
Share This