Daresbury Laboratory Benchmarks Multi-Core Chips

By Christopher Lazou

December 16, 2005

On December 6th and 7th about 250 people attended the 16th Machine Evaluation Workshop at EPSRC Daresbury Laboratories, UK. This workshop is a leading UK national event dedicated to distributed, high performance scientific computing. The principle objective is to encourage close contact between the research communities from the Mathematics, Chemistry, Physics, Engineering and Materials Programmes of EPSRC and the major vendors of mid-range computing systems, workstations, servers, software and peripherals.

Most of the 25 presentations were from vendors, describing their own products, on topics such as hardware, compilers, graphics, storage and networking. They focused on cluster solutions, based on commodity chips, interconnect networks and associated file storage systems. An important component of the workshop is the availability of systems for benchmarking evaluation purposes.

There were exhibitions and presentations from eighteen companies, keen to promote their ready-made products including those based on AMD Opteron and the Intel Itanium 2 processor. A strong presence of AMD Opteron and Intel Bensley, early dual-core systems, as well as various models of blade products, were on display and available for demonstrations. 

Ron Bell, from the UK Atomic Weapons Establishment (AWE), started the workshop with a presentation titled: “The AWE HPC Benchmark”. This was an interesting talk in that it discussed AWE's benchmark based on workload profile characterization and projections of future user requirements. They are currently running an IBM P3 system with 1856 processors rated at 2.88 Tflop/s peak performance. The requirement is to buy a new system delivering 10 to 25 times greater performance than their current system, which translates to a system in the range of 30 to 70 Tflop/s peak performance.

The benchmark contains a combination of codes from the whole AWE user community, physicists, engineers and material scientists, weighted to reflect their workload. It measures both capacity (throughput) and capability (turnaround).

The physicists contributed plasma physics and hydrodynamic codes plus visualization. The engineers codes for solving explicit and implicit (100 MDOFs) models with as much as 30 million elements. From the Materials group, two molecular dynamics codes were used, DL-Poly from Daresbury and WARP from Sandia.

Ron went on to cogently discuss the pitfalls of benchmarking when dealing with systems of differing speeds. He gave an example of two systems A and B with system A twice as fast as B and B having twice the number of processors than A. He then presented the usual scenario where capacity can be achieved by partitioning system B into two, running both partitions in parallel giving the illusion that both systems have the same throughput.
He went on to say: “The answer is not system A, gives better turnaround times but has the same throughput. The correct answer is system A has higher throughput than system B because system B must scale further in order to achieve the required job turnaround and system B may be unable to give the required turnaround for some capability jobs. In conclusion, don't compare N-way on A with N-way on B. Adjust N so that turnaround is about the same on A and B”.

In running the AWE benchmark they found that: “There is a problem with measuring throughput of capability jobs. At modest levels of parallelism, scalability is largely unaffected by interconnect. Scalability is intrinsic to application. The ratio between systems is constant with PE count. At higher PE counts where performance turns over, relative throughput varies wildly and becomes meaningless.”

Also, it was very difficult to perform the capability test. There were very few benchmark data available, up to the turn over point. To measure the job turnaround at a point just before turn over, the best the system can do irrespective of number of PEs, became difficult. Generally, a system, scoring better on this measure would need more PEs to achieve it – so throughput was probably lower. For this reason, capacity and capability figures were presented as separate measures, with a warning about the large uncertainties on the capability figures.

Ron Bell wanted to be able to say things like: “System A has 10 percent higher throughput than system B for modestly parallel work, but system B has better scalability – so capability jobs show 20 percent higher throughput on B. If we assume half of the system will be dedicated to capability jobs, then System B gives more overall throughput”. This was not apparent from the benchmark results.

Ron Bell concluded: “The benchmark was too complex for vendors. The throughput test is not really useful as a benchmark. Multiple jobs don't usually interfere much. There was inadequate I/O on benchmark systems, but this should be excellent as an acceptance test. Capability was very difficult to quantify”.

In short, Ron Bell was saying that for large-scale applications (30 million elements), time to completion must be heavily weighted in the selection criteria of a system. Capability systems can always deliver capacity where capability is often beyond capacity systems, but capability is difficult to measure accurately as vendors do not usually have systems of that size for benchmarking.

Of course other factors often dominate the selection of large-scale systems. For example, one trade-off is price/performance; another, are perceived hidden benefits from having the same system as your collaborators. One suspects it depends on how one constructs the Total Cost of Ownership (TCO) integral. For my money, I would also include ease of use and Mean Time Between Failures (MTBF), especially with systems with tens or hundred thousand processors.

Chris Brown gave the “IBM HPC systems perspective” talk emphasizing that one size does not fit all. He explained that in addition to the IBM P5 line, IBM is exploring the low-power Cell processor technology. The speaker claimed that because IBM is a large company with a strong track record of innovation and in control of component developments, it is able to leverage these innovations across the whole design spectrum. For example, take the game processor developed from Cell technology for the large consumer market, add a couple of floating point pipes and what one has is the Blue Gene. Both the Blue Gene and the MareNostrum systems, based on the Power PC 970 FX processor, are examples of experimental technologies. IBM believes that next generation chip designs are focusing on high performance/power consumption ratios and that semiconductor power trends are driving future systems. With hundreds of thousands of processors, software tool makers will be challenged to create an easy to use development environment. I may also add, reliability, such as MTBF.

Jörg Stadler from NEC HPC Europe described the NEC SX-8 vector parallel system whose raison d'être is capability supercomputing. NEC is committed to continue to develop products at the top end of HPC using the latest technologies. Their next generation system is likely to be heterogeneous with a strong vector processing component. He also explained that they are offering total solutions to customers, from high computation to data management infrastructure, based on their super-scalar Itanium 2 system. As an example, he cited the DKRZ climate center in Germany where the computation element is delivered by the NEC SX-6 and the data management of some 220 terabytes using the NEC TX7 and Oracle. They also deliver tailor-made scalar systems in collaboration with other vendors. The 100 Tflop/s scalar system for the Tokyo Institute of Technology, where NEC is acting as the integrator, but collaborating with ClearSpeed, AMD and Sun Microsystems, was cited as an example.     

As in previous years, the Daresbury Benchmark results were of great interest. These consisted of a plethora of distributed memory benchmark results, compiled by Martyn Guest and his team from Daresbury, from many systems including the latest products from vendors using their latest two core chips. The Daresbury benchmark suite, used to obtain these results, consists of many computational chemistry kernel codes, molecular dynamics, Quantum Monte Carlo, Jacobi Solver, STREAM – measured sustainable memory bandwidth in HPC (TRIAD), the Ab Initio molecular electronic structure package, GAMESS-UK, and the parallel molecular dynamics benchmark, DL_POLY. The results from SPECfp2000, SPECInt2000 and other well-known benchmarks were also presented.

Martyn Guest gave a similar talk as previous years, this time the results were normalised against an Opteron – the AMD Opteron 852/2600 (EKO2.2) – see table below. Martyn emphasised that single processors are complex and often provide misleading results as they are almost always used in n-way nodes. If they are dual-core, use both processors; if quad-cores, use all four so that interactions of cache memory and communications are accounted in the performance measure. For example, in the case of the 32-way IBM P4 this normalizes availability of L3 cache memory, which reduces performance compared to using all L3 cache for one CPU. This applies to other vendors' products too. When more cores are added on a chip the interconnect fabric for transfer rates (bandwidth) to L3 cache need to be increased proportionately to handle the extra computational power from the extra cores. Otherwise, memory paths, if shared with the additional cores, are likely to degrade the overall system performance.

Note that the results presented below, reflect performance in computational chemistry and this does not necessarily reflect the performance of these computers in other application domains. Also, the results given are not normalized on price/performance, so no specific value-for-money comparisons are made or implied.

Looking at SPECfp 2000 relative to the HP RX5670 Itanium 2 (1.5GHz), the SGI Altix 3700 Bx2/1600-9M Itanium 2 system is 26 percent faster and the IBM e-Server P5 570/1900 is 22 percent faster, while the HP RX4640 Itanium 2/1600-9M is 29 percent faster. These systems deliver around 40 percent of their peak performance on the SPECfp 2000 metric.
The benchmark results compiled by the Daresbury team indicate that the Intel Itanium 2 systems are still the bright stars of today, but the IBM e-server P5 570/1900 is not far behind. They fare well as far as performance is concerned compared to other super-scalar chips, although one can detect a trend towards convergence. Performance of a particular chip tends to vary on different benchmarks and the version of compiler is run on, but one can see a pattern emerging. To give you a flavor of the benchmarks, here are some of the results, normalized on the AMD Opteron 852/2600 using the PathScale compiler EKO2.2 (for the Matrix-97, Chemistry Kernels, GAMES-UK and DL-POLY benchmarks):

The AMD Opteron 852/2600 (EKO2.2)     (100)
The AMD Opteron 852/2600 (PGI5.2)        (88)
The Dell PowerEdge 1850/3600 2MBL2      (72)
Sun Fire V40z 2.2GHz                                 (76)
The Bull NovaScale 4040 Itanium 2/1500     (98)
SGI Altix3700 Itanium 2/1500                      (98)
SGI Altix3700 Itanium 2/1600                    (105)
The HP RX5670 Itanium 2/1500-H              (99)
The HP RX1620 Itanium 2/1600-L             (106)
The HP RX1620 Itanium 2/1600-H             (121)
The IBM p-Series 690/1700                         (80)
The IBM e-Server p5 575/1500                    (78)
The IBM e-Server P5 570/1900                  (105)

Note that the HP compiler delivers about 15 percent improvement compared to its Linux counterpart on the HP RX1620. A similar pattern is also evident on the AMD Opteron 852/2600, when using PathScale's EKO (Ver.2.2) and the Portland Group's PGI (Ver.5.2) Fortran compilers.

A more interesting table was presented showing scaling performance when using multi-cores on a chip. In general they scaled reasonably well. There were few exceptions. One major bottleneck identified is the bandwidth imbalance from L2 to L3 cache, where the transfer rate remained as for one core, but has to deal with bandwidth requirements of two cores.

The rest of the workshop consisted of presentations from vendors, with a strong contingent from the FPGAs fraternity, Mitrion, Celoxica and Nallatech, and from users sharing their experience and presentations from a number of companies, specializing in providing tailored system solutions from commodity components on demand. Instead of buying pre-packaged products from traditional vendors, a contract is placed with a small computer integration company, such as ClusterVision or Streamline, to built a cluster from favored chips and an interconnect network, such as Gigabit Ethernet, QsNet, InfiniBand or Myrinet.

The presentations focused on cluster systems affordable by academic departments and associated organizations, which make the departmental computing environment. For example, Fiona Burgess and Michal Harasimiuk from ClearSpeed, presented their new co-processor, the CSX600, suitable for offloading compute-intense math library functions from serial CPUs. Michal claimed that: “Each CSX600 co-processor can sustain 25 Gflop/s on DGEMM, while consuming only 5 watts of power. The CSX600, now available, is a SIMD array of processors. Each PE is a VLIW processor with a multiple execution floating point adder and floating point multiplier in both 32-bit and 64-bit IEEE754 standard.

A two-chip board delivers 50 Gflop/s peak and uses 10 watts in total. It has up to one gigabyte shared DRAM for local processing. A dual CSX600 using PCI-X, 20 watts/card accelerator board gives 100 Gflop/s peak on DGEMM, BLAS, LAPACK, GROMACS, CHARMM NAND and so on. It is suitable for computation in finance, Monte Carlo and Generic math, providing transparent acceleration for packages such as MATLAB, Mathematica, NAG, Maple and entire applications in biochemistry.

There are some very positive trends in multi-core chip developments, but the workshop also had some surprises. These are too sensitive to be printed here. For those of you keen to add the latest 'gizmos' to your facilities, remember the Latin adage: “caveat emptor.” As the season of goodwill is upon us: Wishing you all, Seasons Greetings and a peaceful Happy New Year.

(Brands and names are the property of their respective owners)
Copyright: Christopher Lazou, HiPerCom Consultants, Ltd., UK., December 2005.

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

What’s New in Computing vs. COVID-19: Fugaku, Congress, De Novo Design & More

July 2, 2020

Supercomputing, big data and artificial intelligence are crucial tools in the fight against the coronavirus pandemic. Around the world, researchers, corporations and governments are urgently devoting their computing reso Read more…

By Oliver Peckham

OpenPOWER Reboot – New Director, New Silicon Partners, Leveraging Linux Foundation Connections

July 2, 2020

Earlier this week the OpenPOWER Foundation announced the contribution of IBM’s A21 Power processor core design to the open source community. Roughly this time last year, IBM announced open sourcing its Power instructio Read more…

By John Russell

HPC Career Notes: July 2020 Edition

July 1, 2020

In this monthly feature, we'll keep you up-to-date on the latest career developments for individuals in the high-performance computing community. Whether it's a promotion, new company hire, or even an accolade, we've got Read more…

By Mariana Iriarte

Supercomputers Enable Radical, Promising New COVID-19 Drug Development Approach

July 1, 2020

Around the world, innumerable supercomputers are sifting through billions of molecules in a desperate search for a viable therapeutic to treat COVID-19. Those molecules are pulled from enormous databases of known compoun Read more…

By Oliver Peckham

HPC-Powered Simulations Reveal a Looming Climatic Threat to Vital Monsoon Seasons

June 30, 2020

As June draws to a close, eyes are turning to the latter half of the year – and with it, the monsoon and hurricane seasons that can prove vital or devastating for many of the world’s coastal communities. Now, climate Read more…

By Oliver Peckham

AWS Solution Channel

Maxar Builds HPC on AWS to Deliver Forecasts 58% Faster Than Weather Supercomputer

When weather threatens drilling rigs, refineries, and other energy facilities, oil and gas companies want to move fast to protect personnel and equipment. And for firms that trade commodity shares in oil, precious metals, crops, and livestock, the weather can significantly impact their buy-sell decisions. Read more…

Intel® HPC + AI Pavilion

Supercomputing the Pandemic: Scientific Community Tackles COVID-19 from Multiple Perspectives

Since their inception, supercomputers have taken on the biggest, most complex, and most data-intensive computing challenges—from confirming Einstein’s theories about gravitational waves to predicting the impacts of climate change. Read more…

Hyperion Forecast – Headwinds in 2020 Won’t Stifle Cloud HPC Adoption or Arm’s Rise

June 30, 2020

The semiannual taking of HPC’s pulse by Hyperion Research – late fall at SC and early summer at ISC – is a much-watched indicator of things come. This year is no different though the conversion of ISC to a digital Read more…

By John Russell

OpenPOWER Reboot – New Director, New Silicon Partners, Leveraging Linux Foundation Connections

July 2, 2020

Earlier this week the OpenPOWER Foundation announced the contribution of IBM’s A21 Power processor core design to the open source community. Roughly this time Read more…

By John Russell

Hyperion Forecast – Headwinds in 2020 Won’t Stifle Cloud HPC Adoption or Arm’s Rise

June 30, 2020

The semiannual taking of HPC’s pulse by Hyperion Research – late fall at SC and early summer at ISC – is a much-watched indicator of things come. This yea Read more…

By John Russell

Racism and HPC: a Special Podcast

June 29, 2020

Promoting greater diversity in HPC is a much-discussed goal and ostensibly a long-sought goal in HPC. Yet it seems clear HPC is far from achieving this goal. Re Read more…

Top500 Trends: Movement on Top, but Record Low Turnover

June 25, 2020

The 55th installment of the Top500 list saw strong activity in the leadership segment with four new systems in the top ten and a crowning achievement from the f Read more…

By Tiffany Trader

ISC 2020 Keynote: Hope for the Future, Praise for Fugaku and HPC’s Pandemic Response

June 24, 2020

In stark contrast to past years Thomas Sterling’s ISC20 keynote today struck a more somber note with the COVID-19 pandemic as the central character in Sterling’s annual review of worldwide trends in HPC. Better known for his engaging manner and occasional willingness to poke prickly egos, Sterling instead strode through the numbing statistics associated... Read more…

By John Russell

ISC 2020’s Student Cluster Competition Winners Announced

June 24, 2020

Normally, the Student Cluster Competition involves teams of students building real computing clusters on the show floors of major supercomputer conferences and Read more…

By Oliver Peckham

Hoefler’s Whirlwind ISC20 Virtual Tour of ML Trends in 9 Slides

June 23, 2020

The ISC20 experience this year via livestreaming and pre-recordings is interesting and perhaps a bit odd. That said presenters’ efforts to condense their comments makes for economic use of your time. Torsten Hoefler’s whirlwind 12-minute tour of ML is a great example. Hoefler, leader of the planned ISC20 Machine Learning... Read more…

By John Russell

At ISC, the Fight Against COVID-19 Took the Stage – and Yes, Fugaku Was There

June 23, 2020

With over nine million infected and nearly half a million dead, the COVID-19 pandemic has seized the world’s attention for several months. It has also dominat Read more…

By Oliver Peckham

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

[email protected] Turns Its Massive Crowdsourced Computer Network Against COVID-19

March 16, 2020

For gamers, fighting against a global crisis is usually pure fantasy – but now, it’s looking more like a reality. As supercomputers around the world spin up Read more…

By Oliver Peckham

[email protected] Rallies a Legion of Computers Against the Coronavirus

March 24, 2020

Last week, we highlighted [email protected], a massive, crowdsourced computer network that has turned its resources against the coronavirus pandemic sweeping the globe – but [email protected] isn’t the only game in town. The internet is buzzing with crowdsourced computing... Read more…

By Oliver Peckham

Global Supercomputing Is Mobilizing Against COVID-19

March 12, 2020

Tech has been taking some heavy losses from the coronavirus pandemic. Global supply chains have been disrupted, virtually every major tech conference taking place over the next few months has been canceled... Read more…

By Oliver Peckham

Supercomputer Simulations Reveal the Fate of the Neanderthals

May 25, 2020

For hundreds of thousands of years, neanderthals roamed the planet, eventually (almost 50,000 years ago) giving way to homo sapiens, which quickly became the do Read more…

By Oliver Peckham

DoE Expands on Role of COVID-19 Supercomputing Consortium

March 25, 2020

After announcing the launch of the COVID-19 High Performance Computing Consortium on Sunday, the Department of Energy yesterday provided more details on its sco Read more…

By John Russell

Steve Scott Lays Out HPE-Cray Blended Product Roadmap

March 11, 2020

Last week, the day before the El Capitan processor disclosures were made at HPE's new headquarters in San Jose, Steve Scott (CTO for HPC & AI at HPE, and former Cray CTO) was on-hand at the Rice Oil & Gas HPC conference in Houston. He was there to discuss the HPE-Cray transition and blended roadmap, as well as his favorite topic, Cray's eighth-gen networking technology, Slingshot. Read more…

By Tiffany Trader

Honeywell’s Big Bet on Trapped Ion Quantum Computing

April 7, 2020

Honeywell doesn’t spring to mind when thinking of quantum computing pioneers, but a decade ago the high-tech conglomerate better known for its control systems waded deliberately into the then calmer quantum computing (QC) waters. Fast forward to March when Honeywell announced plans to introduce an ion trap-based quantum computer whose ‘performance’ would... Read more…

By John Russell

Leading Solution Providers


Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

‘Billion Molecules Against COVID-19’ Challenge to Launch with Massive Supercomputing Support

April 22, 2020

Around the world, supercomputing centers have spun up and opened their doors for COVID-19 research in what may be the most unified supercomputing effort in hist Read more…

By Oliver Peckham

Australian Researchers Break All-Time Internet Speed Record

May 26, 2020

If you’ve been stuck at home for the last few months, you’ve probably become more attuned to the quality (or lack thereof) of your internet connection. Even Read more…

By Oliver Peckham

Nvidia’s Ampere A100 GPU: Up to 2.5X the HPC, 20X the AI

May 14, 2020

Nvidia's first Ampere-based graphics card, the A100 GPU, packs a whopping 54 billion transistors on 826mm2 of silicon, making it the world's largest seven-nanom Read more…

By Tiffany Trader

15 Slides on Programming Aurora and Exascale Systems

May 7, 2020

Sometime in 2021, Aurora, the first planned U.S. exascale system, is scheduled to be fired up at Argonne National Laboratory. Cray (now HPE) and Intel are the k Read more…

By John Russell

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Summit Supercomputer is Already Making its Mark on Science

September 20, 2018

Summit, now the fastest supercomputer in the world, is quickly making its mark in science – five of the six finalists just announced for the prestigious 2018 Read more…

By John Russell

TACC Supercomputers Run Simulations Illuminating COVID-19, DNA Replication

March 19, 2020

As supercomputers around the world spin up to combat the coronavirus, the Texas Advanced Computing Center (TACC) is announcing results that may help to illumina Read more…

By Staff report

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This