Gravitational Waves Detected! Historic LIGO Success Strikes Chord with Larry Smarr

By John Russell

February 12, 2016

By now you’ve likely heard that scientists reported detecting the long-sought gravitational waves; this is roughly a 100 years since their prediction by Einstein’s theory of general relativity and more than 20 years since first funding for the LIGO (Laser Interferometer Gravitational-wave Observatory (LIGO) project to find them. You can safely assume the National Science Foundation (NSF), a prime funder of the LIGO, is feeling well rewarded (and relieved) for the $620 Million invested. It was a big risk/big reward project.

The event that created these particular gravitational waves – the spiraling of two black holes into each other about 1.3 billion light-years away (or 1.3 billion years ago) – was the most energetic event ever detected. For a brief moment, the collision of these black holes generated more power than all of the stars in the universe, meaning it briefly outshone the output of 100 billion galaxies, each with 100 billion stars.

“It’s hard to overstate how historic the moment is today. Think about all of the amazing astronomical objects we’re studying with radio astronomy, x-ray astronomy, ultraviolet astronomy, infrared astronomy or optical. Those are all just parts of the electromagnetic spectrum. This is an entirely new spectrum, from end to end, with as diverse sources as the electromagnetic spectrum that we have been working on since Galileo,” said Larry Smarr, founding Director of the California Institute for Telecommunications and Information Technology (Calit2), a UC San Diego/UC Irvine partnership, who holds the Harry E. Gruber professorship in Computer Science and Engineering (CSE) at UCSD’s Jacobs School.

Larry Smarr
Larry Smarr

Modern science, of course, is propelled forward in large measure by advanced computing; in this case Smarr was one of the early architects of supercomputing tools needed to solve Einstein’s equations. His Ph.D. theses written in 1975 – The Structure of General Relativity with a Numerical Illustration: The Collision of Two Black Holes – worked out the foundations of numerical relativity and began winning over theoretical physicists whose bias then favored analytic over computational methods.

First, let’s look at the remarkable LIGO project achievement itself. LIGO was originally proposed as a means of detecting gravitational waves in the 1980s by Rainer Weiss, professor of physics, emeritus, from MIT; Kip Thorne, Caltech’s Richard P. Feynman Professor of Theoretical Physics, emeritus; and Ronald Drever, professor of physics, emeritus, also from Caltech. Both Weiss and Thorne were present at the announcement.

Besides providing more proof for Einstein’s general theory of relativity, the ability to detect gravity waves opens an entirely new window through which to observe the universe. This event was the collision of relatively modest size black holes. As LIGO’s sensitivity is improved – it’s at one third of its spec now and expected to reach full sensitivity fairly quickly – a wealth of new and fascinating phenomena will be observed. Nobel prizes will ensue.

The achievement was announced yesterday at the annual American Association for the Advancement of Science (AAAS) meeting being held in Washington over the next few days. Gravitational waves carry information about their dramatic origins and about the nature of gravity that cannot be obtained from elsewhere. Physicists have concluded that the detected gravitational waves were produced during the final fraction of a second of the merger of two black holes to produce a single, more massive spinning black hole. This collision of two black holes had been predicted but never observed.

LIGO event signals
LIGO event signals

For the record, these gravitational waves were detected on Sept. 14, 2015 at 5:51 a.m. EDT (09:51 UTC) by both of the twin Laser Interferometer Gravitational-wave Observatory (LIGO) detectors, located in Livingston, Louisiana, and Hanford, Washington. The LIGO observatories were conceived, built and are operated by Caltech and MIT. The discovery, accepted for publication in the journal Physical Review Letters, was made by the LIGO Scientific Collaboration (which includes the GEO Collaboration and the Australian Consortium for Interferometric Gravitational Astronomy) and the Virgo Collaboration using data from the two LIGO detectors.

Based on the observed signals, LIGO scientists estimate that the black holes for this event were about 29 and 36 times the mass of the sun, and the event took place 1.3 billion years ago. About three times the mass of the sun was converted into gravitational wave energy in a fraction of a second — with a peak power output about 50 times that of the whole visible stellar universe. By looking at the time of arrival of the signals — the detector in Livingston recorded the event 7 milliseconds before the detector in Hanford — scientists can say that the source was located in the Southern Celestial Hemisphere.

LIGO research is carried out by the LIGO Scientific Collaboration (LSC), a group of more than 1,000 scientists from universities around the United States and in 14 other countries. More than 90 universities and research institutes in the LSC develop detector technology and analyze data; approximately 250 students are strong contributing members of the collaboration. The LSC detector network includes the LIGO interferometers and the GEO600 detector. The GEO team includes scientists at the Max Planck Institute for Gravitational Physics (Albert Einstein Institute, AEI), Leibniz Universität Hannover, along with partners at the University of Glasgow, Cardiff University, the University of Birmingham, other universities in the United Kingdom and the University of the Balearic Islands in Spain.

“This detection is the beginning of a new era: The field of gravitational wave astronomy is now a reality,” says Gabriela González, LSC spokesperson and professor of physics and astronomy at Louisiana State University.

The discovery was made possible by the enhanced capabilities of Advanced LIGO, a major upgrade that increases the sensitivity of the instruments compared to the first generation LIGO detectors, enabling a large increase in the volume of the universe probed — and the discovery of gravitational waves during its first observation run. NSF is the lead financial supporter of Advanced LIGO. Funding organizations in Germany (Max Planck Society), the U.K. (Science and Technology Facilities Council, STFC) and Australia (Australian Research Council) also have made significant commitments to the project.

There are many excellent accounts of the LIGO achievement (see links at the end of this article). The New York Times has a good brief video. There’s also a paper describing the LIGO discovery published online in Physical Review Letters (Observation of Gravitational Waves from a Binary Black Hole Merger) at the same time as yesterday’s announcement. Yet another interesting thread in the LIGO success story is Smarr’s personal connection to gravitational wave research, which as mentioned earlier stretches back to his graduate school days.

Larry Smarr PhD cover page 1975“My Ph.D. dissertation was on solving Einstein’s equations for the collision of two black holes and the gravitational radiation it would generate. In those days, I started out working on a CDC 6600,” recalled Smarr.

He was at the University of Texas at Austin, which had a relativity research center, and his Ph.D. advisor was Bryce DeWitt. These were early days for supercomputing. The CDC 6600 cranked out around 3 or 4 megaflops. Smarr moved up to a CDC 7600, roughly ten-times more powerful with a peak of 36 megaflops. Today, a typical smart phone is a gigaflops device, Smarr noted.

The task was to develop computational techniques to solve Einstein’s equations for general relativity in curved space-time. “I used hundreds of hours doing that. I had to do the head-on collision, which actually makes the problem axisymmetric; so imagine two black holes gravitationally attracting each other on a line and crashing into each other. You couldn’t possibly compute orbiting. We knew that’s what we needed to do for reality, but you couldn’t do it with the supercomputers of those days.”

Access to sufficient compute power wasn’t the only challenge. Many relativity experts resisted using of computational approaches. “Most had done all of their research relying on exact analytic solutions. I was arguing for computational science and that the only way to understand the dynamics was to use computers. There were a lot of people who were the leaders of general relativity in the 60s who felt that I was besmirching their beautiful analytic solutions,” said Smarr.

Smarr credits DeWitt with the vision that you could use supercomputers to solve the Einstein equations. Upon completion of his degree, DeWitt advised Smarr “to work with the masters of computational science and use the supercomputers. I said, well where are those, and he said you’ll have to get a top secret nuclear weapons clearance to go work inside Livermore (National Laboratory) or Los Alamos (National Lab).”

At the time, said Smarr, it was considered a crazy notion that universities would be able to support national supercomputer centers as they do now. The prevailing view was universities couldn’t handle anything that complicated. He went to Livermore. While there Smarr met a graduate student, Michael Norman, who had just started doing summer research at Livermore. That initiated a four-decade collaboration on many areas of computational astrophysics. Mike is currently the director of the San Diego Supercomputer Center.

“I realized if you could solve something as complicated as general relativity on supercomputers you could also be solving many other problems in science and engineering that researchers in academia were working on. That’s what led me over the years to develop the idea of putting in a proposal [along those lines] to NSF.”

So it is that Smarr’s graduate school experience working to computationally describe the collision of two black holes using Einstein’s equations led to a broader interest in solving science problems with supercomputers. He also became convinced that such computers could thrive usefully in academia, and in 1983 made an unsolicited proposal to NSF that ultimately became NCSA – the National Center for Supercomputing Applications.

Smarr was awarded the Golden Goose award in 2014 which the American Association for the Advancement of Science, the American Association of Universities, other organizations, and supporters in Congress gives to people whose seemingly foolish government funded ideas turn out to be whopping successes for society at large. “It’s given to people who are working on some totally obscure project by that people would normally make fun of and that ends up changing the World,” laughed Smarr.

“When I got to NCSA I wanted to continue my personal research on colliding black holes and so I hired a bright energetic post doc, Ed Seidel who is now the director of NCSA and in between was the NSF Associate Director for Math and Physical Sciences, which has the funding for LIGO. You can’t make this stuff up,” said Smarr who served as NCSA director for 15 years.

At NCSA, Smarr formed a numerical group, led by Seidel. The group quickly became a leader in applying supercomputers to black hole and gravitational wave problems. For example, in 1994 one of the first 3-dimension simulation of two colliding black holes providing computed gravitational waveforms was carried out at NCSA by this group in collaboration with colleagues at Washington University.

NCSANCSA as a center has continued to support the most complex problems in numerical relativity and relativistic astrophysics, including working with several groups addressing models of gravitational waves sources seen by LIGO in this discovery. Even more complex simulations will be needed for anticipated future discoveries such as colliding neutron stars and black holes or supernovae explosions.

NCSA has also played a role in developing the tools needed for simulating relativistic systems. The work of Seidel’s NCSA group led to the development of the Cactus Framework, a modular and collaborative framework for parallel computing which since 1997 has supported numerical relativists as well as other disciplines developing applications to run on supercomputers at NCSA and elsewhere. Built on the Cactus Framework, the NSF-supported Einstein Toolkit developed at Georgia Tech, RIT, LSU, AEI, Perimeter Institute and elsewhere now supports many numerical relativity groups modeling sources important for LIGO on the NCSA Blue Waters supercomputer.

“This historic announcement is very special for me. My career has centered on understanding the nature of black hole systems, from my research work in numerical relativity, to building collaborative teams and technologies for scientific research, and then also having the honor to be involved in LIGO during my role as NSF Assistant Director of Mathematics and Physical Sciences. I could not be more excited that the field is advancing to a new phase,” said Seidel, who is also Founder Professor of Physics and professor of astronomy at Illinois.

Gabrielle Allen, professor of astronomy at Illinois and NCSA associate director, previously led the development of the Cactus Framework and the Einstein Toolkit. “NCSA was a critical part of inspiring and supporting the development of Cactus for astrophysics. We held our first Cactus workshop at NCSA and the staff’s involvement in our projects was fundamental to being able to demonstrate not just new science but new computing technologies and approaches,” said Allen.

Last October Smarr was awarded an NSF grant as the principal investigator for the Pacific Research Platform. PRP is a five-year award to UC San Diego and UC Berkeley to establish a science-driven high-capacity data-centric “freeway system” on a large regional-scale. PRP’s data-sharing architecture, with connections of 10-100 gigabits per second, will enable region-wide virtual co-location of data with computing resources and enhanced security options.

PRP links most of the research universities on the West Coast (the 10 campuses of the University of California, San Diego State University, Caltech, USC, Stanford, University of Washington) via the Corporation for Education Network Initiatives in California (CENIC)/Pacific Wave’s 100G infrastructure. PRP also extends to the University of Hawaii System, Montana State University, University of Illinois at Chicago, Northwestern, and University of Amsterdam. Other research institutions in the PRP include Lawrence Berkeley National Laboratory and four national supercomputer centers. In addition, PRP will interconnect with the experimental research testbed Chameleon NSFCloud and the Chicago StarLight/MREN community.

“So yesterday’s LIGO announcement was for the first event detected. The expectation is for many more, and more regularly. We’ll be hooking up Caltech to SDSC at 10/100Gbs to do data analysis on supercomputers as LIGO goes from just one event to many events a year,” said Smarr.

Smarr noted the data-intensive science that PRP will enable is a “completely separate world from simulation and solving the Einstein’s equations” but quickly added, “I feel privileged that I started off helping figure out how to do the simulations and I’m ending up 40 years later helping LIGO to the data science.”

Looking back, Smarr said “After my PhD, I realized that we were going to eventually create a new gravitational wave astronomy, and so in 1978 I organized a two-week international workshop on the potential sources of gravitational radiation and then I edited the book which you can get on Amazon even today (Sources of Gravitational Radiation: Proceedings of the Battelle Seattle Workshop). The first paper in this collection of papers that we spent two weeks working on is from Ray Weiss, who was the inventor of LIGO and the podium at the announcement, and the third article was by Kip Thorne [also at ceremony]. So in ‘78 the global community was being formed and over the next four decades grew to what you see today.”

Further reading

NCSA article: Gravitational waves detected 100 years after Einstein’s prediction http://www.ncsa.illinois.edu/news/story/gravitational_waves_detected_100_years_after_einsteins_prediction

New York Times article and great video: Gravitational Waves Detected, Confirming Einstein’s Theory, http://www.nytimes.com/2016/02/12/science/ligo-gravitational-waves-black-holes-einstein.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news&_r=0

Physical Review Letters: Observation of Gravitational Waves from a Binary Black Hole Merger, https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.061102

APS Physics: Viewpoint: The First Sounds of Merging Black Holes, http://physics.aps.org/articles/v9/17

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

Long Flights to Cluster Fights: Meet the Asian Student Cluster Teams

November 22, 2017

Five teams from Asia traveled thousands of miles to compete at the SC17 Student Cluster Competition in Denver. Our cameras were there to meet ‘em, greet ‘em, and grill ‘em about their clusters and how they’re doi Read more…

By Dan Olds

Japan Unveils Quantum Neural Network

November 22, 2017

The U.S. and China are leading the race toward productive quantum computing, but it's early enough that ultimate leadership is still something of an open question. The latest geo-region to throw its hat in the quantum co Read more…

By Tiffany Trader

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC thinking of booth visitors. Attendees of tutorials, workshop Read more…

By Andrew Jones

HPE Extreme Performance Solutions

HPE Wins “Best HPC Server” for the Apollo 6000 Gen10 System

Hewlett Packard Enterprise (HPE) was nominated for 14 HPCwire Readers’ and Editors’ Choice Awards—including “Best High Performance Computing (HPC) Server Product or Technology” and “Top Supercomputing Achievement.” The HPE Apollo 6000 Gen10 was named “Best HPC Server” of 2017. Read more…

Turnaround Complete, HPE’s Whitman Departs

November 22, 2017

Having turned around the aircraft carrier the Silicon Valley icon had become, Meg Whitman is leaving the helm of a restructured Hewlett Packard. Her successor, technologist Antonio Neri will now guide what Whitman assert Read more…

By George Leopold

Long Flights to Cluster Fights: Meet the Asian Student Cluster Teams

November 22, 2017

Five teams from Asia traveled thousands of miles to compete at the SC17 Student Cluster Competition in Denver. Our cameras were there to meet ‘em, greet ‘em Read more…

By Dan Olds

Perspective: What Really Happened at SC17?

November 22, 2017

SC is over. Now comes the myriad of follow-ups. Inboxes are filled with templated emails from vendors and other exhibitors hoping to win a place in the post-SC Read more…

By Andrew Jones

SC Bids Farewell to Denver, Heads to Dallas for 30th Anniversary

November 17, 2017

After a jam-packed four-day expo and intensive six-day technical program, SC17 has wrapped up another successful event that brought together nearly 13,000 visit Read more…

By Tiffany Trader

SC17 Keynote – HPC Powers SKA Efforts to Peer Deep into the Cosmos

November 17, 2017

This week’s SC17 keynote – Life, the Universe and Computing: The Story of the SKA Telescope – was a powerful pitch for the potential of Big Science projects that also showcased the foundational role of high performance computing in modern science. It was also visually stunning. Read more…

By John Russell

How Cities Use HPC at the Edge to Get Smarter

November 17, 2017

Cities are sensoring up, collecting vast troves of data that they’re running through predictive models and using the insights to solve problems that, in some Read more…

By Doug Black

Student Cluster LINPACK Record Shattered! More LINs Packed Than Ever before!

November 16, 2017

Nanyang Technological University, the pride of Singapore, utterly destroyed the Student Cluster Competition LINPACK record by posting a score of 51.77 TFlop/s a Read more…

By Dan Olds

Hyperion Market Update: ‘Decent’ Growth Led by HPE; AI Transparency a Risk Issue

November 15, 2017

The HPC market update from Hyperion Research (formerly IDC) at the annual SC conference is a business and social “must,” and this year’s presentation at S Read more…

By Doug Black

Nvidia Focuses Its Cloud Containers on HPC Applications

November 14, 2017

Having migrated its top-of-the-line datacenter GPU to the largest cloud vendors, Nvidia is touting its Volta architecture for a range of scientific computing ta Read more…

By George Leopold

US Coalesces Plans for First Exascale Supercomputer: Aurora in 2021

September 27, 2017

At the Advanced Scientific Computing Advisory Committee (ASCAC) meeting, in Arlington, Va., yesterday (Sept. 26), it was revealed that the "Aurora" supercompute Read more…

By Tiffany Trader

NERSC Scales Scientific Deep Learning to 15 Petaflops

August 28, 2017

A collaborative effort between Intel, NERSC and Stanford has delivered the first 15-petaflops deep learning software running on HPC platforms and is, according Read more…

By Rob Farber

Oracle Layoffs Reportedly Hit SPARC and Solaris Hard

September 7, 2017

Oracle’s latest layoffs have many wondering if this is the end of the line for the SPARC processor and Solaris OS development. As reported by multiple sources Read more…

By John Russell

AMD Showcases Growing Portfolio of EPYC and Radeon-based Systems at SC17

November 13, 2017

AMD’s charge back into HPC and the datacenter is on full display at SC17. Having launched the EPYC processor line in June along with its MI25 GPU the focus he Read more…

By John Russell

Nvidia Responds to Google TPU Benchmarking

April 10, 2017

Nvidia highlights strengths of its newest GPU silicon in response to Google's report on the performance and energy advantages of its custom tensor processor. Read more…

By Tiffany Trader

Google Releases Deeplearn.js to Further Democratize Machine Learning

August 17, 2017

Spreading the use of machine learning tools is one of the goals of Google’s PAIR (People + AI Research) initiative, which was introduced in early July. Last w Read more…

By John Russell

GlobalFoundries Puts Wind in AMD’s Sails with 12nm FinFET

September 24, 2017

From its annual tech conference last week (Sept. 20), where GlobalFoundries welcomed more than 600 semiconductor professionals (reaching the Santa Clara venue Read more…

By Tiffany Trader

Amazon Debuts New AMD-based GPU Instances for Graphics Acceleration

September 12, 2017

Last week Amazon Web Services (AWS) streaming service, AppStream 2.0, introduced a new GPU instance called Graphics Design intended to accelerate graphics. The Read more…

By John Russell

Leading Solution Providers

SC17 Booth Video Tours

EU Funds 20 Million Euro ARM+FPGA Exascale Project

September 7, 2017

At the Barcelona Supercomputer Centre on Wednesday (Sept. 6), 16 partners gathered to launch the EuroEXA project, which invests €20 million over three-and-a-half years into exascale-focused research and development. Led by the Horizon 2020 program, EuroEXA picks up the banner of a triad of partner projects — ExaNeSt, EcoScale and ExaNoDe — building on their work... Read more…

By Tiffany Trader

Delays, Smoke, Records & Markets – A Candid Conversation with Cray CEO Peter Ungaro

October 5, 2017

Earlier this month, Tom Tabor, publisher of HPCwire and I had a very personal conversation with Cray CEO Peter Ungaro. Cray has been on something of a Cinderell Read more…

By Tiffany Trader & Tom Tabor

Cray Moves to Acquire the Seagate ClusterStor Line

July 28, 2017

This week Cray announced that it is picking up Seagate's ClusterStor HPC storage array business for an undisclosed sum. "In short we're effectively transitioning the bulk of the ClusterStor product line to Cray," said CEO Peter Ungaro. Read more…

By Tiffany Trader

Reinders: “AVX-512 May Be a Hidden Gem” in Intel Xeon Scalable Processors

June 29, 2017

Imagine if we could use vector processing on something other than just floating point problems.  Today, GPUs and CPUs work tirelessly to accelerate algorithms Read more…

By James Reinders

Intel Launches Software Tools to Ease FPGA Programming

September 5, 2017

Field Programmable Gate Arrays (FPGAs) have a reputation for being difficult to program, requiring expertise in specialty languages, like Verilog or VHDL. Easin Read more…

By Tiffany Trader

HPC Chips – A Veritable Smorgasbord?

October 10, 2017

For the first time since AMD's ill-fated launch of Bulldozer the answer to the question, 'Which CPU will be in my next HPC system?' doesn't have to be 'Whichever variety of Intel Xeon E5 they are selling when we procure'. Read more…

By Dairsie Latimer

Flipping the Flops and Reading the Top500 Tea Leaves

November 13, 2017

The 50th edition of the Top500 list, the biannual publication of the world’s fastest supercomputers based on public Linpack benchmarking results, was released Read more…

By Tiffany Trader

IBM Advances Web-based Quantum Programming

September 5, 2017

IBM Research is pairing its Jupyter-based Data Science Experience notebook environment with its cloud-based quantum computer, IBM Q, in hopes of encouraging a new class of entrepreneurial user to solve intractable problems that even exceed the capabilities of the best AI systems. Read more…

By Alex Woodie

Share This