Gravitational Waves Detected! Historic LIGO Success Strikes Chord with Larry Smarr

By John Russell

February 12, 2016

By now you’ve likely heard that scientists reported detecting the long-sought gravitational waves; this is roughly a 100 years since their prediction by Einstein’s theory of general relativity and more than 20 years since first funding for the LIGO (Laser Interferometer Gravitational-wave Observatory (LIGO) project to find them. You can safely assume the National Science Foundation (NSF), a prime funder of the LIGO, is feeling well rewarded (and relieved) for the $620 million invested. It was a big risk/big reward project.

The event that created these particular gravitational waves – the spiraling of two black holes into each other about 1.3 billion light-years away (or 1.3 billion years ago) – was the most energetic event ever detected. For a brief moment, the collision of these black holes generated more power than all of the stars in the universe, meaning it briefly outshone the output of 100 billion galaxies, each with 100 billion stars.

“It’s hard to overstate how historic the moment is today. Think about all of the amazing astronomical objects we’re studying with radio astronomy, x-ray astronomy, ultraviolet astronomy, infrared astronomy or optical. Those are all just parts of the electromagnetic spectrum. This is an entirely new spectrum, from end to end, with as diverse sources as the electromagnetic spectrum that we have been working on since Galileo,” said Larry Smarr, founding director of the California Institute for Telecommunications and Information Technology (Calit2), a UC San Diego/UC Irvine partnership, who holds the Harry E. Gruber professorship in Computer Science and Engineering (CSE) at UCSD’s Jacobs School.

Larry Smarr
Larry Smarr

Modern science, of course, is propelled forward in large measure by advanced computing; in this case Smarr was one of the early architects of supercomputing tools needed to solve Einstein’s equations. His Ph.D. theses written in 1975 – The Structure of General Relativity with a Numerical Illustration: The Collision of Two Black Holes – worked out the foundations of numerical relativity and began winning over theoretical physicists whose bias then favored analytic over computational methods.

First, let’s look at the remarkable LIGO project achievement itself. LIGO was originally proposed as a means of detecting gravitational waves in the 1980s by Rainer Weiss, professor of physics, emeritus, from MIT; Kip Thorne, Caltech’s Richard P. Feynman Professor of Theoretical Physics, emeritus; and Ronald Drever, professor of physics, emeritus, also from Caltech. Both Weiss and Thorne were present at the announcement.

Besides providing more proof for Einstein’s general theory of relativity, the ability to detect gravity waves opens an entirely new window through which to observe the universe. This event was the collision of relatively modest size black holes. As LIGO’s sensitivity is improved – it’s at one third of its spec now and expected to reach full sensitivity fairly quickly – a wealth of new and fascinating phenomena will be observed. Nobel prizes will ensue.

The achievement was announced yesterday at the annual American Association for the Advancement of Science (AAAS) meeting being held in Washington over the next few days. Gravitational waves carry information about their dramatic origins and about the nature of gravity that cannot be obtained from elsewhere. Physicists have concluded that the detected gravitational waves were produced during the final fraction of a second of the merger of two black holes to produce a single, more massive spinning black hole. This collision of two black holes had been predicted but never observed.

LIGO event signals
LIGO event signals

For the record, these gravitational waves were detected on Sept. 14, 2015 at 5:51 a.m. EDT (09:51 UTC) by both of the twin Laser Interferometer Gravitational-wave Observatory (LIGO) detectors, located in Livingston, Louisiana, and Hanford, Washington. The LIGO observatories were conceived, built and are operated by Caltech and MIT. The discovery, accepted for publication in the journal Physical Review Letters, was made by the LIGO Scientific Collaboration (which includes the GEO Collaboration and the Australian Consortium for Interferometric Gravitational Astronomy) and the Virgo Collaboration using data from the two LIGO detectors.

Based on the observed signals, LIGO scientists estimate that the black holes for this event were about 29 and 36 times the mass of the sun, and the event took place 1.3 billion years ago. About three times the mass of the sun was converted into gravitational wave energy in a fraction of a second — with a peak power output about 50 times that of the whole visible stellar universe. By looking at the time of arrival of the signals — the detector in Livingston recorded the event 7 milliseconds before the detector in Hanford — scientists can say that the source was located in the Southern Celestial Hemisphere.

LIGO research is carried out by the LIGO Scientific Collaboration (LSC), a group of more than 1,000 scientists from universities around the United States and in 14 other countries. More than 90 universities and research institutes in the LSC develop detector technology and analyze data; approximately 250 students are strong contributing members of the collaboration. The LSC detector network includes the LIGO interferometers and the GEO600 detector. The GEO team includes scientists at the Max Planck Institute for Gravitational Physics (Albert Einstein Institute, AEI), Leibniz Universität Hannover, along with partners at the University of Glasgow, Cardiff University, the University of Birmingham, other universities in the United Kingdom and the University of the Balearic Islands in Spain.

“This detection is the beginning of a new era: The field of gravitational wave astronomy is now a reality,” says Gabriela González, LSC spokesperson and professor of physics and astronomy at Louisiana State University.

The discovery was made possible by the enhanced capabilities of Advanced LIGO, a major upgrade that increases the sensitivity of the instruments compared to the first generation LIGO detectors, enabling a large increase in the volume of the universe probed — and the discovery of gravitational waves during its first observation run. NSF is the lead financial supporter of Advanced LIGO. Funding organizations in Germany (Max Planck Society), the U.K. (Science and Technology Facilities Council, STFC) and Australia (Australian Research Council) also have made significant commitments to the project.

There are many excellent accounts of the LIGO achievement (see links at the end of this article). The New York Times has a good brief video. There’s also a paper describing the LIGO discovery published online in Physical Review Letters (Observation of Gravitational Waves from a Binary Black Hole Merger) at the same time as yesterday’s announcement. Yet another interesting thread in the LIGO success story is Smarr’s personal connection to gravitational wave research, which as mentioned earlier stretches back to his graduate school days.

Larry Smarr PhD cover page 1975“My Ph.D. dissertation was on solving Einstein’s equations for the collision of two black holes and the gravitational radiation it would generate. In those days, I started out working on a CDC 6600,” recalled Smarr.

He was at the University of Texas at Austin, which had a relativity research center, and his Ph.D. advisor was Bryce DeWitt. These were early days for supercomputing. The CDC 6600 cranked out around 3 or 4 megaflops. Smarr moved up to a CDC 7600, roughly ten-times more powerful with a peak of 36 megaflops. Today, a typical smart phone is a gigaflops device, Smarr noted.

The task was to develop computational techniques to solve Einstein’s equations for general relativity in curved space-time. “I used hundreds of hours doing that. I had to do the head-on collision, which actually makes the problem axisymmetric; so imagine two black holes gravitationally attracting each other on a line and crashing into each other. You couldn’t possibly compute orbiting. We knew that’s what we needed to do for reality, but you couldn’t do it with the supercomputers of those days.”

Access to sufficient compute power wasn’t the only challenge. Many relativity experts resisted using of computational approaches. “Most had done all of their research relying on exact analytic solutions. I was arguing for computational science and that the only way to understand the dynamics was to use computers. There were a lot of people who were the leaders of general relativity in the 60s who felt that I was besmirching their beautiful analytic solutions,” said Smarr.

Smarr credits DeWitt with the vision that you could use supercomputers to solve the Einstein equations. Upon completion of his degree, DeWitt advised Smarr “to work with the masters of computational science and use the supercomputers. I said, well where are those, and he said you’ll have to get a top secret nuclear weapons clearance to go work inside Livermore (National Laboratory) or Los Alamos (National Lab).”

At the time, said Smarr, it was considered a crazy notion that universities would be able to support national supercomputer centers as they do now. The prevailing view was universities couldn’t handle anything that complicated. He went to Livermore. While there Smarr met a graduate student, Michael Norman, who had just started doing summer research at Livermore. That initiated a four-decade collaboration on many areas of computational astrophysics. Mike is currently the director of the San Diego Supercomputer Center.

“I realized if you could solve something as complicated as general relativity on supercomputers you could also be solving many other problems in science and engineering that researchers in academia were working on. That’s what led me over the years to develop the idea of putting in a proposal [along those lines] to NSF.”

So it is that Smarr’s graduate school experience working to computationally describe the collision of two black holes using Einstein’s equations led to a broader interest in solving science problems with supercomputers. He also became convinced that such computers could thrive usefully in academia, and in 1983 made an unsolicited proposal to NSF that ultimately became NCSA – the National Center for Supercomputing Applications.

Smarr was awarded the Golden Goose award in 2014 which the American Association for the Advancement of Science, the American Association of Universities, other organizations, and supporters in Congress gives to people whose seemingly foolish government funded ideas turn out to be whopping successes for society at large. “It’s given to people who are working on some totally obscure project by that people would normally make fun of and that ends up changing the World,” laughed Smarr.

“When I got to NCSA I wanted to continue my personal research on colliding black holes and so I hired a bright energetic post doc, Ed Seidel who is now the director of NCSA and in between was the NSF Associate Director for Math and Physical Sciences, which has the funding for LIGO. You can’t make this stuff up,” said Smarr who served as NCSA director for 15 years.

At NCSA, Smarr formed a numerical group, led by Seidel. The group quickly became a leader in applying supercomputers to black hole and gravitational wave problems. For example, in 1994 one of the first 3-dimension simulation of two colliding black holes providing computed gravitational waveforms was carried out at NCSA by this group in collaboration with colleagues at Washington University.

NCSANCSA as a center has continued to support the most complex problems in numerical relativity and relativistic astrophysics, including working with several groups addressing models of gravitational waves sources seen by LIGO in this discovery. Even more complex simulations will be needed for anticipated future discoveries such as colliding neutron stars and black holes or supernovae explosions.

NCSA has also played a role in developing the tools needed for simulating relativistic systems. The work of Seidel’s NCSA group led to the development of the Cactus Framework, a modular and collaborative framework for parallel computing which since 1997 has supported numerical relativists as well as other disciplines developing applications to run on supercomputers at NCSA and elsewhere. Built on the Cactus Framework, the NSF-supported Einstein Toolkit developed at Georgia Tech, RIT, LSU, AEI, Perimeter Institute and elsewhere now supports many numerical relativity groups modeling sources important for LIGO on the NCSA Blue Waters supercomputer.

“This historic announcement is very special for me. My career has centered on understanding the nature of black hole systems, from my research work in numerical relativity, to building collaborative teams and technologies for scientific research, and then also having the honor to be involved in LIGO during my role as NSF Assistant Director of Mathematics and Physical Sciences. I could not be more excited that the field is advancing to a new phase,” said Seidel, who is also Founder Professor of Physics and professor of astronomy at Illinois.

Gabrielle Allen, professor of astronomy at Illinois and NCSA associate director, previously led the development of the Cactus Framework and the Einstein Toolkit. “NCSA was a critical part of inspiring and supporting the development of Cactus for astrophysics. We held our first Cactus workshop at NCSA and the staff’s involvement in our projects was fundamental to being able to demonstrate not just new science but new computing technologies and approaches,” said Allen.

Last October Smarr was awarded an NSF grant as the principal investigator for the Pacific Research Platform. PRP is a five-year award to UC San Diego and UC Berkeley to establish a science-driven high-capacity data-centric “freeway system” on a large regional-scale. PRP’s data-sharing architecture, with connections of 10-100 gigabits per second, will enable region-wide virtual co-location of data with computing resources and enhanced security options.

PRP links most of the research universities on the West Coast (the 10 campuses of the University of California, San Diego State University, Caltech, USC, Stanford, University of Washington) via the Corporation for Education Network Initiatives in California (CENIC)/Pacific Wave’s 100G infrastructure. PRP also extends to the University of Hawaii System, Montana State University, University of Illinois at Chicago, Northwestern, and University of Amsterdam. Other research institutions in the PRP include Lawrence Berkeley National Laboratory and four national supercomputer centers. In addition, PRP will interconnect with the experimental research testbed Chameleon NSFCloud and the Chicago StarLight/MREN community.

“So yesterday’s LIGO announcement was for the first event detected. The expectation is for many more, and more regularly. We’ll be hooking up Caltech to SDSC at 10/100Gbs to do data analysis on supercomputers as LIGO goes from just one event to many events a year,” said Smarr.

Smarr noted the data-intensive science that PRP will enable is a “completely separate world from simulation and solving the Einstein’s equations” but quickly added, “I feel privileged that I started off helping figure out how to do the simulations and I’m ending up 40 years later helping LIGO to the data science.”

Looking back, Smarr said “After my PhD, I realized that we were going to eventually create a new gravitational wave astronomy, and so in 1978 I organized a two-week international workshop on the potential sources of gravitational radiation and then I edited the book which you can get on Amazon even today (Sources of Gravitational Radiation: Proceedings of the Battelle Seattle Workshop). The first paper in this collection of papers that we spent two weeks working on is from Ray Weiss, who was the inventor of LIGO and the podium at the announcement, and the third article was by Kip Thorne [also at ceremony]. So in ‘78 the global community was being formed and over the next four decades grew to what you see today.”

Further reading

NCSA article: Gravitational waves detected 100 years after Einstein’s prediction http://www.ncsa.illinois.edu/news/story/gravitational_waves_detected_100_years_after_einsteins_prediction

New York Times article and great video: Gravitational Waves Detected, Confirming Einstein’s Theory, http://www.nytimes.com/2016/02/12/science/ligo-gravitational-waves-black-holes-einstein.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news&_r=0

Physical Review Letters: Observation of Gravitational Waves from a Binary Black Hole Merger, https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.061102

APS Physics: Viewpoint: The First Sounds of Merging Black Holes, http://physics.aps.org/articles/v9/17

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

University of Chicago Researchers Generate First Computational Model of Entire SARS-CoV-2 Virus

January 15, 2021

Over the course of the last year, many detailed computational models of SARS-CoV-2 have been produced with the help of supercomputers, but those models have largely focused on critical elements of the virus, such as its Read more…

By Oliver Peckham

Pat Gelsinger Returns to Intel as CEO

January 14, 2021

The Intel board of directors has appointed a new CEO. Intel alum Pat Gelsinger is leaving his post as CEO of VMware to rejoin the company that he parted ways with 11 years ago. Gelsinger will succeed Bob Swan, who will remain CEO until Feb. 15. Gelsinger previously spent 30 years... Read more…

By Tiffany Trader

Roar Supercomputer to Support Naval Aircraft Research

January 14, 2021

One might not think “aircraft” when picturing the U.S. Navy, but the military branch actually has thousands of aircraft currently in service – and now, supercomputing will help future naval aircraft operate faster, Read more…

By Staff report

DOE and NOAA Extend Computing Partnership, Plan for New Supercomputer

January 14, 2021

The National Climate-Computing Research Center (NCRC), hosted by Oak Ridge National Laboratory (ORNL), has been supporting the climate research of the National Oceanic and Atmospheric Administration (NOAA) for the last 1 Read more…

By Oliver Peckham

Using Micro-Combs, Researchers Demonstrate World’s Fastest Optical Neuromorphic Processor for AI

January 13, 2021

Neuromorphic computing, which uses chips that mimic the behavior of the human brain using virtual “neurons,” is growing in popularity thanks to high-profile efforts from Intel and others. Now, a team of researchers l Read more…

By Oliver Peckham

AWS Solution Channel

Now Available – Amazon EC2 C6gn Instances with 100 Gbps Networking

Amazon EC2 C6gn instances powered by AWS Graviton2 processors are now available!

Compared to C6g instances, this new instance type provides 4x higher network bandwidth, 4x higher packet processing performance, and 2x higher EBS bandwidth. Read more…

Intel® HPC + AI Pavilion

Intel Keynote Address

Intel is the foundation of HPC – from the workstation to the cloud to the backbone of the Top500. At SC20, Intel’s Trish Damkroger, VP and GM of high performance computing, addresses the audience to show how Intel and its partners are building the future of HPC today, through hardware and software technologies that accelerate the broad deployment of advanced HPC systems. Read more…

Honing In on AI, US Launches National Artificial Intelligence Initiative Office

January 13, 2021

To drive American leadership in the field of AI into the future, the National Artificial Intelligence Initiative Office has been launched by the White House Office of Science and Technology Policy (OSTP). The new agen Read more…

By Todd R. Weiss

Pat Gelsinger Returns to Intel as CEO

January 14, 2021

The Intel board of directors has appointed a new CEO. Intel alum Pat Gelsinger is leaving his post as CEO of VMware to rejoin the company that he parted ways with 11 years ago. Gelsinger will succeed Bob Swan, who will remain CEO until Feb. 15. Gelsinger previously spent 30 years... Read more…

By Tiffany Trader

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

By John Russell

Intel ‘Ice Lake’ Server Chips in Production, Set for Volume Ramp This Quarter

January 12, 2021

Intel Corp. used this week’s virtual CES 2021 event to reassert its dominance of the datacenter with the formal roll out of its next-generation server chip, the 10nm Xeon Scalable processor that targets AI and HPC workloads. The third-generation “Ice Lake” family... Read more…

By George Leopold

Researchers Say It Won’t Be Possible to Control Superintelligent AI

January 11, 2021

Worries about out-of-control AI aren’t new. Many prominent figures have suggested caution when unleashing AI. One quote that keeps cropping up is (roughly) th Read more…

By John Russell

AMD Files Patent on New GPU Chiplet Approach

January 5, 2021

Advanced Micro Devices is accelerating the GPU chiplet race with the release of a U.S. patent application for a device that incorporates high-bandwidth intercon Read more…

By George Leopold

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

Intel Touts Optane Performance, Teases Next-gen “Crow Pass”

January 5, 2021

Competition to leverage new memory and storage hardware with new or improved software to create better storage/memory schemes has steadily gathered steam during Read more…

By John Russell

Farewell 2020: Bleak, Yes. But a Lot of Good Happened Too

December 30, 2020

Here on the cusp of the new year, the catchphrase ‘2020 hindsight’ has a distinctly different feel. Good riddance, yes. But also proof of science’s power Read more…

By John Russell

Esperanto Unveils ML Chip with Nearly 1,100 RISC-V Cores

December 8, 2020

At the RISC-V Summit today, Art Swift, CEO of Esperanto Technologies, announced a new, RISC-V based chip aimed at machine learning and containing nearly 1,100 low-power cores based on the open-source RISC-V architecture. Esperanto Technologies, headquartered in... Read more…

By Oliver Peckham

Azure Scaled to Record 86,400 Cores for Molecular Dynamics

November 20, 2020

A new record for HPC scaling on the public cloud has been achieved on Microsoft Azure. Led by Dr. Jer-Ming Chia, the cloud provider partnered with the Beckman I Read more…

By Oliver Peckham

NICS Unleashes ‘Kraken’ Supercomputer

April 4, 2008

A Cray XT4 supercomputer, dubbed Kraken, is scheduled to come online in mid-summer at the National Institute for Computational Sciences (NICS). The soon-to-be petascale system, and the resulting NICS organization, are the result of an NSF Track II award of $65 million to the University of Tennessee and its partners to provide next-generation supercomputing for the nation's science community. Read more…

Is the Nvidia A100 GPU Performance Worth a Hardware Upgrade?

October 16, 2020

Over the last decade, accelerators have seen an increasing rate of adoption in high-performance computing (HPC) platforms, and in the June 2020 Top500 list, eig Read more…

By Hartwig Anzt, Ahmad Abdelfattah and Jack Dongarra

Aurora’s Troubles Move Frontier into Pole Exascale Position

October 1, 2020

Intel’s 7nm node delay has raised questions about the status of the Aurora supercomputer that was scheduled to be stood up at Argonne National Laboratory next year. Aurora was in the running to be the United States’ first exascale supercomputer although it was on a contemporaneous timeline with... Read more…

By Tiffany Trader

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Julia Update: Adoption Keeps Climbing; Is It a Python Challenger?

January 13, 2021

The rapid adoption of Julia, the open source, high level programing language with roots at MIT, shows no sign of slowing according to data from Julialang.org. I Read more…

By John Russell

Leading Solution Providers

Contributors

Programming the Soon-to-Be World’s Fastest Supercomputer, Frontier

January 5, 2021

What’s it like designing an app for the world’s fastest supercomputer, set to come online in the United States in 2021? The University of Delaware’s Sunita Chandrasekaran is leading an elite international team in just that task. Chandrasekaran, assistant professor of computer and information sciences, recently was named... Read more…

By Tracey Bryant

Top500: Fugaku Keeps Crown, Nvidia’s Selene Climbs to #5

November 16, 2020

With the publication of the 56th Top500 list today from SC20's virtual proceedings, Japan's Fugaku supercomputer – now fully deployed – notches another win, Read more…

By Tiffany Trader

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Texas A&M Announces Flagship ‘Grace’ Supercomputer

November 9, 2020

Texas A&M University has announced its next flagship system: Grace. The new supercomputer, named for legendary programming pioneer Grace Hopper, is replacing the Ada system (itself named for mathematician Ada Lovelace) as the primary workhorse for Texas A&M’s High Performance Research Computing (HPRC). Read more…

By Oliver Peckham

At Oak Ridge, ‘End of Life’ Sometimes Isn’t

October 31, 2020

Sometimes, the old dog actually does go live on a farm. HPC systems are often cursed with short lifespans, as they are continually supplanted by the latest and Read more…

By Oliver Peckham

Nvidia and EuroHPC Team for Four Supercomputers, Including Massive ‘Leonardo’ System

October 15, 2020

The EuroHPC Joint Undertaking (JU) serves as Europe’s concerted supercomputing play, currently comprising 32 member states and billions of euros in funding. I Read more…

By Oliver Peckham

Gordon Bell Special Prize Goes to Massive SARS-CoV-2 Simulations

November 19, 2020

2020 has proven a harrowing year – but it has produced remarkable heroes. To that end, this year, the Association for Computing Machinery (ACM) introduced the Read more…

By Oliver Peckham

Nvidia-Arm Deal a Boon for RISC-V?

October 26, 2020

The $40 billion blockbuster acquisition deal that will bring chipmaker Arm into the Nvidia corporate family could provide a boost for the competing RISC-V architecture. As regulators in the U.S., China and the European Union begin scrutinizing the impact of the blockbuster deal on semiconductor industry competition and innovation, the deal has at the very least... Read more…

By George Leopold

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This