Dark Energy Survey Data Processed by NCSA Now Available to Scientists Everywhere

January 11, 2018

Jan. 11, 2018 — Researchers around the world can now explore the first three years of data from the Dark Energy Survey (DES) processed at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign. At a special session held during the American Astronomical Society meeting in Washington, D.C., scientists on the Dark Energy Survey announced the first major release of data from the survey including information on more than 400 million astronomical objects, including distant galaxies billions of light years away as well as stars in our own galaxy.

DES scientists are using this data, collected as part of the largest and deepest cosmological survey to date, to learn more about dark energy, the mysterious force believed to be accelerating the expansion of the universe. The survey enables astronomers to map out the galaxy distribution half-way back in cosmic time, which represents a major breakthrough in survey science and will enable many scholarly achievements in the future.

The Dark Energy Camera, the primary observation tool of DES, is one of the most powerful digital imaging devices in existence. It was built and tested at Fermilab, the lead laboratory on DES, and is mounted on the National Science Foundation’s (NSF) 4-meter Blanco telescope, part of the Cerro Tololo Inter-American Observatory in Chile, a division of the National Optical Astronomy Observatory (NOAO).

The images captured by the DECam are then sent to NCSA’s Dark Energy Survey Data Management (DESDM) Project, which has been fully developed and operated by NCSA with support from NSF. The DESDM system processes and calibrates the DES data and the DECam Community Pipeline, used by NOAO to process DECam data obtained by non-DES observers.

The DES images captured each night are received in large volumes of observations over high-speed networks from the telescope in Chile. Using the Blue Waters supercomputerIllinois Campus Cluster Program, and Open Science Grid at Fermilab, the DESDM team first processes all of the images. After each individual exposure has been processed, the DESDM team combines all of the images with the help of Blue Waters in order to create a full and deep map of the southern sky, allowing the DES project to achieve depth. Through the efforts of NCSA’s DESDM team, the raw data generated by the DECAM instrument at the CTIO observatory is turned into science-ready data products.

“We’re excited that this anticipated release of high-quality imaging and catalog data is now accessible to researchers around the world. While this project was designed with the goal of understanding dark energy and dark matter, the huge amount of data in images and catalogs, processed at the National Center for Supercomputing Applications (NCSA) at Illinois, will bring new scientific applications, challenges and opportunities for discovery not only to astronomers but also data scientists,” said Matias Carrasco Kind, Release Scientist for the Dark Energy Survey and member of the DESDM team at NCSA, who coordinated this release. “This was truly a collaborative effort from many DES members and institutions around the world,” he added.

“This is an outstanding collective achievement and with the help of NOAO and LIneA we are providing the tools and resources to access and analyze this rich and robust data in an unprecedented partnership,” added Don Petravick, Principal Investigator for the DESDM team at NCSA.

“NCSA recognized many years ago the key role that advanced computing and data management would have in astronomy and is thrilled with the results of this collaboration with campus and our partners at Fermilab and the National Optical Astronomy Observatory,” said NCSA Director Bill Gropp. NCSA’s commitment to astronomy reaches beyond DES, however, as they will also serve as the global central hub and data center for the Large Synoptic Survey Telescope (LSST) project, the next generation survey which is set to be fully operational in 2023.

In addition to announcing the public release of their data, DES scientists shared some of their preliminary cosmological findings in the special session at the American Astronomical Society meeting today, such as the discovery of a dozen new stellar streams, remnants of smaller galaxies torn apart and devoured by our Milky Way.

The public release of the first three years of DES data fulfills the project’s commitment to sharing their findings with the astronomy community and the public. The data cover the full DES footprint—about 5,000 square degrees, or one eighth of the entire sky—and include more than 38,000 exposures taken with the Dark Energy Camera. The images correspond to hundreds of terabytes of data and are being released along with catalogs of hundreds of millions of galaxies and stars.

“There are all kinds of discoveries waiting to be found in the data,” said Dark Energy Survey Director Josh Frieman of the U.S. Department of Energy’s Fermi National Accelerator Laboratory. “While DES scientists are focused on using it to learn about dark energy, we wanted to enable astronomers to explore these images in new ways, to improve our understanding of the universe.”

“The great thing about a big astronomical survey like this is that it also opens a door to many other studies, like the new stellar streams,” added Adam Bolton, Associate Director for the Community Science and Data Center at NOAO. “With the DES data now available as a ‘digital sky,’ accessible to all, my hope is that these data will lead to the crowdsourcing of new and unexpected discoveries.”

The DES data can be accessed online from the NCSA site.

One new discovery enabled by the data set is the detection of roughly a dozen new streams of stars around our Milky Way. Our home galaxy is surrounded by a massive halo of dark matter, which exerts a powerful gravitational pull on smaller, nearby galaxies. The Milky Way grows by pulling in, ripping apart and absorbing these smaller systems. As stars are torn away, they form streams across the sky that can be detected using the Dark Energy Camera. Even so, stellar streams are extremely difficult to find since they are composed of relatively few stars spread out over a large area of sky.

“It’s exciting that we found so many stellar streams,” said astrophysicist Alex Drlica-Wagner of Fermilab. “We can use these streams to measure the amount, distribution, and clumpiness of dark matter in the Milky Way. Studies of stellar streams will help constrain the fundamental properties of dark matter.”

Prior to the new discoveries by DES, only about two dozen stellar streams had been discovered. Many of them were found by the Sloan Digital Sky Survey, a precursor to the Dark Energy Survey. The effort to detect new stellar streams in the Dark Energy Survey was led by University of Chicago graduate student Nora Shipp.

“We’re interested in these streams because they teach us about the formation and structure of the Milky Way and its dark matter halo. Stellar streams give us a snapshot of a larger galaxy being built out of smaller ones,” said Shipp. “These discoveries are possible because DES is the widest, deepest and best-calibrated survey out there.”

Other DES work concerns quasars, the supermassive black holes in the the middle of many galaxies. While the black holes themselves give off almost no light, a huge surrounding disk of inspiraling material can make quasars some of the brightest objects in the universe. It was long thought that these massive disks could only change slowly and would be stable for thousands or even millions of years. But recently, astronomers began noticing that some rare quasars became brighter or dimmer by factors of 2 or even 10 in only a few years.

Dr. Nick Rumbaugh and Prof. Yue Shen at the University of Illinois at Urbana-Champaign used DES data to study for these Extremely Variable Quasars. They identified 977 quasars with massive luminosity changes that occurred in only fifteen years or less. Their work suggests than many if not most quasars routinely undergo such wild changes. What’s more, it suggests that this variability is a normal part of a quasar’s life cycle and not due to a passing galactic cloud obscuring the quasar or a star suddenly slamming into the black hole.

Currently in its fifth year of observation, DES plans one more major public data release after the survey is completed. The second release will be the same one eighth of the sky but much deeper and more detailed.

This work is supported in part by the U.S. Department of Energy Office of Science.

About DES

The Dark Energy Survey (DES) is a collaboration of more than 400 scientists from 26 institutions in seven countries. Funding for the DES Projects has been provided by the U.S. Department of Energy Office of Science, U.S. National Science Foundation, Ministry of Science and Education of Spain, Science and Technology Facilities Council of the United Kingdom, Higher Education Funding Council for England, ETH Zurich for Switzerland, National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign, Kavli Institute of Cosmological Physics at the University of Chicago, Center for Cosmology and AstroParticle Physics at Ohio State University, Mitchell Institute for Fundamental Physics and Astronomy at Texas A&M University, Financiadora de Estudos e Projetos, Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro, Conselho Nacional de Desenvolvimento Científico e Tecnológico and Ministério da Ciência e Tecnologia, Deutsche Forschungsgemeinschaft, and the collaborating institutions in the Dark Energy Survey, the list of which can be found at www.darkenergysurvey.org/collaboration.


Source: NCSA

Subscribe to HPCwire's Weekly Update!

Be the most informed person in the room! Stay ahead of the tech trends with industy updates delivered to you every week!

What’s New in Computing vs. COVID-19: AMD, Remdesivir, Fab Spending & More

September 29, 2020

Supercomputing, big data and artificial intelligence are crucial tools in the fight against the coronavirus pandemic. Around the world, researchers, corporations and governments are urgently devoting their computing reso Read more…

By Oliver Peckham

Global QC Market Projected to Grow to More Than $800 million by 2024

September 28, 2020

The Quantum Economic Development Consortium (QED-C) and Hyperion Research are projecting that the global quantum computing (QC) market - worth an estimated $320 million in 2020 - will grow at an anticipated 27% CAGR betw Read more…

By Staff Reports

DoE’s ASCAC Backs AI for Science Program that Emulates the Exascale Initiative

September 28, 2020

Roughly a year after beginning formal efforts to explore an AI for Science initiative the Department of Energy’s Advanced Scientific Computing Advisory Committee last week accepted a subcommittee report calling for a t Read more…

By John Russell

Supercomputer Research Aims to Supercharge COVID-19 Antiviral Remdesivir

September 25, 2020

Remdesivir is one of a handful of therapeutic antiviral drugs that have been proven to improve outcomes for COVID-19 patients, and as such, is a crucial weapon in the fight against the pandemic – especially in the abse Read more…

By Oliver Peckham

NOAA Announces Major Upgrade to Ensemble Forecast Model, Extends Range to 35 Days

September 23, 2020

A bit over a year ago, the United States’ Global Forecast System (GFS) received a major upgrade: a new dynamical core – its first in 40 years – called the finite-volume cubed-sphere, or FV3. Now, the National Oceanic and Atmospheric Administration (NOAA) is bringing the FV3 dynamical core to... Read more…

By Oliver Peckham

AWS Solution Channel

The Water Institute of the Gulf runs compute-heavy storm surge and wave simulations on AWS

The Water Institute of the Gulf (Water Institute) runs its storm surge and wave analysis models on Amazon Web Services (AWS)—a task that sometimes requires large bursts of compute power. Read more…

Intel® HPC + AI Pavilion

Berlin Institute of Health: Putting HPC to Work for the World

Researchers from the Center for Digital Health at the Berlin Institute of Health (BIH) are using science to understand the pathophysiology of COVID-19, which can help to inform the development of targeted treatments. Read more…

AI Silicon Startup Graphcore Launches Channel Partner Program

September 23, 2020

AI compute platform vendor Graphcore has launched its first formal global channel partner program to promote and boost the sales of its AI processors and blade computing products. The formalized, all-new Graphcore Elite Partner Program follows the company’s past history of working with several... Read more…

By Todd R. Weiss

DoE’s ASCAC Backs AI for Science Program that Emulates the Exascale Initiative

September 28, 2020

Roughly a year after beginning formal efforts to explore an AI for Science initiative the Department of Energy’s Advanced Scientific Computing Advisory Commit Read more…

By John Russell

NOAA Announces Major Upgrade to Ensemble Forecast Model, Extends Range to 35 Days

September 23, 2020

A bit over a year ago, the United States’ Global Forecast System (GFS) received a major upgrade: a new dynamical core – its first in 40 years – called the finite-volume cubed-sphere, or FV3. Now, the National Oceanic and Atmospheric Administration (NOAA) is bringing the FV3 dynamical core to... Read more…

By Oliver Peckham

Arm Targets HPC with New Neoverse Platforms

September 22, 2020

UK-based semiconductor design company Arm today teased details of its Neoverse roadmap, introducing V1 (codenamed Zeus) and N2 (codenamed Perseus), Arm’s second generation N-series platform. The chip IP vendor said the new platforms will deliver 50 percent and 40 percent more... Read more…

By Tiffany Trader

Oracle Cloud Deepens HPC Embrace with Launch of A100 Instances, Plans for Arm, More 

September 22, 2020

Oracle Cloud Infrastructure (OCI) continued its steady ramp-up of HPC capabilities today with a flurry of announcements. Topping the list is general availabilit Read more…

By John Russell

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

Future of Fintech on Display at HPC + AI Wall Street

September 17, 2020

Those who tuned in for Tuesday's HPC + AI Wall Street event got a peak at the future of fintech and lively discussion of topics like blockchain, AI for risk man Read more…

By Alex Woodie, Tiffany Trader and Todd R. Weiss

IBM’s Quantum Race to One Million Qubits

September 15, 2020

IBM today outlined its ambitious quantum computing technology roadmap at its virtual Quantum Summit. The eye-popping million qubit number is still far out, agrees IBM, but perhaps not that far out. Just as eye-popping is IBM’s nearer-term plan for a 1,000-plus qubit system named Condor... Read more…

By John Russell

Supercomputer-Powered Research Uncovers Signs of ‘Bradykinin Storm’ That May Explain COVID-19 Symptoms

July 28, 2020

Doctors and medical researchers have struggled to pinpoint – let alone explain – the deluge of symptoms induced by COVID-19 infections in patients, and what Read more…

By Oliver Peckham

Nvidia Said to Be Close on Arm Deal

August 3, 2020

GPU leader Nvidia Corp. is in talks to buy U.K. chip designer Arm from parent company Softbank, according to several reports over the weekend. If consummated Read more…

By George Leopold

10nm, 7nm, 5nm…. Should the Chip Nanometer Metric Be Replaced?

June 1, 2020

The biggest cool factor in server chips is the nanometer. AMD beating Intel to a CPU built on a 7nm process node* – with 5nm and 3nm on the way – has been i Read more…

By Doug Black

Intel’s 7nm Slip Raises Questions About Ponte Vecchio GPU, Aurora Supercomputer

July 30, 2020

During its second-quarter earnings call, Intel announced a one-year delay of its 7nm process technology, which it says it will create an approximate six-month shift for its CPU product timing relative to prior expectations. The primary issue is a defect mode in the 7nm process that resulted in yield degradation... Read more…

By Tiffany Trader

Google Hires Longtime Intel Exec Bill Magro to Lead HPC Strategy

September 18, 2020

In a sign of the times, another prominent HPCer has made a move to a hyperscaler. Longtime Intel executive Bill Magro joined Google as chief technologist for hi Read more…

By Tiffany Trader

HPE Keeps Cray Brand Promise, Reveals HPE Cray Supercomputing Line

August 4, 2020

The HPC community, ever-affectionate toward Cray and its eponymous founder, can breathe a (virtual) sigh of relief. The Cray brand will live on, encompassing th Read more…

By Tiffany Trader

Neocortex Will Be First-of-Its-Kind 800,000-Core AI Supercomputer

June 9, 2020

Pittsburgh Supercomputing Center (PSC - a joint research organization of Carnegie Mellon University and the University of Pittsburgh) has won a $5 million award Read more…

By Tiffany Trader

European Commission Declares €8 Billion Investment in Supercomputing

September 18, 2020

Just under two years ago, the European Commission formalized the EuroHPC Joint Undertaking (JU): a concerted HPC effort (comprising 32 participating states at c Read more…

By Oliver Peckham

Leading Solution Providers

Contributors

Oracle Cloud Infrastructure Powers Fugaku’s Storage, Scores IO500 Win

August 28, 2020

In June, RIKEN shook the supercomputing world with its Arm-based, Fujitsu-built juggernaut: Fugaku. The system, which weighs in at 415.5 Linpack petaflops, topp Read more…

By Oliver Peckham

Google Cloud Debuts 16-GPU Ampere A100 Instances

July 7, 2020

On the heels of the Nvidia’s Ampere A100 GPU launch in May, Google Cloud is announcing alpha availability of the A100 “Accelerator Optimized” VM A2 instance family on Google Compute Engine. The instances are powered by the HGX A100 16-GPU platform, which combines two HGX A100 8-GPU baseboards using... Read more…

By Tiffany Trader

DOD Orders Two AI-Focused Supercomputers from Liqid

August 24, 2020

The U.S. Department of Defense is making a big investment in data analytics and AI computing with the procurement of two HPC systems that will provide the High Read more…

By Tiffany Trader

Supercomputer Modeling Tests How COVID-19 Spreads in Grocery Stores

April 8, 2020

In the COVID-19 era, many people are treating simple activities like getting gas or groceries with caution as they try to heed social distancing mandates and protect their own health. Still, significant uncertainty surrounds the relative risk of different activities, and conflicting information is prevalent. A team of Finnish researchers set out to address some of these uncertainties by... Read more…

By Oliver Peckham

Microsoft Azure Adds A100 GPU Instances for ‘Supercomputer-Class AI’ in the Cloud

August 19, 2020

Microsoft Azure continues to infuse its cloud platform with HPC- and AI-directed technologies. Today the cloud services purveyor announced a new virtual machine Read more…

By Tiffany Trader

Japan’s Fugaku Tops Global Supercomputing Rankings

June 22, 2020

A new Top500 champ was unveiled today. Supercomputer Fugaku, the pride of Japan and the namesake of Mount Fuji, vaulted to the top of the 55th edition of the To Read more…

By Tiffany Trader

Joliot-Curie Supercomputer Used to Build First Full, High-Fidelity Aircraft Engine Simulation

July 14, 2020

When industrial designers plan the design of a new element of a vehicle’s propulsion or exterior, they typically use fluid dynamics to optimize airflow and in Read more…

By Oliver Peckham

Intel Speeds NAMD by 1.8x: Saves Xeon Processor Users Millions of Compute Hours

August 12, 2020

Potentially saving datacenters millions of CPU node hours, Intel and the University of Illinois at Urbana–Champaign (UIUC) have collaborated to develop AVX-512 optimizations for the NAMD scalable molecular dynamics code. These optimizations will be incorporated into release 2.15 with patches available for earlier versions. Read more…

By Rob Farber

  • arrow
  • Click Here for More Headlines
  • arrow
Do NOT follow this link or you will be banned from the site!
Share This