A big data tool developed by IBM in partnership with Deutsches Elektronen-Synchrotron (DESY) will enable scientists around the world to more quickly manage and share massive volumes of x-ray data produced by a super microscope in Germany.

German research center DESY develops, builds and operates large particle accelerators used to investigate the structure of matter. One of these accelerators, the 1.7 mile-long PETRA III, speeds up electrically charged particles to nearly the speed of light – about 186,000 miles per second – and sends them onto a magnetic track to generate remarkably brilliant x-rays, known as synchrotron radiation.

About 2,000 scientists a year use the instrument to study the atomic structure of novel semiconductors, catalysts, biological cells and other materials. This translates to huge volumes of x-ray data. The challenge prompted IBM and DESY to implement a Big Data and Analytics architecture using IBM’s software-defined technology, codenamed Elastic Storage, which can handle more than 20 gigabytes of data per second at peak performance.

“A typical detector generates a data stream of about 5 Gigabit per second, which is about the data volume of one complete CD-ROM per second,” said Dr. Volker Gülzow, head of DESY IT. “And at PETRA III we do not have just one detector, but 14 beamlines equipped with many detectors, and they are currently being extended to 24. All this Big Data must be stored and handled reliably.”

Elastic Storage is described as a scalable, high-performance data and file management solution, based upon General Parallel File System (GPFS) technology, that offers:

• Enhanced security – native encryption and secure erase, NIST SP 800-131A encryption compliance.
• Increased performance – Server-side Elastic Storage Flash caches increase IO performance up to 6X.
• Improved usability – data migration; AFM, FPO, and backup/restore enhancements; reliability, availability and serviceability enhancements.

The storage architecture empowers geographically distributed workflows by placing critical data close to everyone and everything that needs it, no matter where they are in the world.

IBM reports the technology will enable DESY to provide analysis-as-a-service and cloud-based solutions to its worldwide user base. A plan is also in the works to expand the big data architecture to support the European x-ray free electron laser (European XFEL), an x-ray research laser facility currently under construction that is scheduled to start operation in 2017.

“We expect about 100 Petabyte per year from the European XFEL,” said Dr. Gülzow. That is on par with the data volume produced at the world’s largest particle accelerator, the Large Hadron Collider (LHC) at CERN in Geneva.