Visit additional Tabor Communication Publications
September 27, 2012
One of technology’s most pervasive buzzwords echoed in the ears of attendees at this year’s one-day HPC on Wall Street conference in New York City, as panel after panel addressed the challenges and opportunities that big data presents. From the opening remarks regarding Wall Street’s traditional concern of low latency, delivered by Cisco CTO Paul Perez, to the multiple open-ended discussions that took place in concurrent panels, the “big data” problem was a much-discussed topic.
For this industry, however, the concerns around what the overall technology ecosystem is touting as big data are quite different. The exploding volume of data that other industries are dealing with is compounded in the financial space by regulations mandating massive, long-term storage.
But the industry itself is finding value in the ability to tap those datasets in both real-time and historical context. What this means is that Wall Street is looking for snappy new ways to keep the meaningful data at the fore, while maintaining a monster archive of historical transactions and other data for more leisurely access and analysis.
During the course of a panel on the exploding demands for storage, analytics, risk management and ultra-low latency (not to mention the compute horsepower required), Emile Werr, VP and Head of Enterprise Architecture at NYSE Euronext described the system-wide challenges of massive, swift data across their HPC infrastructure. He noted that, for them, the challenges went far beyond the “three Vs” of big data: volume, variety and velocity. Their entire approach and methodologies had to shift.
The volume and complexity challenges were keenly felt in the context of the volatility of changing systems, new markets, and even new businesses his firm is exploring. Note that NYSE Technologies is the spin-out company from the exchange of the same name, and offers financial services that encompasses an increasingly large buffet of software and services, from custom middleware packages to hosted exchange analysis.
They have had to keep pace with an evolving exchange market for their customers, necessitating new approaches to their system environments on both the hardware and software sides. According to him, these tweaks and new services have allowed them to expand their traditional market business significantly.
Werr, who proudly notes that he’s the “big data guy” at NYSE, says that one thing that isn’t obvious in terms of their requirements is that the data that is fed into their systems is not user-friendly and certainly doesn’t come read-made for BI platforms. This means there is a whole, often invisible layer of complex data enrichment that is required.
But when you’re talking about billions of transactions per day, building systems that can take this unfriendly data and turn it into regulation-friendly, analysis-ready information is a key, ongoing struggle. Still, they think they may have solved some pieces of that system-wide puzzle and they’re marketing their architecture as a big data, HPC problem solver for this industry.
As mentioned earlier, another aspect of NYSE’s “macro data architecture strategy” that Werr defines is the regulatory-plus-storage problem. “We are obligated to maintain data for seven years,” he said, not without some exasperation. “There’s not one system out there that could actually store that data and have it online. Besides, it wouldn’t be practical. It’s old, old data, it’s just used for regulatory needs and then maybe trending over time details.”
But if the big data hype that insists all bytes are a potential goldmine rings with any validity, NYSE Euronext has a solution that could lend some credence to that ideal. The company has developed a clever system whereupon data is scattered across distributed resources in such a way that makes it possible to provision it on the fly. Using an on-demand approach they’ve refined, the system can serve an array of applications, everything from an historical audit to an analyst’s real-time query.
NYSE Technologies is commercializing its reported success with its inventive macro data architecture, which Werr says has been rolling along nicely in production for four years. While skipping on the specifics, he noted that the system works in harmony with messaging systems and feed handlers designed to capture certain transactions with keen latency.
Those files are generated in small mini-batches and then fired off to the firm’s “transformation-archive farm” that offloads a lot of the ETL processing across a commodity cluster. The data then moves into the enrichment phase where relational models can be constructed and dropped into distributed storage for the rapid, on-demand access capabilities he hinted at earlier. At the prettier end of the process is a services layer that allows for rapid provisioning and access for all applications as well as APIs for systems and schedulers, not to mention a more seamless end-result for that data to be analyzed for any other business purpose.
A well-oiled machine, no? Werr says that it took a lot of determination to climb out of their old paradigm of being a big database shop with the standard Oracle, Sybase, etc. tools. At the heart of that shift is the need for ever-faster ingestion of data. They’re at the point now where they can load around 20 terabytes per hour into their federated server farm. Since they have a short window of genuine production data, they’re able to then quickly provision that data into sandboxes to allow for more refined operation on specific subsets of that data, or use narrowly defined tools and integration approaches.
Whether or not we want to think abstractly about this big data craze as a mere concept or hype-bubble, the fact remains that the vendors on every conference panel throughout the day seemed to find some element of value in this topic. By presenting the opportunities and challenges of all the hardware and software this technology touches, attendees were left with the impression that the financial industry is in for some major retooling.
May 16, 2013 |
When it comes to cloud, long distances mean unacceptably high latencies. Researchers from the University of Bonn in Germany examined those latency issues of doing CFD modeling in the cloud by utilizing a common CFD and its utilization in HPC instance types including both CPU and GPU cores of Amazon EC2.
May 15, 2013 |
Supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center (NERSC) have worked on important computational problems such as collapse of the atomic state, the optimization of chemical catalysts, and now modeling popping bubbles.
May 10, 2013 |
Program provides cash awards up to $10,000 for the best open-source end-user applications deployed on 100G network.
May 09, 2013 |
The Japanese government has revealed its plans to best its previous K Computer efforts with what they hope will be the first exascale system...
May 08, 2013 |
For engineers looking to leverage high-performance computing, the accessibility of a cloud-based approach is a powerful draw, but there are costs that may not be readily apparent.
05/10/2013 | Cleversafe, Cray, DDN, NetApp, & Panasas | From Wall Street to Hollywood, drug discovery to homeland security, companies and organizations of all sizes and stripes are coming face to face with the challenges – and opportunities – afforded by Big Data. Before anyone can utilize these extraordinary data repositories, however, they must first harness and manage their data stores, and do so utilizing technologies that underscore affordability, security, and scalability.
04/15/2013 | Bull | “50% of HPC users say their largest jobs scale to 120 cores or less.” How about yours? Are your codes ready to take advantage of today’s and tomorrow’s ultra-parallel HPC systems? Download this White Paper by Analysts Intersect360 Research to see what Bull and Intel’s Center for Excellence in Parallel Programming can do for your codes.
In this demonstration of SGI DMF ZeroWatt disk solution, Dr. Eng Lim Goh, SGI CTO, discusses a function of SGI DMF software to reduce costs and power consumption in an exascale (Big Data) storage datacenter.
The Cray CS300-AC cluster supercomputer offers energy efficient, air-cooled design based on modular, industry-standard platforms featuring the latest processor and network technologies and a wide range of datacenter cooling requirements.