HPCwire: IBM Unveils Enterprise Stream Processing System

The Leading Source for Global News and Information Covering the Ecosystem of High Productivity Computing

HPCwire >> Features

June 22, 2007

IBM Unveils Enterprise Stream Processing System

by Michael Feldman, HPCwire Editor

Page: 1 of 2
1 | 2 All »

On Tuesday at the Security Industry and Financial Market Association (SIFMA) Technology Management Conference in New York, IBM announced System S, a software framework that uses a stream processing model to support a new class of applications. The result of a $5 million initiative at IBM Research, System S is designed to perform real-time analytics using high-throughput data streams.

The company will initially aim this technology at Wall Street trading applications, but the system is generally applicable to all kinds of real-time intelligence gathering. Relevant domains include surveillance, manufacturing, inventory management, public health, and biological research, among others. At this point, the System S technology is more than a prototype, but less than a product. This week's announcement is aimed at garnering interest from Wall Street firms that might want to partner with IBM to develop commercial applications.

The System S software is designed to run in a heterogeneous hardware environment, taking advantage of x86, Cell, Blue Gene, or even Power-based servers. Cell-based systems, in particular, appear to be a well-suited for these types of applications because of that processor's natural abilities as a stream computing platform. Suitable platforms can range from a single CPU up to 10,000 servers. The initial version of System S is targeted to IBM BladeCenters running Red Hat or SUSE Linux. According to IBM, in larger configurations, System S is capable of processing in the neighborhood of a million messages per second, depending on the application behavior and the nature of the data streams.

The intention of the framework is to host applications that turn heterogeneous data streams into actionable intelligence. The source of such streams could be manufacturing sensors, television broadcasts, market exchange streams, phone conversations, video feeds, email traffic, and so on. Essentially, the system works by enabling different types of software processing elements (PE) or modules to be strung together to act on data streams. The system exposes the profile of each processing element to the others in the framework so they can interoperate. The software contains an "Omniscient Scheduler" that ensures the data pipelines between the PEs operate efficiently. A user hypothesis or query drives the application and specifies the kinds of data correlations to be performed.

For example, if one were searching for a certain subject matter in conversations being conducted over a secure telephone line, this would require a number of stream processing elements. The first step would be to pass the communication feed into a data decryption PE, which would produce decrypted audio. Then, using a speech recognition PE, the audio stream would be converted into text. Next, the text data would pass through a semantic analyzer PE to identify those conversations that contained content of interest. If one was processing many such conversations, the system could automatically focus on those that met the specified criteria and drop the remainder. A more complex application with additional data feeds could be accommodated by plugging in the appropriate PEs.

According to Nagui Halim, director of high performance stream computing at IBM, System S represents a significant departure from current intelligence extraction, which traditionally relies on fixed-format data that has been stored on a disk somewhere. This model can only provide a retrospective look a problem. By contrast, System S applications are able to take unstructured raw data and process it in real time. And rather than performing simple data mining or recreating a simulation of some well-defined structure or process, System S applications attempt to make correlations and generate some type of prediction. In addition, the system is supposedly capable of refining its behavior over time by learning from the successes and failures of past correlations.

"This is about what's going to happen," explains Halim. "The thesis is that there are many signals that foreshadow what will occur if we have a system that is smart enough to pick them up and understand them. We tend to think it's impossible to predict what's going to happen; and in many cases it is. But in other cases there is a lot of antecedent information in the environment that strongly indicates what's likely to be occurring in the future."

To Halim's surprise, in his research he found that streaming data analytics was a much better tool than he expected for many classes of applications. He discovered that events are often very predictable if one examines the correct data. For example the occurrence of a "perfect storm" is the result of a number of more subtle conditions which build up over time that interact to produce a big event.

If successfully implemented, predictive systems certainly have a high value for a range of enterprises and government organizations. This is especially true in the financial services industry, where accurate forecasts of options and derivatives pricing can translate directly into profits. Being able to correlate market activity with the effects of qualitative data, like news events, would open up some interesting avenues for financial trading application. IBM envisions algorithmic trading engines connected to media feeds such as CNN and Al Jazeera to correlate news reports with financial market behavior. For example, an application could be set up to look for events that could precipitate an oil price spike in the next ten minutes.

An application could also be devised to search for rogue traders or money laundering activities. Traditionally this is accomplished by examining account histories and performing manual inspection of suspicious transactions. But this sort of retrospective analysis may allow the perpetrator to get way.

Page: 1 of 2
1 | 2 All »

Article Tools

Share Options

(Digg, Technorati, more)

Subscribe

Subscribe to HPCwire

Discussion

There are 0 discussion items posted.

Join the Discussion

You must be a registered user in order to comment on HPCwire stories.

Members enjoy:

	Access to Special Content, including Podcasts, Webcasts, and Whitepapers and more!
	Ability to take an active role in the discussions that are shaping the HPC technology discipline!
	Special offers available only to HPCwire members!

Become a Registered User Today!

Registered Users Log in Here to Comment

Email Address:	Password (case sensitive)

Remember me	Forgot Password?

RSS Feeds

Feeds by Topic

Feeds by Industry

Feeds by Content Type

Top Headlines

Australia Commissions Cray Supercomputer

Mar 19 | OfficialWire | New super to support intelligence work Down Under. Read more...

Intel Partners See 'Easy' Upgrade Path With Xeon 5600 Chips

Mar 18 | ChannelWeb | Westmere parts already showing up in HPC machines. Read more...

AMD: OEMs primed for Opteron 6100s

Mar 17 | The Register | But what about the tier ones? Read more...

Arrival of the Desktop Supercomputer

Mar 17 | Cadalyst Magazine | A new generation of workstations is changing the nature of technical computing. Read more...

Scheduling HPC In The Cloud

Mar 17 | Linux Magazine | Latest iteration of Sun Grid Engine able to tap into Cloud. Read more...

Read more headlines...

Featured Whitepapers

Virtualization for Aggregation And The vSMP Architecture™

Jan 12 | | In-depth look at vSMP Foundation server virtualization technology, technical implementation, use cases and capabilities. The technical whitepaper provides an architectural overview and details on the three vSMP Foundation products: vSMP Foundation for SMP, vSMP Foundation for Cluster and vSMP Foundation for Cloud.

Copper Cable Technologies for High Performance Computing

Jan 18 | | This white paper discusses Gore’s copper cable assemblies, and how they continue to exceed the standards for providing reliable, cost-effective solutions for high-performance computer applications.

View the White Paper Library

Multimedia

Webcast: Virtualized Data Center Roundtable

Join this online panel discussion for live Q&A with leading industry experts, analysts, and end-users to discuss the latest innovations, best practices, barriers to implementation, and measurable benefits of server virtualization with a particular focus on today's real world solutions.

Webcast: Watch SC09 Birds of a Feather Video: Scalable Fault-Tolerant HPC Supercomputers

Learn about scalable fault-tolerant architectures and examples of energy efficient and scalable supercomputing clusters using dual QDR InfiniBand to combine capacity computing with network failover capabilities with the help of programming languages such as MPI and a robust Linux cluster management package.

Webcast: High Performance Computing for a Smarter Planet

LIVE@SCO9: The IBM team discusses new innovations in hardware, software and services that help clients better understand their workloads and get insight from their R&D efforts. Technology demonstrations include the soon-to-be-released Power7 HPC processor, the DCS990 system with 2.4 petabytes of storage, the xCAT management tool, secure HPC cloud computing and more. Winners of two HPCwire Readers' and Editors’ Choice Awards! Take the IBM virtual tour at SC09 or more information go online to: http://www-03.ibm.com/systems/deepcomputing/sc09.html