The phrase “big data” has come to mean different things to different people, morphing from the huge data volumes too big for traditional databases to handle to the infrastructure required to ingest, store and analyze varieties of structured and, increasingly, unstructured data.
As this big data construct expands to include streaming and archived data along with sensor information and transactions, the financial sector continues its steady embrace of big data analytics for high-frequency trading, fraud detection and a growing list of consumer-oriented applications.
Those applications have prompted banks, insurers credit card and payment processing vendors along with other financial services companies to invest an estimated $9 billion in big data technologies, according to market survey released this week by SNS Telecom & IT. The market tracker forecasts those fintech investments in big data platforms will grow at a 17-percent annual clip over the next three years.
That works out to $14 billion investment in hardware, software and services related to big data implementations through 2021, according to the market intelligence and consulting firm based in Dubai, UAE.
As with most data-driven sectors, the market researcher found that banks and financial services vendors are embracing cloud-based analytics, with a particular emphasis on hybrid implementations, as they seek to retain control of sensitive data in internal datacenters while leveraging agile public clouds. The strategy aims to “alleviate the technical and scalability challenges associated with on-premise big data environments,” SNS said in a report summary.
That trend is being driven in part by new services such as online lending and money transfer services. Meanwhile, established players are embracing cloud-based analytics as they pull in social media and other unstructured data used for applications like credit scoring and high-frequency trading.
The big data forecast for the financial services sector is available here.
Among the examples of emerging big data platforms making headway on Wall Street is graph technology. Graph databases are finding applications in areas ranging from risk assessment to portfolio management, proponents say.
Among the early deployments of graph technology are cloud and financial technology vendors specializing in real-time intelligence tools for trading and risk management. The shift to graph technologies also is propelling new trading platforms based on standard components like the Python programming language.
This article originally appeared in sister publication Datanami.