2011; With unstoppable advances in trading technologies, including the dawn of the sub-millisecond trade, financial services organisations are having to deal with infinitely greater data volumes and complexity. Processing and making sense of these kinds of datasets or ‘big data’, as they are now known, throws up challenges which themselves are driving forward advances in technology, for example within real-time analytics and complex event processing (CEP).

The latest example of volatility in market data volumes is the recent round of market turbulence on Friday 5th August, during which Bloomberg processed 41 billion single ticks, up 33% on the Japanese earthquake.’* Managing this kind of data on a day-to-day basis is becoming increasingly difficult.

Simplifying Today’s Data
Companies such as Sybase, Progress and StreamBase are market leaders in allowing organisations in the capital markets space to better understand market data, design and implement trading strategies, manage risk and ensure profitable positions through the provision of CEP and real-time technologies. These kind of companies are changing the way institutions understand data and how it is managed on both the buy and sell side. But what is the future?

Simplifying Financial Data in 2050?
Fast forward to 2050; a world where ticks and market data are talked about in the quadrillion not billions and where data is infinitely more complex. A world where the global market is a single entity and where some commodities are more expensive than gold. A farfetch scenario? Or a simple realisation that given the current speed of technological advance, today's big data will not be that big in the future. How will we simplify and make this mass of data transparent?

Living Data
Back to 2011; Correlating, aggregating, and computing such massive data volumes may seem an insurmountable challenge but then again maybe not! At the NanoSystems Institute at the University of California a team of scientists, artist and engineers have created the AlloSphere. A metal sphere that creates interactive 3-D visualizations of data. Scientists are able to stand in a 3-storey sphere visualizing, hearing and exploring complex multi-dimensional data in true 3D.

The AlloSphere maps multiple data sources through a super computer that utilises intelligent computer agents and algorithms that grow, interact, communicate and learn like a swarm of insects to create a virtual data world that is more attune to Xmen than Banking. But maybe this is a look into the future of managing financial data.

With infinite data types and volumes comes the requirement for infinite processing power. But maybe we need to do is turn this on its head! What if individual pieces of data had small amounts of processing power. On their own they are ineffectual and cannot work alone. But as a collective, like ants, they become a self-organising network, with each piece of data knowing how to interact with other data types. In essence, they become ‘intelligent data’ and like strands of DNA work together to create a meaning full structure that can be represented simply through virtual 3D data worlds.

Conclusions
For now, there is obviously no sure way of knowing the future, but what is certain is that the practical problems of data simplification and market transparency will remain a challenge. How companies address this is of crucial importance, from choosing the right technology and implementation partner, to determining how technologies such as CEP underpin trading strategies. This requires market awareness, informed decision-making, sound investment and a willingness to embrace living data. September 2011

* Source: www.computerworlduk.com