Enabling and governing streaming data

Enterprise IT News talks with Richard Koh, Area VP in Asia at Confluent, about streaming data and how it is governed and enabled for organisations.

EITN: What is your definition of streaming data?

Richard: Streaming data can be described as a process where the continuous flow of data, generated from various sources, is processed, stored and analysed instantaneously. In the past, we used the term ‘data lakes’ to refer to a source of data. Streaming data is more like a river – data is constantly set in motion, serving as an immediate link between all of an organisation’s footprints.

It is collected and delivered simultaneously, creating a never-ending feed of data that can be used without the need to download it beforehand. Whether it is information stored in data centres or being utilised in mainframe systems, data across all platforms is processed in real-time, flowing continuously, seamlessly and securely in cloud architectures that are elastically scalable.

EITN: What are the top three ways that organisations are accessing streaming data and what do they use it for?

Richard: Our most recent State of Data in Motion report shows that the top three benefits for companies in Singapore are the improvement of operational efficiency (52%), gaining higher customer engagement (51%), and generating more accurate customer insights (46%).

Organisations usually gather data from various sources, ranging from both internal and external systems, IoT servers, applications and the list goes on. To specify and unify streaming data, companies can make use of these three methods: event streaming, real-time applications and database streaming.

The first method, event streaming, pertains to an organisation accessing and analysing data specific to an event, right when the event is taking place. Event streaming allows for the use of smart applications to respond to the specific event based on the data collected.

Real–time applications can help organisations become more agile, connecting and using data in new ways. It also enables more powerful customer experiences across devices, platforms and channels, helping to deliver more accurate customer insights, higher retention and higher revenue overall.

Streaming data mainly involves two components – storage and processing. Data storage should be optimised to ensure that large streams of data can be organised in a manner that is sequential and consistent.

In a traditional database, data is integrated into storage and nothing happens. But as the world changes and customers expect always-on data, streaming databases become an important part of the system. When data flows into a streaming database, it is immediately processed and used to update application results in real-time.

EITN: What kind of environments (infrastructure and software and right technology) can streaming data be used in?

Richard: Real-time data streaming can be deployed in on-premise,  hybrid, or multi-cloud environments. When it comes to a hybrid cloud environment, companies have data stored both on-premise and in the cloud, while multi-cloud data streaming involves the synchronisation of all data across various cloud storages.

Streaming data mainly involves two components – storage and processing. Data storage should be optimised to ensure that large streams of data can be organised in a manner that is sequential and consistent.

Cloud infrastructure used should also be scalable on demand so that companies are not limited by their data storage space as they grow. Across both hybrid or multi-cloud environments, processing software must be able to interact, analyse and compute available data. Software, incorporating a rich ecosystem of connectors and state-of-the-art stream processing capabilities, operates as a central nervous system that helps seamlessly integrate data across silos regardless which environment streaming data is used in.

EITN: What is your definition of right technology?

Richard: There are often many ways in which technology can be used. A right solution is one in which different tools and systems are successfully adopted to achieve a company’s digital strategy, delivering relevant business value that allows the organisation to reach their goals. The use of technology can be considered effective when it enables efficiency, allowing organisations free up time and resources to spend on business plans, instead of worrying about infrastructure management.

From a data standpoint, there is one important component that should, on multiple fronts, be part of the technology implemented by an organisation for it to work ‘right’ – data governance tools.

This is to ensure that the availability, integrity and security of data employed is well managed across all systems and infrastructure. At present, 80% of companies in Singapore say that their data integration initiatives are impeded due to their organisation’s lack of data governance tools implemented for real-time data streams according to our latest Data in Motion report. This is significantly higher as compared to the global percentage of 63%.

With more than 70% of those surveyed planning to adopt such tools within this and next year, it is clear that data governance technology is essential for companies to innovate autonomously and uncover critical insights. Data governance tools are able to provide companies with the trust and confidence that their data is reliable – a necessity for data streams to be put to effective use.

From a data standpoint, there is one important component that should, on multiple fronts, be part of the technology implemented by an organisation for it to work ‘right’ – data governance tools.

EITN: How does Confluent do it differently compared to its competitors?

Richard: Confluent has pioneered a new category of data capture that we call data in motion, that is powered by open source technology, Apache Kafka®. Kafka has fast become one of the most successful open source projects globally, and is estimated to be used by over 70% of the Fortune 500.

With data in motion, Confluent provides modern customer experiences that relies on the movement of data. Objects at rest stay at rest, but objects in motion keep on moving, sparking action and, in turn, reaction.

In a world where everything is constantly moving, if an organisation’s data stays still then its customer experiences, goals and business will too.

Data in motion triggers a reaction across organisations, allowing them to respond and make decisions in real-time. That is the Confluent difference.

EITN: How important is cloud as an enabler for real-time data streaming?

Richard: Cloud architecture can be said to be the backbone of real-time data streaming. The use of cloud solutions enables real-time interoperability between all systems, applications and data stores that make up an organisation’s technological framework. Due to the virtually infinite scalability of cloud, this means that companies are able to sync every single component of their infrastructure.

In a world where everything is constantly moving, if an organisation’s data stays still then its customer experiences, goals and business will too.

In addition, the full integration of data across all systems means that organisations are able to get a detailed and complete overview of insights. Having all data taken into account is essential for data streaming in order to ensure that insights generated are reliable. Using cloud to govern systems also ensures higher security and compliance measures, ensuring that data collected is precise, reliable and trustworthy. Data is generated at a staggering rate daily, and for companies to reap the best results, insights must be delivered instantaneously. The reliance on the cloud to set data in motion is key to unlocking these insights for better business value. It is thus clear that the cloud has a paramount role to play for companies to innovate faster, increase revenues and reduce operational management in a secure manner.

EITN: What are the main challenges/ barriers to real-time streaming? How does Confluent help with this?

Richard: Organisations often face a tough challenge in synchronising their data between storage environments effectively. To achieve a state of true data in motion, real-time synchronisation between various data sources, be it in a hybrid or multi-cloud environment, is critical. For 89% of IT leaders in Singapore, this functionality is critical to achieving this year’s target revenue.

Another common complication faced is maintaining a high level of security and meeting regulatory compliance. This one in particular, can be tricky, especially with governments worldwide clamping down on privacy policies and how data should be collected, stored and used.

When dealing with data, consistency and access to it will always present a challenge as the data read at any given time could already be modified and stale once it gets to its destination. When working with data streams on the cloud, data durability also remains an issue.

Another common complication faced is maintaining a high level of security and meeting regulatory compliance. This one in particular, can be tricky, especially with governments worldwide clamping down on privacy policies and how data should be collected, stored and used.

To help organisations manage these challenges, Confluent offers a fully managed governance suite that enables companies to connect every single one of their data streams, from applications to systems and various parts of the business. Having a rich ecosystem of connectors, full synchronisation is enabled through Confluent’s cloud services. The integration provided allows the formation of a real-time central nervous system that ensures every bit of data is synchronised and easily accessible for use.