The drive to deliver real-time data

An image of , News, The drive to deliver real-time data

Data is the lifeblood of the business but its value lies in being able to quickly turn it into insights. It’s been described by the Economist as “to this century what oil was to the last one: a driver of growth and change” but it can often prove just as difficult to mine. The focus for many businesses now is to improve their data maturity, by looking at how they can optimise their data and use it to deliver real-time insights.

There’s a continual drive to reduce latency and time to insight through the use of practices, processes and technologies, commonly referred to as DataOps, which seeks to improve analysis. It’s a driving force that has recently seen data management turned on its head. Before, the typical approach was centralised data warehousing that saw using Extract Load and Transform (ETL) processes used to copy it from domains into a giant data lake. But the emphasis now is on decentralising data to create pools that remain in the ownership of the domain.

Deliberating Data Mesh

An example of this in action is Data Mesh. The brainchild of Zhamak Dehghani who has founded a stealth tech startup of the same name dedicated to reimagining data platforms with Data-Mesh-native technologies, Data Mesh sees each domain to handle their own data pipelines while the mesh provides a consistency in terms of syntax and standards. (It differs from Data Fabric in this respect because it also embraces a federated governance operating model). 

Data Mesh effectively enables data to be consumed as a product and has spawned the concept of Data Infrastructure-as-a-Platform. It’s been compared to the shift in software engineering to microservices because it’s marked a seachange in how big data is managed and stored, prompting large businesses to embark on data transformation projects in a bid to access their data more rapidly, sustainably and at scale. But moving to Data Mesh needs careful consideration.

Zhamak cautions that implementing Data Mesh is not just a technological undertaking but requires cultural change too within the enterprise. The organisation needs to embrace new behaviours as well as new technology to fully reap the benefits. It’s an approach that seeks to facilitate the sharing of data in each operational domain and in so doing has the power to close the gap that exists between operations and analysis. But for that to happen, practitioners, architects, technical leaders and decision makers need to be involved in its adoption, making it a sociotechnical undertaking.

Demand for data engineers 

Overseeing any data transformation project has always involved some aspect of data engineering. Previously responsible for the preparation of data for analysis, these days data engineers occupy centre stage in data transformation. From developing or selecting data products and services to their integration into existing systems and business processes, they determine what the Modern Data Stack (MDS) will look like. Consequently, Data Engineers are now in high demand but the shift in their remit means they also need to look to upskill to ensure they can effectively collaborate with development teams and meet the needs of data analysts.

Vendors are also continually pushing the boundaries of what can be achieved, with the likes of Google, SAP and Select Star to DBT and Snowflake reinventing ways in which data can be stored, accessed and analysed more efficiently. The resulting cloud-based platforms can support numerous data types and feature analysis tools that enable analysis during the ETL process, for example. 

The desire for real-time access to data has also given rise to Fast Data which sees data analysed as it is created. Fast Data sees batch processing replaced by event-streaming systems, promising instant insights. But there are, of course, other issues that need to be considered as everything becomes bigger, better, faster, more.

Security and GRC

Aside from the technological issues – such as micro-partitioning, syntax issues or discrepancies, and monitoring for any data errors during the conversion process – there’s also the need to consider security and data governance. 

Managing data in observance with data governance frameworks is a must and consideration should also be given to how this will be done in the face of changing regulatory requirements. Businesses need to be able to document how their data is used but while this used to be a laborious and time-consuming process there are now a litany of solutions out there for automating every aspect of the process. And encapsulating it all is the data strategy, which describes how the business manages its people, policies and culture around its data.

So how do you go about moving from a centralised data architecture to Data Mesh? How can you optimise your Data Mesh? How should you build the team in such a way that data engineers and analysts work together? How do you measure your data maturity and use this to steer future projects? Is Fast Data for you? And how can you ensure you continue to observe security and governance requirements, particularly in a decentralised architecture? 

Hear it straight from the experts

Finding out the answers to these questions requires access to the best minds in the business. At the Big Data LDN Conference and Exhibition, held from 21-22 September at Olympia in London, you can hear from over 200 expert speakers across 12 technical and business-oriented theatres, all of whom are focused on how to build the dynamic, data-driven enterprise.

Zhamak Dehghani will be opening the event as the keynote speaker and will deliver her session Rewire for Data Mesh: atomic steps to rewire the sociotechnical backbone of your organisation at 10am on 21 September while other experts will be sharing unique stories, peerless expertise and real-world use cases. Starburst, Snowflake, LNER, Deliveroo, Microsoft, Actian, Confluent, Dataiku and Deloitte will be taking a deep dive into topics ranging from Modern Analytics and DataOps to Data Governance and AI & MLOPS.

Big Data LDN is free to attend but you’ll need to register. To secure your place, please do sign up via the website at

An image of , News, The drive to deliver real-time data

Bill Hammond

Bill Hammond is Event Director at Reed Exhibitions and CEO and founder of Big Data LDN. He is an experienced Managing Director with a demonstrated history of working in the events services industry.

AI alignment: teaching tech human language

Daniel Langkilde • 05th February 2024

However, Embodied AI refers to robots, virtual assistants or other intelligent systems that can interact with and learn from a physical environment. In order to do this, they’re built with sensors that can gather data from their surroundings, with this they also have AI systems that help them analyse data they collect, and ultimately learn...

CARMA announces acquisition of mmi Analytics

Jason Weekes • 01st February 2024

CARMA announces acquisition of mmi Analytics, expanding expertise in Beauty, Fashion, and Lifestyle sectors The combined organisation is set to redefine the landscape of media intelligence, providing unparalleled expertise and comprehensive insights for PR professional and marketers in the exciting world of beauty, fashion and lifestyle.

Managing Private Content Exposure Risk in 2024

Tim Freestone • 31st January 2024

Managing the privacy and compliance of sensitive content communications is getting more and more difficult for businesses. Cybercriminals continue to evolve their approaches, making it harder than ever to identify, stop, and mitigate the damages of malicious attacks. But, what are the key issues for IT admins to look out for in 2024?

Revolutionizing Ground Warfare Environment with Software-Enabled Armored Vehicles

Wind River • 31st January 2024

Armoured vehicles which are purpose-built for mission-critical operations are reliant on control systems that provide deterministic behaviour to meet hard real-time requirements, deliver extreme reliability, and meet rigorous security requirements against evolving threats. Wind River® has the partners and the expertise, a proven real-time operating system (RTOS), software lifecycle management techniques, and an extensive track...

The need to prove environmental accountability

Matt Tormollen • 31st January 2024

We are currently in the midst of one of the most consequential energy transitions since records began. The increasing availability of clean electrons has motivated businesses in the UK and beyond to think green. And for good reason. Being environmentally conscious attracts customers, appeases regulators, retains staff, and can even gain handouts from government. The...

Fuelling Innovation in Aftermarket

Jim Monaghan • 31st January 2024

One section of the motor trade is benefitting from the cost-of-living crisis: with consumers keeping their cars for longer, independent repairers are in huge demand. But they are also under pressure. Older cars need more repairs. They require more replacement parts, tyres and fluids. With car owners looking for value and a fast turn-around, independents...

The return of the five-day office week

Virgin Media • 25th January 2024

Virgin Media O2 Business has today published its inaugural Annual Movers Index, revealing four in ten companies are back to the office full time, despite widespread travel delays and disruptions With 2023 cementing the cost-of-living crisis, second hand shopping and public transport use surged as Brits sought to save money Using aggregated and anonymised UK...