Justin Borgman, CEO of Starburst Data, explores the emerging data mesh concept, and the architectural shift it could bring.
Data is at the heart of everything we do, and this trend is not slowing down, especially as the pandemic accelerates digital transformation and updates the way companies think about their data. According to McKinsey, the digital offerings for companies around the world has surpassed seven years of progress during the pandemic as companies quickly moved their businesses online and forced millions of people to work remotely; this increase in digital assets and processes means more data is accumulating than ever before. And data analysis plays a key role in shaping businesses.
The value of data in today’s world
The variety of data sources available today enables companies to identify trends and make predictions about their business through data analysis that would otherwise be considered impossible. It is these data-driven organizations that thrive in today’s digital world. Investments in the data analytics market are projected to reach $103 billion by 2023 as organizations demand more valuable and practical insights from their data. However, human complexities are also involved in finding the right tools and solutions to their data challenges.
Introducing the ‘Data Mesh’
Data mesh is a new and innovative approach to overcome these barriers. Like many software development teams who have moved from monolithic applications to microservice architectures, data mesh is in many ways the data platform version of microservices, aside from the fact that it has come to remove human bottlenecks as they are technical. Developed by Zhamak Dehghani, a Senior Technology Consultant at ThoughtWorks, Data Mesh recognizes that data is inherently decentralized within an organization and that, contrary to all previous data warehouse thinking, this decentralization is indeed a good thing!
Rather than having a centralized monolithic platform for all of your data, the data network views each group of human experts managing a particular set of data as a “domain” responsible for making “data products” that are then used by everyone in the in the organisation in a self-serve manner. At the core of this of this philosophy is a distributed architecture in which each domain has its own data product owner, which ultimately enables the company to achieve higher analysis speed and scalability.
The future of data management
Data mesh is the new way to manage data. With data volume growing at an annual rate of 19.2%, it is critical that organizations of all sizes rethink their strategy in order to build a data architecture that can stand the test of time. The data mesh provides a framework for companies to democratize both data access and data management by treating data as a product that is curated and managed by the domain’s own experts. If you are concerned about the scalability of the data warehouse model, a data mesh approach is worth serious consideration.
