AI—both operational and generative—is knocking on enterprises’ doors, forcing data managers to make new architectural choices on what it will take to support these data-hungry initiatives. Data mesh and data fabric are increasingly favored to help organizations get better control of their data, but which is the best option?
Industry leaders are divided on this question, although more tend to lean in favor of data fabric to make AI deliver as it should. Both architectural patterns are being adopted to enable a vibrant flow of data to support emerging initiatives that include AI. However, the two have distinct footprints. Data mesh is a highly decentralized, self-service architecture in which datasets are managed or controlled by business units across enterprises. Data fabric is a more centralized architecture that supports metadata designed to integrate disparate, multiple data platforms and pipelines that simplify access to these assets. bonded fabric
In many cases, both data mesh and fabric may serve to advance AI capabilities.
“Both help organizations harmonize their data for AI,” said Hillary Ashton, chief product officer for Teradata. “How companies organize and access their data building blocks can make a big difference in how reliable and trustworthy the AI systems are.”
In a world with an abundance of technology choices, both options fit into next-generation scenarios, agreed Pranabesh Sarkar, senior distinguished architect for data architecture, engineering, and governance at Altimetrik. Even data lakes need to be considered. “It depends on the overall design implementation and data governance strategy integrated into the underlying data platform,” he noted.
Still, when it comes to managing AI, the tilt appears to be toward data fabric, with its centralized resources and management capabilities. “Data fabric is a more technology-centric approach that aims to provide a unified and integrated layer of data services across an organization,” said Heath Thompson, president and GM for Quest Software.
At least half of organizations “are moving toward modernized data architectures like data fabric and data mesh in order to better realize capabilities like AI and machine learning,” observed Adam Glaser, SVP of product management at Appian. “I imagine this percentage will continue to increase significantly over the next year. We are seeing this adoption at a much higher rate among our customers: 94% of new customers are implementing data fabric solutions.”
Data fabric “is better suited to carry companies into the future and tackle pressing initiatives with AI, GenAI, and IoT more appropriately than a data mesh,” Glaser explained. The reasons are that with data mesh, “you are trading sophisticated data engineering work for sophisticated software engineering work. To implement and leverage these APIs, you need to have the right skills, the right information about how the integrations work, and the right tools for each integration. Despite the effectiveness of the data mesh architecture, only specialists can make use of it.”
Data fabric is optimal, as it involves locating data close to cloud compute and storage resources, a vital necessity for AI workloads, agreed Ryan Mallory, COO at Flexential. “Users, applications, and devices are creating large amounts of data interacting with AI applications, which then feed large language models—requiring increased bandwidth and the need to move data from multiple sources residing at the edge. These edge nodes must be able to scale and connect to distributed core locations where the data is aggregated. An interconnected fabric is needed to effectively facilitate the consumption of data upstream via cloud providers or the processing of data downstream via edge nodes.”
Data fabric “offers the flexibility to power AI at scale,” Ashton said. “It seamlessly channels vast amounts of data across diverse environments, from cloud to on-prem, providing advanced AI models with the fuel they require.”
Data fabric is also more democratic, Glaser added. “It enables any number of non-technical people on your teams to work with data modeling—not just developers. Data fabric’s single-pane-of-glass approach is virtualized and centralized so non-technical employees can use low-code tools to do data modeling work themselves, which leads to increased speed and agility.”
Still, it doesn’t mean data mesh should not also be considered for AI initiatives, as there are other architectural factors to consider in deciding between data mesh or fabric for AI. “Both offer pathways to support AI at scale, but the optimal approach hinges on an organization’s specific needs and goals,” said Ashton. “The primary challenges for delivering AI and machine learning projects at scale are rooted in architecture and data governance.”
Data mesh “acts as a potent enabler for AI at scale by democratizing data access, improving data quality, optimizing resources, and fostering a collaborative environment for AI development and deployment,” Ashton said. “Data mesh architecture empowers domain teams to own and manage their data as data products, allowing AI and data science teams to readily find and access high-quality, well-documented data relevant to their specific initiatives.”
While there are a number of technical reasons why data mesh and data fabric make data management—and therefore AI work—more scalable and accessible, there are also even more compelling business reasons that need to be considered and sold to the organization.
Up to this point, traditional data integration or data analysis challenges tend to hold up many an effort to become more data-driven. “Business and IT leaders know that data silos, database technical debt, and data that isn’t accessible in real time can create decades-old dilemmas that impede the progress of top-line organizational goals like digital transformation,” said Glaser. Architectures such as data fabric and data mesh can speed up those processes. “These approaches both seek to unify disparate, fragmented enterprise data through modern, API-driven techniques in order to better serve the demands of today’s modern, data-driven business applications,” said Glaser.
There has been “a significant shift in thinking about how data should be stored and managed to optimize business results,” said Olga Kupriyanova, principal consultant at ISG. Data mesh and data fabric are two of several efforts “to blend daily business need for data with the need for structure that lends itself to optimized security, data protection, and data governance. We’re seeing clients apply a mix of approaches to data management, doing their best to decentralize when possible.”
In addition, data mesh and fabric help pave the pathway to greater self-service. “There is a dependency on centralized IT teams to create new data products, which affects the go-to-market strategy,” said Sarkar. “Exploring data and enabling self-serving capabilities often gets restrictive. Both data fabric and data mesh are seen as options that, when implemented properly, help reduce the dependency on the IT team. They can help business teams find and work with the data and create new data products on their own.” As a result, a data mesh- or fabric-based architecture enables more rapid innovation—especially if it involves AI.
“For initiatives like AI and generative AI, data fabric and data mesh are particularly well-suited,” said Tony Mariotti, CEO of RubyHome, an analytics-based luxury real estate provider. The ability of data mesh and data fabric “to facilitate seamless data integration and flow across disparate systems makes them ideal for powering the sophisticated analyses and real-time decision making these technologies require.”
The bottom line, said Ashton, is that “successful AI adoption relies on having the right data—and that is both accessible and trustworthy. Data fabric and data mesh both offer powerful tools to achieve that end-goal. As AI becomes key to enterprise competitiveness, its success hinges on a data environment that is trustworthy, flexible, and harmonized.”
Another driving trend is the shift away from centralization in data management, said Eli M. Dow, CTO for Converge CONSUMER by Deloitte, a division of Deloitte Consulting. “Data mesh enables functional teams within an organization to manage their own data needs,” he explained. “This is a shift from the traditional approach to data management, where a centralized data team manages cross-functionally.”
“Many companies are finding success with a hybrid approach, combining the easy access of data fabric with the domain-specific ownership of data mesh,” said Ashton. “This creates a flexible and secure environment for AI initiatives while keeping data quality and trust at the forefront.”
Consider “how a finance team thinks about and uses data in a specific way that is different from a supply chain or marketing team, but how each team is still enabled with data, technology, and skills to do their own development,” Dow continued. “All of that development is made possible by a standard technical infrastructure, sound governance, and strategic organizational workflows.”
Data fabric and data mesh also may eventually start displacing data lakes as options for storing and providing access to diverse data types. According to Sarkar, “Many organizations are restructuring their data strategy towards one of these options, either as an evolution from a data lake, or as a new modern data strategy.” At this time, he acknowledged, adoption is not yet as widespread as data lakes.
organic fabric Subscribe to Big Data Quarterly E-Edition