In your quest to build the best data architecture for your organization’s current and future needs, you have many options. Thanks to the mailability of software, those options are nearly infinite. But luckily for you, certain patterns have emerged from the maw that can help you on your data path, including data fabrics and data meshes.
At first glance, the data fabric and the data mesh concepts sound quite similar. Meshes are often made from a type of fabric, after all, and they are both malleable items that can be lain atop things–in this case, your IT systems that are subject to the ever-growing data crush.
But there are fundamental differences to these two approaches, so it’s worth taking some time to learn their differences.
Data Fabric
Forrester analyst Noel Yuhanna was among the first individuals to define the data fabric back in the mid-2000s. Conceptually, a big data fabric is essentially a metadata-driven way of connecting a disparate collection of data tools that address key pain points in big data projects in a cohesive and self-service manner. Specifically, data fabric solutions deliver capabilities in the areas of data access, discovery, transformation, integration, security, governance, lineage, and orchestration. Graph is often used to link data assets and users, too.
Momentum is building behind the data fabric concept as a way to simplify access to, and management of, data in an increasingly heterogenous environment that includes transactional and operational data stores, data warehouses, data lakes, and lake houses. Organizations are building more data silos, not fewer, and with the growth of cloud computing, the problems surrounding data diversification are bigger than ever.
A data fabric consists of multiple data management layers (Image source: Eckerson Group)
With a singular data fabric overlayed virtually atop the various data repositories, an organization can bring some semblance of unified management to the disparate data sources and downstream consumers, including data stewards, data engineers, data analysts, and data scientists. But it’s important to note that the management is unified, not the actual storage, which remains distributed.
Some tools vendors, including Informatica and Talend, offer a soup-to-nuts data fabric that encompasses many of the capabilities discussed above, while others such as Ataccama and Denodo, deliver specific pieces of the data fabric. Google Cloud is also a supporter of the data fabric approach with its new Dataplex offering. Integration among the various components in a data fabric typically is handled via APIs and through the common JSON data format.
Data Mesh
While a data mesh aims to solve many of the same problems as a data fabric–namely, the difficulty of managing data in a heterogenous data environment–it tackles the problem in a fundamentally different manner. In short, while the data fabric seeks to build a single, virtual management layer atop distributed data, the data mesh encourages distributed groups of teams to manage data as they see fit, albeit with some common governance provisions.
The data mesh concept was first written down by Zhamak Dehghani, who is now the director of next tech incubation at Thoughtworks North America. Dehghani laid out many of the principles and concepts of the data mesh in her May 2019 report “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh,” which she followed up with the December 2020 report titled “Data Mesh Principles and Logical Architecture.”
The logical architecture of the data mesh approach (Source: Zhamak Dehghani)
The core principle driving the data mesh is rectifying the incongruence between the data lake and the data warehouse, as we wrote earlier this year. Whereas the first-generation data warehouse is designed to store largely structured data that’s used by data analysts for backward-looking SQL analytics, the second-generation data lake is used primarily to store largely unstructured data that by the data scientist for building predictive machine learning models. Dehghani writes about a third-generation system (Kappa) marked by real-time data flows and embrace of cloud services, but it doesn’t solve the underlying usability gap between first- and second-generation systems.
Many organizations build and maintain elaborate ETL data pipelines in an attempt to keep the data in synch. This also drives the need for “hyper-specialized data engineers” who are tasked with maintaining the byzantine system working.
The key insight that Dehghani brought to bear on the problem is that data transformation cannot be hardwired into the data by engineers, but instead should be a sort of filter that is applied on a common set of data that’s available to all users. So instead of building a complex set of ETL pipelines to move and transform data to specialized repositories where the various communities can analyze it, the data is retained in roughly its original form, and a series of domain-specific teams take ownership of that data as they shape the data into a product. Dehghani’s distributed data mesh addresses this concern with a new architecture that is marked by four primary characteristics:
- Domain-oriented decentralized data ownership and architecture;
- Data as a product;
- Self-serve data infrastructure as a platform;
- Federated computational governance.
In effect, the data mesh approach recognizes that only data lakes have the scalability to handle today’s analytics needs, but the top-down style of management that organizations have tried to impose on data lakes has been a failure. The data mesh tries to re-imagine that ownership structure in a bottoms-up manner, empowering individual teams to build the systems that meet their own needs, albeit with some cross-team governance.
Mesh Vs. Fabric
As we can see, there are similarities between the data mesh and the data fabric approach. However, there are differences that should be taken into account too.
According to Forrester’s Yuhanna, the key difference between the data mesh and the data fabric approach are in how APIs are accessed.
“A data mesh is basically an API-driven [solution] for developers, unlike [data] fabric,” Yuhanna said. “[Data fabric] is the opposite of data mesh, where you’re writing code for the APIs to interface. On the other hand, data fabric is low-code, no-code, which means that the API integration is happening inside of the fabric without actually leveraging it directly, as opposed to data mesh.”
(agsandrew/Shutterstock)
For James Serra, who is a data platform architecture lead at EY (Earnst and Young) and previously was a big data and data warehousing solution architect at Microsoft, the difference between the two approaches lies in which users are accessing them.
“A data fabric and a data mesh both provide an architecture to access data across multiple technologies and platforms, but a data fabric is technology-centric, while a data mesh focuses on organizational change,” Serra writes in a June blog post. “[A] data mesh is more about people and process than architecture, while a data fabric is an architectural approach that tackles the complexity of data and metadata in a smart way that works well together.”
You can simultaneously use a data mesh and a data fabric, and even a data hub, according to Eckerson Group analyst David Wells
“First, they are concepts, not things,” Wells writes in a recent blog post, “Data Architecture: Complex vs Complicated.” “Data hub as an architectural concept is different from data hub as a database. Second, they are components, not alternatives. It is practical for architecture to include both data fabric and data mesh. They are not mutually exclusive. Finally, they are architectural frameworks, not architectures. You don’t have architecture until the frameworks are adapted and customized to your needs, your data, your processes, and your terminology.”
Both data meshes and data fabrics have a seat at the big data table. In the search for architectural concepts and architectures to support your big data projects, it all comes down to finding what works best for your own particular needs.
Related Items:
Data Fabrics Emerge to Soothe Cloud Data Management Nightmares
The Data Mesh Emerges In Pursuit of Data Harmony
Big Data Fabrics Emerge to Ease Hadoop Pain
Applications:Artificial Intelligence, Data Mining, Enterprise Analytics
Technologies:Frameworks, Middleware
Vendors:Eckerson Group, Forrester Research, Google Cloud, Informatica, Microsoft, Talend
Tags:data fabric, data mesh, Dave Wells, James Serra, Noel Yuhanna, Zhamak Dehghani
FAQs
“Simply put, a data fabric is a data integration solution, while data mesh is an approach to data management, not a specific technology architecture or platform,” said Hudson.
What is the difference between data fabric and data virtualization? ›
Data fabric is used to simplify data discovery, governance and active metadata management. Data virtualization is used when there is a need to integrate data quickly. Data fabric should be used when an organization requires a centralized platform to access, manage and govern all data.
What is the difference between data lake and data fabric? ›
Among the main differentiators among the three data structures is that data lakes can store raw data, while data warehouses only stores processed and refined data, and data fabric connects one or more of the other structures for better connectivity.
What is the benefit of a data mesh? ›
Data mesh makes your data discoverable, widely accessible, secure, and interoperable — giving you better decision-making power and faster time to value.
What is the difference between mesh and fabric? ›
Unlike most types of fabric, which feature closely-woven textures, mesh is woven loosely, which results in thousands of tiny holes being present in each mesh garment.
What is data fabric in simple words? ›
A data fabric is an architecture and set of data services that provide consistent capabilities across a choice of endpoints spanning hybrid multicloud environments. It is a powerful architecture that standardizes data management practices and practicalities across cloud, on premises, and edge devices.
What are the benefits of data fabric? ›
Data fabrics provide organizations with a powerful tool for managing and analyzing data. By consolidating data from multiple sources onto a single platform, data fabrics make it possible to process and analyze data in real-time. In addition, data fabrics offer the ability to access data from anywhere, at any time.
What problems does data fabric solve? ›
That's where the scalable data fabric comes in for it not only helps to manage the collection, governance, integration, and sharing of data but also to solve challenges along the way. And, it prevents data from accumulating in silos.
How does a data fabric work? ›
Think of data fabric as a weave that is stretched over a large space that connects multiple locations, types, and sources of data, with methods for accessing that data. The data can be processed, managed, and stored as it moves within the data fabric.
What is the difference between data lake and data mesh? ›
Data Mesh. Data architecture specialists are familiar with these three concepts. Data Lake and Data Warehouse refer to different formats of data storage, analysis, and queries, while Data Mesh encompasses a series of concepts related to data management in a decentralized and large-scale manner.
Unlike traditional monolithic data infrastructures that handle the consumption, storage, transformation, and output of data in one central data lake, a data mesh supports distributed, domain-specific data consumers and views “data-as-a-product,” with each domain handling their own data pipelines.
What is meant by data mesh? ›
A hot topic in enterprise software, data mesh is a new approach to thinking about data based on a distributed architecture for data management. The idea is to make data more accessible and available to business users by directly connecting data owners, data producers, and data consumers.
What is data mesh example? ›
Data mesh supports many different operational and analytical use cases, across multiple domains. Here are a few examples: Customer 360 view, to support customer care in reducing average handle time, increase first contact resolution, and improve customer satisfaction.
What is mesh data structure? ›
It typically consists of an ordered array of vertices, and a list of faces. Each face is a list of the indices of the vertices that form its boundary. In this data structure, edges are implicit in the sequence of verts around the face. One of the largest advantages of this data structure is how easy it is to populate.
What are the different types of mesh fabric? ›
Mesh fabric is fabricated most commonly from stainless steel, copper, bronze, polyester (or nylon) and polypropylene. As the fibers are woven together, they create a very flexible, net-type finish that has a tremendous range of end-uses.
What is the use of mesh fabric? ›
Mesh fabrics are used to create products for activities such as sports, camping, hunting and fishing, and more. Examples of products and industries that incorporate this material include: Golf simulator/impact screens and nets. Aquaculture.
How do you implement data from fabric? ›
Data Fabric Implementation
Collect and analyze all types of metadata. Convert passive metadata to active metadata. Create and curate knowledge graphs that enrich data with semantics. Ensure a robust data integration foundation.
What is common data fabric? ›
CDF is a commercial software based data broker that provides enterprise intelligence, surveillance, and reconnaissance (ISR) data to consuming machines and applications throughout the Defense Intelligence Enterprise (DIE).
What is fabric and its types? ›
The fabric consists of a weft (when the yarn goes across the width of the fabric) and a warp (when the yarn goes down the length of the loom). There are three types of woven fabric: plain weave, satin weave and twill weave. Examples of popular woven fabrics are chiffon, crepe, denim, linen, satin and silk.
Who needs data fabric? ›
One of the ways organizations are addressing these data management challenges is by implementing a data fabric. Using a data fabric is a viable strategy to help companies overcome the barriers that previously made it hard to access data and process it in a distributed data environment.
Washing fabrics also causes pilling. Pilling is generally seen in man made fabrics like polyester and jute. A slower agitation and shorter cycle should be used along with detergents. Shrinking is the process where a fabric becomes smaller than its original size through the process of laundry.
Why is it called a data fabric? ›
Data fabric is defined as an emerging approach to handling data using a network-based architecture instead of point-to-point connections. This enables an integrated data layer (fabric) right from the data sources level to analytics, insight generation, orchestration, and application.
How would you know if the fabric is appropriate for a project? ›
Know the properties of the fabric you're working with beforehand (this will allow you to pick one that washes, wears, and drapes in the desired manner). A fabric with a high cotton content is great for beginner sewists. Knits are stretchier than woven fabrics, so keep that in mind when choosing your fabric.
How do you test quality of fabric? ›
How to Check Fabric Quality
- Different Fabrics, Different Standards.
- Fiber Weave. High-quality fabrics feature fibers that are closely and tightly woven together. ...
- Color. A good quality fabric should have a dye job to match. ...
- Thread Count. ...
- Finish. ...
- Price. ...
- Finished Product. ...
- Source.
- Textile Testing.
- Chemical Analysis (ECO Lab)
- Composite Testing. Mechanical Testing. Heat & Flame Testing.
Fabric analysis is the study of the construction, properties, features, orientation and dip of particles within a fabric. Reasons: Identification of face and back side of the fabric. Identification of warp and weft threads.
How is a data fabric like a map? ›
Data fabric is the same in concept, but the landscape or pattern you're creating is your data itself. It creates two layers. One layer is like the quilt, giving you the big picture of your data and helping put things in context. The second is like the map, telling you where data is and how it travels.
Does data mesh replace data lake? ›
If you are looking for affordable storage space for big data, then data lake is the answer. If you need real-time insights, go for a data mesh. If you need near real-time reporting support, go for the data mesh. If you have to quickly gather data from many disconnected systems for instant processing, go for a data mesh.
How do you implement data mesh? ›
Setting up the data mesh architecture requires you to follow four primary steps:
- Treat your data as a product.
- Map the distribution of domain ownership clearly.
- Build a self-serve data infrastructure.
- Ensure federated governance.
Zhamak Dehghani, founder of the data mesh, dispels common misunderstandings around this the data mesh, an increasingly popular approach to building a distributed data architecture, and shares how some of the best teams are getting started.
A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video).
Why data lake is faster? ›
Data in Data Lakes is stored in its native format
Data can be loaded faster and accessed quicker since it does not need to go through an initial transformation process. For traditional relational databases, data would need to be processed and manipulated before being stored.
What is the main difference between a data warehouse and a data lake? ›
A data lake contains all an organization's data in a raw, unstructured form, and can store the data indefinitely — for immediate or future use. A data warehouse contains structured data that has been cleaned and processed, ready for strategic analysis based on predefined business needs.
Who invented data fabric? ›
Noel Yuhanna of Forrester Research is credited with being one of the first to quantify the idea of a data fabric architecture. Yuhanna refers to data fabric as a platform that helps organizations adopt new business processes faster.
Is data fabric a software? ›
Data fabric software is a unified data platform that enables organizations to integrate their data and data management processes. Adopting a data fabric allows for the creation of complete views of their data, helping power existing processes and applications and enabling the rapid development of new use cases.
Is Snowflake a data fabric? ›
Talend and Snowflake
Talend Data Fabric works hand in hand with cloud data warehouses like Snowflake to provide data management with real-time speed and unshakeable trust.
Is mesh considered fabric? ›
Mesh is a light woven fabric with a mesh-like look. The fabric was once designed by a textile owner who was looking for a product that is breathable and can withstand extreme temperature changes. He produced the fabric with looms in such a way that open spaces were created between the yarns.
What fabric is similar to mesh? ›
Nylon and polyester are both viable options for knitting mesh fabrics, and each is used to produce knitted mesh solutions for many different purposes. As synthetic fibers, nylon and polyester share several similar beneficial properties such as: Durability.
What fabric is mesh? ›
When it comes to mesh fabric, the material is typically made from polyester or nylon. The synthetic fibers are woven to create a flexible, net-like fabric which has a huge range of uses. Contrasting to this, mesh can also be created from metals for a sturdier and more structured material, often for industrial use.
What is called mesh? ›
A mesh is a barrier made of connected strands of metal, fiber, or other flexible or ductile materials. A mesh is similar to a web or a net in that it has many attached or woven strands.
Treat mesh like you would a very delicate or lightweight fabric when finishing, due to the holes. If you don't have an overlocker, using a double line of short straight stitch or narrow ziz-zag stitch along the seam also provides a strong finish, and can be done with a twin needle, or two rows with a single needle.
How do you make mesh fabric? ›
Mesh fabrics used in the textile industry are made through a process called knitting. A fabric mesh is created by loosely knitting yarns or fibers into a grid effect. The looseness of the knit is what makes the holes. Looser knits create larger open spaces in the fabric.
How strong is mesh fabric? ›
The standard material used in most screen enclosures is fiberglass. Polyester mesh is 100% stronger than fiberglass screens. The warp tensile strength of polyester mesh is about 112 pounds of force, the warp tear strength is about 31 pounds of force and the fill tear strength is about 27 pounds of force.
Does mesh mean see through? ›
See-through clothing is any garment of clothing made with lace, mesh or sheer fabric that allows the wearer's body or undergarments to be seen through its fabric. See-through fabrics were fashionable in Europe in the eighteenth century.
What is power mesh fabric used for? ›
Power mesh works great in uniforms, sports wear, undergarments, and many other fashion applications. It is a light, soft, breathable, yet sturdy.
How do you create a data mesh? ›
Setting up the data mesh architecture requires you to follow four primary steps:
- Treat your data as a product.
- Map the distribution of domain ownership clearly.
- Build a self-serve data infrastructure.
- Ensure federated governance.
Zhamak Dehghani, founder of the data mesh, dispels common misunderstandings around this the data mesh, an increasingly popular approach to building a distributed data architecture, and shares how some of the best teams are getting started.
What is performance mesh fabric? ›
Description. Power Mesh fabric is a stretchy, lightweight, and sheer fabric that features closely spaced 0.05 mm holes all throughout the fabric. This Power Mesh fabric is constructed from 90% nylon fibers which make the fabric very durable.
What is a fabric with an open mesh? ›
Recent Clues
We found 1 solutions for Fabric With An Open Mesh. . The most likely answer for the clue is ETAMINE.
What is the difference between mesh and polyester? ›
Polyester often has more of a fiber-like feeling, whereas nylon can seem almost more silk-like. When it comes to the stretch ability of the mesh screen, nylon is able to be stretched, more than polyester. Polyester mesh screen includes several properties, making it suited for several applications.
