Organizations use a few different models to manage access to data on, the most dynamic of which is the data fabric. To understand the unique advantages this approach offers, it’s helpful to understand how it fundamentally differs from one of the more common attempts to gain control over the unwieldy volume of data that organizations are currently collecting: the data catalog.
The data catalog emerged partly in response to limitations that were becoming increasingly problematic for data warehouses, which attempted to collect all the data that’s relevant to the organization in a single location, and then organize it so that it’s easy for analysts to find. The problem with this approach was that it required too much structure. It was kind of like a public library, and therefore required the attention to detail and process that one looks for in a librarian. When companies started grappling with Big Data, the idea of having to meticulously curate each data set before making it available in a central repository was slow–data was piling up.
The data catalog took a slightly different approach. Instead of collecting and neatly curating data, it attempted to overlay some structure on the data architecture that already existed within the organization by offering an automated means to apply metadata, or descriptive data, to the data assets. This metadata comes in two forms–technical metadata, like schemas and column names, and business metadata, such as the purpose for the dataset’s existence and its history of usage.
If a data warehouse is like a library, a data catalog is more like the search engine on a library website–it allows more flexible search, and also greatly expands what you can access so that you’re not just limited to information that’s been collected and curated. A data fabric takes it a step further, and acts more like the internet, which is not an actual collection of data, but rather a virtual access layer for info that might be located anywhere in the world. This distinction is important, as many organizations now have data that’s collected in siloed repositories scattered all over the world.
Let’s discuss the transition between data catalog and data fabric by exploring some of the similarities and differences between the two approaches.
How data catalogs and data fabrics are similar
Data catalogs and data fabrics have a few things in common, both conceptually and functionally. For example, both of these systems:
How data catalogs and data fabrics are different
While both of these systems leverage the capturing and applying of descriptive metadata to organize data assets and make them retrievable, they also differ in fundamental ways:
In sum, the data fabric can be viewed as the next step in the evolution of a data catalog. While there is admittedly some overlap in the continuum between data catalogs and data fabrics, you can differentiate between the two by looking at how they apply AI and NLP to allow more advanced classification of data, and also more advanced and user-friendly search features. If the system is limited to only certain data platforms, uses keyword search, is limited in terms of its ability to apply metadata to only what the schema and users tell it about a data asset, and requires a degree of ETL, it’s a data catalog.
On the other hand, if it applies AI and NLP to apply metadata based on user behavior and intent, if it allows virtualized access to all data no matter where it resides, uses semantic search, and doesn’t ever require data to be moved, it’s a data fabric.
If you’d like to learn more about how a data fabric can benefit your organization, read on.
SHARE THIS:
