Data Documentation Neglect: The Hidden AI Killer Lurking in Your Organization
Skip the painful guesswork in data documentation—Metaphor’s AI-powered platform keeps you compliant, collaborative, and innovative.
Pardhu Gunnam
November 2, 2024
The data governance landscape is quickly shifting, and only those willing to keep up will survive.
Eric Kavanaugh: Hello, ladies and gentlemen, and welcome back to Inside Analysis, a virtual summit today. We're excited to have a couple of presentations followed by our usual radio show. I'm your host, Eric Kavanaugh. Today, we're joined by two of the smartest people I know, Aaron Wilson from Athena Solutions, a data governance and data catalog consulting firm, and Kirit Basu from Metaphor, who has a unique approach to data catalogs. Big thanks to Metaphor for sponsoring our events around data catalogs.
Aaron Wilson: Welcome, everyone, and thank you for joining us. Today, we'll discuss the emergence and significance of modern data catalog products in data governance and metadata management. As Eric mentioned, data catalogs have evolved significantly, making them easier to use and more effective for organizations trying to manage their data.
Aaron Wilson: My name is Aaron Wilson, and I’m with Athena Solutions. We’re a data management consultancy. Today, I’ll discuss Athena’s approach to data catalogs, how they fit into an overall data governance framework, and why the modern generation of data catalog products is such a significant advance. Later, we'll hear from Kirit Basu from Metaphor, a company with innovative features in its data catalog solution.
Aaron Wilson: Data catalogs, fundamentally, are tools that start with the concept of a data dictionary or metadata repository. The new generation of products uses technologies like AI and generative AI to make data dictionaries more useful, easier to maintain, and more user-friendly.
At Athena, we use a framework to illustrate what an effective data governance strategy looks like for many organizations. This includes components like people, processes and standards, data architecture, metadata management, data quality, and security and compliance. Data catalogs functionally fall under metadata management but can enhance and integrate all these components.
Aaron Wilson: People are at the heart of an effective data governance strategy. Adoption and participation from people in the organization are crucial for success. Modern data catalog platforms bring the governance process closer to the end user, encouraging participation, which is extremely valuable.
Aaron Wilson: Historically, data dictionaries were static repositories that required massive effort to gather and maintain metadata. These tools often became outdated quickly. The question arose: what if we had a tool that could read and update the organization's actual data assets automatically? What if we could generate up-to-date graphical data lineages and make it easy for users to collaborate?
Aaron Wilson: In the last five years, various new technologies have evolved to address these challenges. Modern data catalog platforms offer features like metadata harvesting, data lineage generation, interactivity, collaboration, and AI-powered search. These advancements save time, enhance accuracy, and make the tools more accessible to users across the organization.
Aaron Wilson: With that, let me hand it over to Kirit Basu from Metaphor, who can tell you more about their innovative approach to modern data catalogs.
Kirit Basu: Thank you, Aaron. You've covered a lot of the issues we’re solving. At Metaphor, we think of our platform as a social tool for data, emphasizing that everyone in the company, whether technical or non-technical, should be able to participate in the data ecosystem.
Kirit Basu: Metaphor was founded by a team from LinkedIn who saw the limitations of existing catalogs. They built a catalog, Data Hub, but found it was mainly used by technical teams. Metaphor aims to engage everyone in the data process. Our platform addresses search and discovery, agile governance, data literacy, and technical needs like lineage and root cause analysis.
Kirit Basu: Let me show you how it works. For example, in a Slack channel, users can ask data-related questions, and our system uses AI to provide answers from the metadata. This reduces the burden on support teams and provides instant value.
We also have a web extension that provides metadata context for dashboards and other data assets, showing data lineage, quality issues, and facilitating user interaction without requiring deep technical knowledge.
Kirit Basu: Our main app interface is user-friendly, with social feed-like features highlighting important data and activities. Advanced search capabilities and automated lineage generation keep the catalog up-to-date and useful. We also have tools for creating and managing documentation with AI assistance, making it easier to maintain comprehensive data records.
Eric Kavanaugh: This is really important stuff. Metaphor’s interactivity and accessibility are key to its success. The integration of generative AI to capture and reflect insights automatically is a game-changer. Thank you, Kirit, for this insightful demonstration.
Our radio show is coming up next, so stay tuned and don’t hesitate to send in your questions.
The Metaphor Metadata Platform represents the next evolution of the Data Catalog - it combines best in class Technical Metadata (learnt from building DataHub at LinkedIn) with Behavioral and Social Metadata. It supercharges an organization’s ability to democratize data with state of the art capabilities for Data Governance, Data Literacy and Data Enablement, and provides an extremely intuitive user interface that turns even the most non-technical user into a fan of the catalog. See Metaphor in action today!
