Data Documentation Neglect: The Hidden AI Killer Lurking in Your Organization
Skip the painful guesswork in data documentation—Metaphor’s AI-powered platform keeps you compliant, collaborative, and innovative.
Pardhu Gunnam
November 2, 2024
Complex business analytics projects have become incredibly intuitive and accessible with Modern Analytics Cloud platforms like...
Complex business analytics projects have become incredibly intuitive and accessible with Modern Analytics Cloud platforms like ThoughtSpot. However, tracing back the lineage of these analytics reports to their sources, without the right tooling, tends to be challenging for data practitioners.
Understanding how changes to upstream systems will impact these reports add yet another level of complexity that often requires teams of analysts and engineers days if not weeks to decipher.
At Metaphor, we’ve built a metadata platform that automatically computes the lineage of ThoughtSpot Liveboards back to their source and allows data teams to pinpoint users all across systems who may be impacted by specific changes. Our platform also exposes several technical, behavioral, and social signals about data that help build a culture of greater trust between data and business teams by making data more consistent, reliable and generally increases data literacy across the organization.
One of the many requests analysts are often called upon to do is to trace back and verify if the data powering a dashboard is correct. A large part of understanding the veracity of the data is to understand where it came from. Metaphor gives a birds-eye view of the lineage at any point in the end-to-end pipeline while highlighting critical signals such as freshness, data quality, query volumes, governance tags, and a host of others.
Metaphor computes the underlying dependencies between objects upstream and downstream of any asset - for example, it will show the connection between the ThoughtSpot Liveboard to the ThoughtSpot worksheet and ultimately trace back to a source table within a cloud data platform that feeds it. Similarly, it will parse SQL on the warehouse and understand relevant dependencies of those tables and views tracing it all the way back to the source.
Without a catalog, analysts have to manually scour through log files, SQL statements, and a host of other artifacts to piece together a picture of the data’s lineage. With Metaphor, a single click on the ThoughtSpot Liveboard automatically exposes the lineage. What would have otherwise taken hours happens instantaneously.
Another time-consuming task analysts or any data users have to deal with is understanding the meaning of a given Liveboard or chart. While the chart descriptions on a Liveboard may be useful in many situations, when trying to get a fuller understanding of the data used to create that chart, it's important to be able to see documentation from a whole variety of systems.
An analyst might typically log into Snowflake and see table or column descriptions, she might look at the dbt Github project to understand any content documented there and depending on how well a company organizes its data perhaps go to a wiki or document on a shared drive to read up on the data. No matter how complete or incomplete this documentation is, she will often spend a lot of time looking through Slack or Teams conversations to understand how others are using or evolving the data.
This rather involved detective work required to understand how a Liveboard has come together is completely unnecessary with Metaphor. Metaphor brings together documentation from all authoritative source systems; it marries technical docs with business-facing docs and puts it squarely in the context of actual tables, Liveboards, or other data elements. In addition, it also indexes all contextual discussions from Slack or Teams giving a 360 view of the organization’s collective knowledge about the data.
With Metaphor, analysts are now empowered to answer questions about a dashboard or chart with a far higher degree of confidence because at the click of a button they can get all knowledge about the data that powers it in one place.
Perhaps one of the most painful parts of an analyst’s journey is to make changes to tables or views. The larger the organization is, or the more upstream a table is, the harder it is to truly understand what, who and how that change will affect the business. So a rather simple request like “Can I use 9-digit zip codes instead of 5-digit zip codes to generate hyperlocal reports” can become an incredibly arduous task. The source data warehouse used to generate ThoughtSpot data sets may only contain 5-digit codes, while potentially hundreds if not thousands of tables or dashboards using that source table would need to be updated to use the new zip code format.
A failure to understand the true impact of that change to technical systems, business processes and, invariably, people, could have a disastrous effect on the business. Without a modern data catalog like Metaphor this Impact Analysis could easily take a team of data analysts days if not weeks; with Metaphor, at a single click analysts can see technical, business and social impacts of changes instantaneously.
Metaphor is the power tool for the analysts building and supporting a company’s journey to self-service analytics with ThoughtSpot. It provides a single pane of glass view to all things data and allows analysts to answer important questions and make changes to the data stack with the maximum amount of insight and context.
The Metaphor Metadata Platform represents the next evolution of the Data Catalog - it combines best in class Technical Metadata (learnt from building DataHub at LinkedIn) with Behavioral and Social Metadata. It supercharges an organization’s ability to democratize data with state of the art capabilities for Data Governance, Data Literacy and Data Enablement, and provides an extremely intuitive user interface that turns even the most non-technical user into a fan of the catalog. See Metaphor in action today!
