Metaphor Introduces User-Aware Impact Analysis for Data

Data or Analytics Engineers charged with making a change to a table or dbt Model often operate in the blind when it comes to the exact impact a change has across the larger organization. It can easily take companies days, if not weeks, to do the due diligence required to ascertain impact. Many times even after the best due-diligence process, changing a table ends up in frenzied damage control and rollback procedures. All because the team didn’t have the tools to truly and accurately gauge the impact on critical dashboards or a particular group of users far downstream of the table they were changing.

Metaphor's new User Aware Impact Analysis capability automatically surfaces all relevant technical details in its visual lineage graph, but also includes key social indicators of users that have interacted with the data, or have mentioned it in documents or conversations.

This added dimension of social analysis easily surfaces to data engineers, far more insidious problems such as one-off but critically important business dependencies that a mere technical analysis would not have revealed.

‍

“Anyone in the data space, especially Analytics engineers, knows the importance of data lineage to help understand the data changes as part of the impact and root cause analysis. But is this enough for building a successful data platform?” says Sunny Pachunuri, Head of Data Engineering at Endpoint Closing. “We always forget that a social collaboration between data providers and consumers is vital for successfully adopting a data platform.

‍Metaphor helps Endpoint's vision of making data accessible to everyone at the company by providing a platform where users can access the data assets and collaborate using social features like follow, knowledge cards, and share. This collaboration helps the Engineering team to understand the impact analysis from a user awareness standpoint to make meaningful decisions when improving the data products.”

‍

A company without access to a visual lineage tool might typically go through the following steps:

• Examine the query history of a table to see which users or systems access it.
• Use the inventory of system access users to infer which other tables/views or Dashboards are downstream of it.
• Reach out to the owners of those downstream systems to give them a heads up on the upcoming change.
• Send an email blast or a slack message to a relatively high traffic slack channel to ‘hopefully’ inform a larger number of people about the change. Hopefully, is an operative word here since many users suffer from notification fatigue and usually ignore the messages.
• Finally, once the data team has sent enough notifications and thinks they have the best possible visibility into dependencies that their current methods offer- they will make the change.

Unfortunately the process doesn’t stop there, once changes are made it is very likely that some individuals did not see or understand the repercussions of the change notification. And a dashboard they own – that’s run once every quarter but is hugely critical to the business – fails. All of a sudden there is a frenzied effort to try and debug the problem and hopefully roll back the changes before going through the motions again.

Unfortunately, the process doesn’t stop there, once changes are made it is very likely that some individuals did not see the change notification and a dashboard they own – that’s run once every quarter but is hugely critical to the business – suddenly fails. All of a sudden there is a frenzied effort to roll back the changes and go through the motions again.

The new Impact analysis feature in Metaphor enables customers to view automatically computed lineage of their data.

‍

They can then choose to examine every node in the graph to understand technical, behavioral and social signals of what, who and how users use each of those resources.

‍

They can also choose to selectively reach out to individuals potentially impacted at each node with personalized messages so as to get their attention and hence not miss out on actions to take before the change is made.

This added capability of bringing in Social and Behavioral signals to Impact Analysis greatly improves the efficiency of Data Teams. Users who previously got mass change notification emails would generally not have too much context about how changes would actually impact them. With Metaphor’s new capability Data Teams can more effectively communicate and provide personalized context to users “You are receiving this message because you used this table or followed this resource”. What used to take many Data/Analytics Engineers days if not weeks to do, now can be performed in hours if not minutes with a far higher degree of confidence by a far smaller number of people.

‍

About Metaphor

The Metaphor Metadata Platform represents the next evolution of the Data Catalog - it combines best in class Technical Metadata (learnt from building DataHub at LinkedIn) with Behavioral and Social Metadata. It supercharges an organization’s ability to democratize data with state of the art capabilities for Data Governance, Data Literacy and Data Enablement, and provides an extremely intuitive user interface that turns even the most non-technical user into a fan of the catalog. See Metaphor in action today!