data_observability.htm

Notes on Data Observability

Source: Metaplane

❝Data observability is the degree of visibility you have into your data at any point in time."

Common data issues, such as data inconsistency, outdated or incorrect data, and data silos, can be addressed with data observability. Additionally, data observability can help businesses make more informed decisions by providing a clear picture of the context and quality of their data.

Data teams with data observability tools leverage their historical metadata to complete the following mission-critical jobs:

• Continuous data monitoring: Is the state of our data sufficient to meet the needs of external use cases and internal standards?
• Data incident management: when a data issue occurs, how do we keep track of the state of this issue, assign owners, and measure the quality and accuracy of our data over time?
• Root cause analysis: When a data quality issue occurs, what upstream dependencies caused it to happen (and how fast can the problem be resolved)?
• Impact analysis: What are the downstream consequences of a data quality issue, when one does occur? Which downstream teams, like a data analytics team or data scientists, should be notified?
• Spend monitoring: How much money are we using for compute resources, and how is it allocated across our data stack?
• Usage analytics: By whom, when, how much, and in what manner are our data assets being used by stakeholders?
• Query profiling: How can I optimize both my data assets and stakeholder queries to minimize time and cost?