Imagine you’re managing a department that handles account openings in a bank. All services seem fine, and the infrastructure seems to be working smoothly. But one day, it becomes clear that no new account has been opened in the last 24 hours. On investigation, you find that this is because one of the microservices involved in the account opening process is taking a very long time to respond.
For such a case, the data analyst examining the problem can use traces with triggers based on processing time. But there must be an easier way to spot anomalies.
Traditional monitoring involves recording the performance of the infrastructure and applications. Data observability allows you to track your data flows and find faults in them (may even extend to business processes). While traditional tools analyze infrastructure and applications using metrics, logs, and traces, data observability uses data analysis in a broader sense.
So, how do we tackle the case of no new account creation in 24 hours?
The data analyst could use traces with time-based triggers. There has to be an easier way of detecting potential anomalies on site.
A machine learning model is used to predict future events, such as the volume of future sales, by utilizing regularly updated historical data. However, because the input data may not always be of perfect quality, the model can sometimes produce inaccurate forecasts. These inaccuracies can lead to either excess inventory for the retailer or, worse, out-of-stock situations when there is consumer demand.
Classifying and Addressing Unplanned Events
The point of Data Observability is to identify so-called data downtime. Data Downtime refers to a sudden unplanned event in your business/infrastructure/code that leads to a sudden change in the data. In other words, it is the process of finding anomalies in data.
How can you classify these events?
- Exceeding a given metric value or an abnormal jump in a given metric. This type is the simplest. Imagine that you add 80-120 clients every day (confidence interval with some probability), and in one day, only 20. Perhaps something caused it to drop suddenly, and it’s worth looking into.
- Abrupt change in data structure. Let’s take a past example with clients. Everything was fine, but one day, the contact information field began to receive empty values. Perhaps something has broken in your data pipeline, and it’s better to check.
- The occurrence of a certain condition or deviation from it. Just as GPS coordinates should not show a truck in the ocean, banking transactions should not suddenly appear in unexpected locations or in unusual amounts that deviate significantly from the norm.
- Statistical anomalies. During a routine check, the bank’s analysts notice that on a particular day, the average ATM withdrawal per customer spiked to $500, which is significantly higher than the historical average.
On the one hand, it seems that there is nothing new in this approach of classifying abnormal events and taking the necessary remedial action. But on the other hand, previously there were no comprehensive and specialized tools for these tasks.
Data Observability is Essential for Ensuring Fresh, Accurate, and Smooth Data Flow
Data observability serves as a checkup for your systems. It lets you ensure your data is fresh, accurate, and flowing smoothly, helping you catch potential problems early on.
| Persona | Why Question | Observability Use case | Business Outcome |
|---|---|---|---|
| Business User |
|
Data Quality, Anomaly Detection and RCA |
|
| Data Engineers/Data Reliability Engineers |
|
Data Pipeline Observability, Troubleshooting and RCA |
|
| Data Scientists |
|
Data Quality Model |
|
Tiger Analytics’ Continuous Observability Solution
Continuous monitoring and alerting of potential issues (gathered from various sources) before a customer/operations reports an issue. Consists of Set of tools, patterns and practices to build Data Observability components for your big data workloads in Cloud platform to reduce DATA DOWNTIME.
Select examples of our experience in Data observability and Quality
Tools and Technology
Tiger Analytics Data Observability is set of tools, patterns and best practices to:
- Ingest MELT(Metrics, Events, Logs, Traces) data
- Enrich, Store MELT for getting insights on Event & Log Correlations, Data Anomalies, Pipeline Failures, Performance Metrics
- Configure Data Quality rules using a Self Service UI
- Monitor Operational Metrics like Data quality, Pipeline health, SLAs
- Alert Business team when there is Data Downtime
- Perform Root cause analysis
- Fix broken pipelines and data quality issues
Which will help:
- Minimize data downtime using automated data quality checks
- Discover data problems before they impact the business KPIs
- Accelerate Troubleshooting and Root Cause Analysis
- Boost productivity and reduce operational cost
- Improve Operational Excellence, QoS, Uptime
Data observability and Generative AI (GenAI) can play crucial roles in enhancing data-driven decision-making and machine learning (ML) model performance.
The combination of data observability primes the pump by instilling confidence with smooth sailing, high-quality and always available data which forms a foundation for any data-driven initiative while GenAI enables to realize what is achievable through it, opening up new avenues into how we can simulate, generate or even go beyond innovate. Organizations can use both to improve their data capabilities, decision-making processes, and innovation with different areas.
Thus, Monte Carlo, a company that produces a tool for data monitoring, raised $135 million, Observe – $112 million, Acceldata – $100 million have an excellent technology medium in the Data Observability space.
To summarize
Data Observability is an approach to identifying anomalies in business processes and the operation of applications and infrastructure, allowing users to quickly respond to emerging incidents.It lets you ensure your data is fresh, accurate, and flowing smoothly, helping you catch potential problems early on.
And if there is no particular novelty in technology, there is certainly novelty in the approach, tools and new terms that make it possible to better convince investors and clients. The next few years will show how successful new players will be in the market.
References
https://www.oreilly.com/library/view/data-observability-for/9781804616024/
https://www.oreilly.com/library/view/data-quality-fundamentals/9781098112035/
