When following any sort of process — be it the steps you take to cook a meal or the order in which you book and pack for a vacation — you want to be proactive in preventing issues down the line. Because let’s face it: Breakfast won’t be enjoyable if you crack the eggs after adding them to the skillet, and your suitcase will do you no good in Alaska if you packed for a tropical Caribbean getaway. Planning ahead in a process leads to a more logical order of operations.
The same can be said of data observability.
In the tech sphere, data is a tool that can allow companies to build smarter processes, make efficient business decisions, and create products that are intuitive and user friendly. But only by implementing observability and monitoring best practices can data-centered teams keep a system online and running properly, while also strategizing for the future.
John Pulikkottil, director of data engineering, explained how Hyatt’s tech landscape faces high traffic between guest and colleague interactions — and this drives the company’s data observability strategy. It’s with this in mind that he stressed the need to understand how important the practice is — and worse, how much it can impact a company if it’s not treated as a critical piece of the data puzzle.
Without thoughtful monitoring and refreshing, data strategies will be rendered null — alongside the systems that keep a tech business up and running. Built In Chicago dove into this idea with Pulikkottil to learn more about what his team at Hyatt is doing to streamline their data processes and overcome observability-related challenges.
What is one of the most critical best practices your team follows when it comes to data observability?
Taking care of our guests and colleagues is one of Hyatt’s utmost priorities, and this serves as our guide as we work to continually improve upon our system uptime, data freshness and data quality. These are some of the key aspects of our technology landscape that support guest and colleague interaction with our systems — be it for booking, searching on our website, sharing feedback or adding amenities.
To deliver a better user experience while using Hyatt’s systems, data observability plays a critical role in monitoring and producing alerts to keep the system up and running, while also providing accurate and relevant information in terms of data freshness and quality. The various elements driving our data observability strategy come together to form an enhanced experience and level of satisfaction among guests and colleagues alike.
How has the evolution of data observability changed the tools you use to implement it?
At a very high level, we can classify data observability into three different generations. The first generation primarily focused on monitoring and alerting aspects, where the main objective was to raise the red flag as quickly as possible when an error in the system happened. The second generation added the ability to drill down into the issues and understand the root cause of the error. The third generation added machine learning capabilities, providing the ability to cautiously predict and automatically fix issues before they occur.
We are in the early stages of implementing data observability to its fullest extent in Hyatt’s data pipelines. The foundational aspects of monitoring, alerting and building the repository are in place. Since there is no one-size-fits-all tool for application, infrastructure and data pipeline monitoring, our approach is to custom-build some aspects of it — especially around monitoring data quality and dependencies — and utilize industry-standard tools for capturing data lineage, logs and traces.
Since there is no one-size-fits-all tool for application, infrastructure and data pipeline monitoring, our approach is to custom-build some aspects.”
What’s the biggest challenge your team has faced with data observability?
Data observability for an application, infrastructure and data pipeline requires different kinds and levels of monitoring. The three typical pillars of data observability — metrics, logs and traces — are not sufficient. Additional pillars or areas of monitoring like data quality, schedule monitoring, dependencies monitoring and data lineage are essential to view a complete picture of a failure as well as come up with the remediation.
Understanding the importance of data observability and the impact of not having it is the biggest challenge. Once the importance is well understood and leadership is aligned, implementation presents the next set of challenges: understanding the “what,” “where” and “how” of monitoring, alerting and data collection. In our environment, we have crossed the awareness stage and are in the implementation stage of identifying and adopting the tools, technology and approach. It’s only a matter of time before we will be in a stage where algorithmic recommendations are a reality.
