At Quantium, VP of Data Science Isaac Konstas thinks of model features for machine learning as he would expenses within a budget.
“There’s no need to spend any more than needed unless there is a compelling return for doing so,” Konstas said.
It’s safe to say he and Kalderos Data Science Lead Patrick Boueri wouldn’t go on a shopping spree of any kind without knowing how much they have in their account.
“My advice to other data scientists: surface recurring costs by tracking how much time you devote to ‘finished’ models so that you can incorporate them into planning,” Boueri said.
In order to prevent model drift, the tech professionals make sure that all parties have agreed upon a metric definition and are able to explain the key drivers of a model in real-world terms. Otherwise, discrepancies in results during development could mask confounding variables.
Konstas warns against a few common pitfalls associated with modern ML algorithms. It can be tempting to include all available features in a model, in addition to excessively tuning hyperparameters. “Hyperparameter choice could make a good model worse,” Konstas said. “And it is unlikely to make a poor model great.”
At Quantium, his team always builds a simple model first, checking for stability during development. While many modern ML algorithms include some inherent randomness or subsampling, Konstas said too much variation can be a sign of unwelcome drift.
What steps do you take early on in the ML modeling process to prevent model drift, and why?
The best antidote for model drift is simplicity. A complex model may fit historical data, but it is more likely to overfit and not forecast future behavior well. A parsimonious model should aim to fit the data closely while disregarding noise that won’t repeat in the future.
Striking this balance is an art that can take years of experience to refine. Some common pitfalls with modern ML algorithms can be including all available features in the model, as well as excessively tuning hyperparameters. Hyperparameter choice could make a good model worse. And it is unlikely to make a poor model great. It’s good practice to ensure you have a comprehensive understanding of key hyperparameters and set reasonable values for them. Then, invest additional model iteration efforts into feature design.
The best antidote for model drift is simplicity.’’
What is your process for monitoring, detecting and correcting for model drift once a model has been deployed?
This depends on a range of factors, including the stability of the environment we are trying to model, how often the model is used and for what purpose the results are produced (marketing targeting versus regulated pricing of financial products).
If model structure is a governed process and requires significant human effort to review and deploy, favor ongoing performance monitoring on new experience as it emerges. If the model is underpinned by a relatively automated ML algorithm and is easy to refit, focus the monitoring process on ensuring a retrained model still performs comparably on the latest data and that feature importance rankings are stable.
Model monitoring tools can sometimes be as simple as a few key metrics against which to re-check performance. Other times, they surface through tools like R Shiny and Tableau, where more detailed diagnostics are required.
What advice do you have for other data scientists who are looking to better manage model drift?
Always build a simple model first. As complexity is added, compare performance to the simpler, naive model and assess if the added complexity is warranted. Think of model features as a budget. There’s no need to spend any more than needed unless there is a compelling return for doing so.
It is also important to be able to explain the key drivers of a model in real-world terms. For example, if a top feature implies a behavior that is opposite of what is expected, you should have a hypothesis explaining the discrepancy in mind. Otherwise, it could be masking confounding drivers, which leads to instability down the track.
Checking data coverage and quality is also important. If any top features have missing data, particularly recently, this may result in a degradation in performance driven by behaviors learned from historical information that are no longer available for forecasting.
Lastly, be sure to check model stability during development. Many modern ML algorithms include some inherent randomness or subsampling. If running multiple iterations on different seeds produces vastly different results, it could be symptomatic of a model that will drift quickly.
If the data science team at healthtech company Kalderos needs to react to model changes, Data Science Lead Patrick Boueri said they first seek to first understand why the change is happening. But before they are able to assess the root cause, tech professionals monitor and detect model drift on a quarterly, weekly and biweekly cadence. That way, model targets are properly aligned with business goals.
What steps do you take early on in the ML modeling process to prevent model drift, and why?
The key to having a good drift strategy is knowing where you are right now. First, all parties should agree on a metric definition. Next, the team must enable anyone to measure key metrics with a self-service approach. This strategy allows for frequent checks on how the model is performing. Finally, make sure you put some thought into leading versus lagging indicators. Frequently, you will need a good leading proxy to react to important business outcomes, or lagging indicators.
At Kalderos, we ensure our current model performance is unanimous and transparent.
The key to having a good drift strategy is knowing where you are right now.’’
What is your process for monitoring, detecting and correcting for model drift once a model has been deployed?
You can monitor and detect model drift on several time horizons. For example, on a quarterly basis, Kalderos uses objectives and key results to set model targets aligned with business goals. Biweekly, we leverage internal dashboards in Tableau to keep track of product performance. Weekly, the data science team keeps track of statistical metrics, such as calibration curves, to see if something in the model materially changed.
If Kalderos does need to react to model changes, we seek to first understand why the change is happening. From a technical standpoint, we should be able to retrain an existing model, redeploy a model’s surrounding code or deploy a new version of a model seamlessly –– and be able to roll them back without issue.
Maybe most importantly, we seek to keep good documentation and runbooks so that anyone who is familiar with the model can make that change.
What advice do you have for other data scientists who are looking to better manage model drift?
Machine learning applications have unavoidable and sizable hidden costs that can cause well-performing applications to deteriorate. Because models rely on data from the outside world, they will always require some intervention when things change –– like we’re experiencing now in the midst of a pandemic.
It’s important to understand the problem type before calculating the maintenance cost of a new model. Is there another way to frame the problems in your model portfolio? My advice to other data scientists: surface the recurring costs by tracking how much time you devote to “finished” models so that you can incorporate them into planning. Make monitoring metrics easy and collaborative so you can correlate changes across your company and accept that things simply happen. Sometimes the ground moves beneath your feet!
