In mid-2020, 35% of banks in the United Kingdom suddenly faced significant issues from their previously successful machine learning models. It turned out that the massive impact of the COVID-19 pandemic threw off their tools, and the models they used had failed to adapt to changes in the data.
The moral of this story is that you can’t expect the same machine learning model to work forever. Businesses are always operating in a dynamic environment, where anything from consumer demand to economic shifts affect the data we feed into ML models. That’s why model retraining will be necessary eventually.
What is model retraining, how often should you do it, and how can you deploy up-to-date machine learning models that reflect current circumstances in your organization?
It Doesn’t End at Deployment
Your work on machine learning doesn’t end after deploying a model. For instance, consider model serving. Does your organization use the model via API or other method? Answer this question before considering automated model retraining to minimize the chance of disrupting your workflow.
Then monitor the performance of the model in action. Once you start feeding it live data, check on the quality of its predictions and look for potential issues in its performance. From there, you can consider retraining your machine learning model.
How Do You Monitor Performance Continuously?
Gather information and analyze the performance of the algorithm continuously. A few aspects to consider here are:
- Changes in data quality– A machine learning algorithm relies on the quality of its input data. Data validation is part of algorithm monitoring since any changes in the distribution of the data can lead to declining performance.
- Versioning– Save all model metadata to assist in continuous improvement. Retraining efforts must include steps to log model versions so that you can compare them to their associated performance metrics.
- Security monitoring– Always keep cybersecurity best practices in mind. Machine learning models operating in high-risk industries are susceptible to malicious actors, such as credit card fraud that aims to deceive algorithms into allowing suspicious transactions.
To validate the model, start by choosing relevant performance metrics and decide on minimum allowable values. If any metrics fall below their thresholds, it could be a sign that retraining is necessary.
How Do You Retrain the Model Continuously?
ML model drift occurs when an algorithm degrades in performance over time in response to changes in the environment. In business, these changes typically involve shifts in consumer behavior or the surrounding market. The goal of retraining in ML revolves around adapting to those changes and redeploying accordingly.
The speed at which a model drifts varies depending on the business use case, though you can categorize most causes into either data or concept drift.
Data Drift
Data scientists initially train a new model with baseline data. But if the statistical distribution of the actual data diverges away from that baseline, the model will require retraining.
Massive events like the COVID-19 pandemic shifting consumer demand have caused data drifts, but even outliers in the data or other causes can make input data no longer representative of real-world circumstances.
Concept Drift
Concept drift occurs when the data doesn’t necessarily change but the model’s interpretation of it is no longer applicable.
For example, a financial model might predict that consumer spending goes up when the strength of the economy rises. However, it’s possible that the model’s input data never covered economic conditions. This hidden context causes the model to output incorrect information since the relationship between the input and output has shifted.
In essence, data drift is a shift in the distribution of the input data; concept drift is a change in how the model should be labeling that data. Both result in model decay and a need for retraining.
How Do You Retrain Machine Learning?
When it comes time to provide that machine learning model update, businesses must answer the following questions to get the job done right.
When and How Often?
Deciding on a frequency is important, as retraining can be expensive. It involves computational alongside labor costs. Data scientists, engineers, and programmers must work on model development, while internal stakeholders must observe and approve of new deployments.
The ideal frequency depends on your organization and use case. Some methods include:
- Periodic intervals– The most straightforward approach would be choosing a weekly, monthly, or annual retraining frequency, generally depending on how quickly the data changes. Keep in mind that a random guess can actually be detrimental, as it adds complexity that could reduce model quality.
- Performance metrics– If your organization already has a monitoring system to check whether important metrics fall below acceptable thresholds, you can use that system to determine whenever retraining is necessary. However, some use cases take time for performance metrics to transpire; models that determine credit, for instance, might take a few months to report on the relevant financial data.
- Data changes– Data scientists may also monitor the distribution of upstream data and determine whether the model is obsolete as a result.
It’s not uncommon to combine these strategies. A change in the data and a dip in performance will likely occur simultaneously and both point to a need to retrain.
How Much Data Should I Use?
Machine learning developers must choose a window size that dictates how much data retraining will involve. If this window is too large, it may introduce noise into the data. If it’s too narrow, the model may suffer from underfitting. Window size may also vary over time, as most environments have dynamic data.
A methodology for choosing window size is representative subsample selection. Take your original production data and analyze it, removing any data points that may have been the result of drift. This way, you end up with a sample of data that reflects the current statistical distribution of data.
What Retraining Strategy Is Ideal?
Exactly how you undergo retraining is up to you and your business use case. Retraining can involve either continual learning or transfer learning.
Continual learning applies a dataset to the algorithm to generate a new one, resulting in incremental updates as the strategy applies over time.
Transfer learning reuses an existing model as a basis for retraining a new model. This approach ensures that you won’t have to start from scratch every time and is the basis for deep learning algorithms.
How Do I Deploy Afterwards?
Once you retrain a model, don’t rush to deploy it right away. Test it out first by comparing the performances of your current and retrained models using the same data. Are the predictions improved on the new version? If so, you’re ready to deploy.