You’ve got a machine learning model trained and running in production, but that’s only half of the battle. Are you certain that it is still serving the predictions that you tested? Are the inputs within the range of tolerance that you designed? Monitoring machine learning products is an essential step of the story so that you know when it needs to be retrained against new data, or parameters need to be adjusted.
In this episode Emeli Dral shares the work that she and her team at Evidently are doing to build an open source system for tracking and alerting on the health of your ML products in production. She discusses the ways that model drift can occur, the types of metrics that you need to track, and what to do when the health of your system is suffering. This is an important and complex aspect of the machine learning lifecycle, so give it a listen and then try out Evidently for your own projects.
Interview
- Introductions
- How did you get introduced to Python?
- Can you describe what Evidently is and the story behind it?
- What are the metrics that are useful for determining the performance and health of a machine learning model?
- What are the questions that you are trying to answer with those metrics?
- How does monitoring of machine learning models compare to monitoring of infrastructure or “traditional” software projects?
- What are the failure modes for a model?
- Can you describe the design and implementation of Evidently?
- How has the architecture changed or evolved since you started working on it?
- What categories of model is Evidently designed to work with?
- What are some strategies for making models conducive to monitoring?
- What is involved in monitoring a model on a continuous basis?
- What are some considerations when establishing useful thresholds for metrics to alert on?
- Once an alert has been triggered what is the process for resolving it?
- If the training process takes a long time, how can you mitigate the impact of a model failure until the new/updated version is deployed?
- What are the most interesting, innovative, or unexpected ways that you have seen Evidently used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while working on Evidently?
- When is Evidently the wrong choice?
- What do you have planned for the future of Evidently?
This article has been published from the source link without modifications to the text. Only the headline has been changed.
