Machine learning crash course 중 “Production ML systems” 챕터.
developers.google.com/machine-learning/crash-course/production-ml-systems
Introduction
Static versus dynamic training
Static versus dynamic inference
Static inference
Dynamic inference
When to transform data?
Transforming data before training
Transforming data while training
Deployment testing
About the unicorn model
Test model updates with reproducible training
Test calls to machine learning API
Write integration tests for pipeline components
Validate model quality before serving
Validate model-infrastructure compatibility before serving
Monitoring pipelines
Write a data schema to validate raw data
Write unit tests to validate feature engineering
Check metrics for important data slices
Use real-world metrics
Check for training-serving skew
Check for label leakage
Monitor model age throughout pipeline
Test that model weights and outputs are numerically stable
Monitor model performance
Test the quality of live model on served data
Randomization
Questions to ask
Is each feature helpful?
Is your data source reliable?
Is your model part of a feedback loop?
What’s next?