Data and model quality monitoring with Amazon SageMaker Model Monitor - Amazon SageMaker AI<br>View a markdown version of this page
Data and model quality monitoring with Amazon SageMaker Model Monitor - Amazon SageMaker AI<br>DocumentationAmazon SageMakerDeveloper Guide<br>How It Works<br>Data and model quality monitoring with Amazon SageMaker Model Monitor
Note
After careful consideration, we have made the decision to close new customer access to Amazon Sagemaker Model Monitor, effective 7/30/26.<br>Existing customers can continue to use the service as normal. AWS continues to invest in security and availability improvements for<br>Model Monitor, but we do not plan to introduce new features. For more information, see Amazon SageMaker Model Monitor availability change.
Amazon SageMaker Model Monitor monitors the quality of Amazon SageMaker AI machine learning models in production. With Model Monitor,<br>you can set up:
Continuous monitoring with a real-time endpoint.
Continuous monitoring with a batch transform job that runs regularly.
On-schedule monitoring for asynchronous batch transform jobs.
With Model Monitor, you can set alerts that notify you when there are deviations in the model<br>quality. Early and proactive detection of these deviations lets you to take corrective<br>actions. You can take actions like retraining models, auditing upstream systems, or fixing<br>quality issues without having to monitor models manually or build additional tooling. You<br>can use Model Monitor prebuilt monitoring capabilities that do not require coding. You also have the<br>flexibility to monitor models by coding to provide custom analysis.<br>Model Monitor provides the following types of monitoring:
Data quality - Monitor drift in data<br>quality.
Model quality - Monitor drift in model quality<br>metrics, such as accuracy.
Bias drift for models in production - Monitor bias in your<br>model's predictions.
Feature attribution drift for models in production - Monitor<br>drift in feature attribution.
Topics<br>Amazon SageMaker Model Monitor availability change
Monitoring a Model in Production
How Amazon SageMaker Model Monitor works
Data capture
Data quality
Model quality
Bias drift for models in production
Feature attribution drift for models in production
Schedule monitoring jobs
Amazon SageMaker Model Monitor prebuilt container
Interpret results
Visualize results for real-time endpoints in Amazon SageMaker Studio
Advanced topics
Model Monitor FAQs
How Amazon SageMaker Model Monitor works
Note
After careful consideration, we have made the decision to close new customer access to Amazon Sagemaker Model Monitor, effective 7/30/26.<br>Existing customers can continue to use the service as normal. AWS continues to invest in security and availability improvements for<br>Model Monitor, but we do not plan to introduce new features. For more information, see Amazon SageMaker Model Monitor availability change.
Amazon SageMaker Model Monitor automatically monitors machine learning (ML) models in production and notifies<br>you when quality issues happen. Model Monitor uses rules to detect drift in your models and alerts<br>you when it happens. The following figure shows how this process works in the case that your<br>model is deployed to a real-time endpoint.
You can also use Model Monitor to monitor a batch transform job instead of a real-time<br>endpoint. In this case, instead of receiving requests to an endpoint and tracking the<br>predictions, Model Monitor monitors inference inputs and outputs. The following figure diagrams the<br>process of monitoring a batch transform job.
To enable model monitoring, take the following steps. These steps follow the path of<br>the data through the various data collection, monitoring, and analysis processes.
For a real-time endpoint, enable the endpoint to capture data from incoming<br>requests to a trained ML model and the resulting model predictions.
For a batch transform job, enable data capture of the batch transform inputs<br>and outputs.
Create a baseline from the dataset that was used to train the model. The<br>baseline computes metrics and suggests constraints for the metrics. Real-time or<br>batch predictions from your model are compared to the constraints. They are reported<br>as violations if they are outside the constrained values.
Create a monitoring schedule specifying what data to collect, how often to<br>collect it, how to analyze it, and which reports to produce.
Inspect the reports, which compare the latest data with the baseline. Watch<br>for any violations reported, metrics, and notifications from Amazon CloudWatch.
Notes
Model Monitor computes model metrics and statistics on tabular data only. For<br>example, an image classification model that takes images as input and<br>outputs a label based on that image can still be monitored. Model Monitor would be<br>able to calculate metrics and statistics for the output, not the<br>input.
Model Monitor currently supports only endpoints that host a single model and does<br>not support monitoring...