Japan EN

Expertise

Cloud & DevOps, Technical Consulting, Software Engineering

Platforms

Amazon Web Services

Highlights

  • Enabled detection of Data/Model Bias 
  • Generated feature importance reporting 
  • Facilitated central management of model Quality/Bias checks  
  • Monitoring for comparison with org thresholds   
  • Automated report of drift detection    

Overview

In a large financial services institution, we built and deployed a centralized, automated model deployment capability from on-prem to the Cloud. We enabled a single click deployment for the entire Cloud resources using Infrastructure as Code.  

We automated the end-to-end system, enabled scheduled monitoring to detect drifts between different Machine Learning models, and provided notifications to stakeholders for remediation actions. We encrypted sensitive information at all levels of data communications from on-prem to Cloud and reverse. The solution enabled a highly available, scalable & secure system.

Our Process

Using sophisticated tools, we built, integrated, and deployed an end-to-end Machine Learning Model Monitoring from a Model Inference system.

Needs assessment

We worked with the client to understand their needs and set objectives/parameters to automate monitoring and detection of Data/Model Bias as well as drift, and generate feature importance reports for Data Scientists and stakeholders for management/governance. 

Choosing Appropriate Tools

Based on the client’s specific needs, we chose best-in-class, fit-for-purpose tools across all accounts and enterprise requirements. These included tools for sensitive data encryption between on-prem to Cloud and vice-versa, and tools for monitoring/reporting.   

Build & Deploy

We enabled the single click end-to-end deployment via automated pipelines, utilized Cloud native services for bias detection, and generated explanations for compliance and legal requirements.   

Cloud & Engineering

Cloud project scope

  • Detect model/data bias metrics 
  • Notify stakeholders continuously for any drift in model metrics 
  • Encrypt sensitive information while data is in motion  
  • Generate reports that are compliant with the legal requirements   

Data flow/storage

  • Data Science teams pre-process the data and place it on the cloud object storage  
  • All models & data are always encrypted while at rest 
  • ML services fetch the latest data from object storage, process it, and place the results back on the object storage   

Tools Deployed

  • SageMaker Pipelines & Clarify 
  • SageMaker Model Monitor & Registry 
  • Amazon Lamda & CloudWatch 
  • Jenkins (on-prem) 
  • Python 

We successfully built and deployed a fully automated Model Monitoring from the Model Inference platform to monitor Data/Model Bias and departures from organizational thresholds, measure drift and allow for reporting and corrective actions to be taken. The platform also uses SHAP values for feature importance reporting and offers high availability, scalability, and a secure design.