Monitor Data Drift over years¶
With this tutorial you: Understand how to use Eurybia to detect datadrift
Contents: - Detect data drift
- Compile Drift over years
Requirements notice : the following tutorial may use third party modules not included in Eurybia.
You can find them all in one file on our Github repository or you can manually install those you are missing, if any.
[1]:
import pandas as pd
import numpy as np
from category_encoders import OrdinalEncoder
from lightgbm import LGBMRegressor
from eurybia import SmartDrift
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_log_error
Import Dataset and split in training and production dataset¶
[2]:
from eurybia.data.data_loader import data_loading
[3]:
house_df, house_dict = data_loading('house_prices')
[4]:
# Let us consider that the column "YrSold" corresponds to the reference date.
#In 2006, a model was trained using data. And in 2007, we want to detect data drift on new data in production to predict
#house price
house_df_learning = house_df.loc[house_df['YrSold'] == 2006]
house_df_2007 = house_df.loc[house_df['YrSold'] == 2007]
[5]:
y_df_learning=house_df_learning['SalePrice'].to_frame()
X_df_learning=house_df_learning[house_df_learning.columns.difference(['SalePrice','YrSold'])]
y_df_2007=house_df_2007['SalePrice'].to_frame()
X_df_2007=house_df_2007[house_df_2007.columns.difference(['SalePrice','YrSold'])]
Building Supervized Model¶
[ ]:
from category_encoders import OrdinalEncoder
categorical_features = [col for col in X_df_learning.columns if X_df_learning[col].dtype == 'object']
encoder = OrdinalEncoder(
cols=categorical_features,
handle_unknown='ignore',
return_df=True).fit(X_df_learning)
X_df_learning_encoded=encoder.transform(X_df_learning)
[7]:
Xtrain, Xtest, ytrain, ytest = train_test_split(X_df_learning_encoded, y_df_learning, train_size=0.75, random_state=1)
[8]:
regressor = LGBMRegressor(n_estimators=200).fit(Xtrain,ytrain)
Use Eurybia for data drift¶
[9]:
from eurybia import SmartDrift
[10]:
SD = SmartDrift(df_current=X_df_2007,
df_baseline=X_df_learning,
deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
encoding=encoder # Optional: if deployed_model and encoder to use this model
)
[11]:
%time
SD.compile(full_validation=True, # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
date_compile_auc = '01/01/2007', # Optional: useful when computing the drift for a time that is not now
datadrift_file = "house_price_auc.csv" # Optional: name of the csv file that contains the performance history of data drift
)
CPU times: user 6 µs, sys: 2 µs, total: 8 µs
Wall time: 17.2 µs
The variable BsmtCond has mismatching unique values:
['Poor -Severe cracking, settling, or wetness'] | []
The variable Condition2 has mismatching unique values:
['Near positive off-site feature--park, greenbelt, etc.', 'Adjacent to North-South Railroad', 'Adjacent to East-West Railroad'] | ['Adjacent to feeder street']
The variable Electrical has mismatching unique values:
['Mixed'] | ['60 AMP Fuse Box and mostly knob & tube wiring (poor)']
The variable ExterQual has mismatching unique values:
['Fair'] | []
The variable Exterior1st has mismatching unique values:
[] | ['Stone', 'Imitation Stucco']
The variable Exterior2nd has mismatching unique values:
['Asphalt Shingles', 'Brick Common'] | ['Other']
The variable Foundation has mismatching unique values:
[] | ['Stone', 'Wood']
The variable Functional has mismatching unique values:
['Major Deductions 2', 'Severely Damaged'] | ['Moderate Deductions']
The variable GarageQual has mismatching unique values:
[] | ['Excellent']
The variable Heating has mismatching unique values:
[] | ['Wall furnace']
The variable HeatingQC has mismatching unique values:
['Poor'] | []
The variable LotConfig has mismatching unique values:
[] | ['Frontage on 3 sides of property']
The variable MSSubClass has mismatching unique values:
['1-Story w/Finished Attic All Ages'] | []
The variable Neighborhood has mismatching unique values:
['Northpark Villa'] | []
The variable RoofMatl has mismatching unique values:
['Roll'] | ['Metal']
The variable RoofStyle has mismatching unique values:
['Mansard', 'Shed'] | []
The variable SaleType has mismatching unique values:
['Warranty Deed - Cash'] | ['Contract Low Interest', 'Contract Low Down', 'Contract Low Down payment and low interest']
The variable Street has mismatching unique values:
['Gravel'] | []
The computed AUC on the X_test used to build datadrift_classifier is equal to: 0.626082251082251
house_price_auc.csv did not exist and is created.
As soon as compile() method, Eurybia displays default consistency checks as warnings. If some modalities are not present during training and are in production dataset, the deployed model will consider them wrongly. Inversely, if some modalities are present during training and are not in production dataset, it means that some profiles are missing.
[12]:
SD.generate_report(
output_file='report_house_price_datadrift_2007.html',
title_story="Data drift",
title_description="""House price Data drift 2007""", # Optional: add a subtitle to describe report
project_info_file="../../eurybia/data/project_info_house_price.yml" # Optional: add information on report
)
Report saved to ./report_house_price_datadrift_2007.html. To upload and share your report, create a free Datapane account by running !datapane signup.
Eurybia is designed to generate an HTML report for analysis, and less for use in notebook mode. However, to illustrate functionalities, we will detail results with notebook mode analysis.
First Analysis of results of the data drift¶
Data driftn methodology is based on the ability of a model to discriminate whether an individual belongs to one of the two datasets. For this purpose a target 0 is assigned to the baseline dataset and a target 1 to the current dataset. Then a classification model (catboost) is learned to predict this target. The level of capacity of the data drift classifier to detect if an individual belongs to one of the 2 datasets represents the level of difference between the 2 datasets
Detection data drift performance¶
[13]:
#Performance of datadrift classifier
SD.plot.generate_indicator(fig_value=SD.auc, height=300, width=500, title="Datadrift classifier AUC")
An Auc close to 0.5 means that there is little drift
Importance of features in data drift¶
This graph represents the variables in the data drift classification model that are most important to differentiate between the two datasets.
[14]:
SD.xpl.plot.features_importance()
We get the features with most gaps, those that are most important to analyse. With date bias introduced, it is normal that date features are the most impacted. We will then decide to remove them. Let’s analyse other important variables
Feature importance overview¶
This graph compares the importance of variables between the data drift classifier model and the deployed model. This allows us to put into perspective the importance of data drift in relation to the impacts to be expected on the deployed model. If the variable is at the top left, it means that the variable is very important for data drift classification, but that the variable has little influence on the deployed model. If the variable is at the bottom right, it means that the variable has little importance for data drift classification, and that the variable has a lot of influence on the deployed model.
[15]:
SD.plot.scatter_feature_importance() # works if deployed_model is filled
Putting importance of the drift into perspective according to the importance of the model to be deployed, can help the data scientist to validate that his model can be deployed.
Univariate analysis¶
This graphs shows a particular feature’s distribution over its possible values. In the drop-down menu, the variables are sorted by importance of the variables in the data drift classification. For categorical features, the possible values are sorted by descending difference between the two datasets.
[16]:
SD.plot.generate_fig_univariate('YearRemodAdd')
Distribution of predicted values¶
This graph shows distributions of the production model outputs on both baseline and current datasets.
[17]:
SD.plot.generate_fig_univariate(df_all=SD.df_predict,col='Score',hue="dataset") # works if deployed_model is filled
Jensen Shannon Divergence (JSD). The JSD measures the effect of a data drift on the deployed model performance. A value close to 0 indicates similar data distributions, while a value close to 1 tend to indicate distinct data distributions with a negative effect on the deployed model performance.
[18]:
SD.plot.generate_indicator(
fig_value=SD.js_divergence,
height=280,
width=500,
title="Jensen Shannon Datadrift",
min_gauge=0,
max_gauge=0.2,
) #works if deployed_model is filled
Compile Drift over years¶
Compile Drift et generate report for Year 2008¶
[19]:
house_df_2008 = house_df.loc[house_df['YrSold'] == 2008]
y_df_2008=house_df_2008['SalePrice'].to_frame()
X_df_2008=house_df_2008[house_df_2008.columns.difference(['SalePrice','YrSold'])]
[20]:
SD = SmartDrift(df_current=X_df_2008,
df_baseline=X_df_learning,
deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
encoding=encoder # Optional: if deployed_model and encoder to use this model
)
[21]:
SD.compile(full_validation=True, # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
date_compile_auc = '01/01/2008', # Optional: useful when computing the drift for a time that is not now
datadrift_file = "house_price_auc.csv" # Optional: name of the csv file that contains the performance history of data drift
)
The variable Condition1 has mismatching unique values:
["Within 200' of East-West Railroad"] | []
The variable Condition2 has mismatching unique values:
['Adjacent to arterial street', "Within 200' of North-South Railroad", 'Adjacent to postive off-site feature', 'Near positive off-site feature--park, greenbelt, etc.'] | []
The variable Electrical has mismatching unique values:
['Mixed'] | []
The variable ExterCond has mismatching unique values:
['Excellent'] | []
The variable ExterQual has mismatching unique values:
['Fair'] | []
The variable Exterior1st has mismatching unique values:
[] | ['Imitation Stucco']
The variable Exterior2nd has mismatching unique values:
[] | ['Other', 'Stone']
The variable Foundation has mismatching unique values:
[] | ['Slab', 'Wood']
The variable Functional has mismatching unique values:
['Major Deductions 2'] | []
The variable GarageCond has mismatching unique values:
['Excellent'] | ['Poor']
The variable GarageQual has mismatching unique values:
[] | ['Poor']
The variable GarageType has mismatching unique values:
[] | ['More than one type of garage']
The variable Heating has mismatching unique values:
['Hot water or steam heat other than gas', 'Floor Furnace'] | ['Wall furnace']
The variable MSSubClass has mismatching unique values:
['1-Story w/Finished Attic All Ages'] | []
The variable Neighborhood has mismatching unique values:
['Northpark Villa', 'Bluestem'] | []
The variable RoofMatl has mismatching unique values:
['Membrane', 'Clay or Tile'] | ['Metal']
The variable SaleCondition has mismatching unique values:
[] | ['Sale between family members']
The variable SaleType has mismatching unique values:
['Contract 15% Down payment regular terms', 'Warranty Deed - Cash'] | ['Contract Low Interest', 'Other']
The variable Street has mismatching unique values:
['Gravel'] | []
The computed AUC on the X_test used to build datadrift_classifier is equal to: 0.6877714667557634
[22]:
SD.plot.generate_historical_datadrift_metric() # works if date_compile_auc and/or datadrift_file are filled
Compile Drift et generate report for Year 2009¶
[23]:
house_df_2009 = house_df.loc[house_df['YrSold'] == 2009]
y_df_2009=house_df_2009['SalePrice'].to_frame()
X_df_2009=house_df_2009[house_df_2009.columns.difference(['SalePrice','YrSold'])]
[24]:
SD = SmartDrift(df_current=X_df_2009,
df_baseline=X_df_learning,
deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
encoding=encoder # Optional: if deployed_model and encoder to use this model
)
[25]:
SD.compile(full_validation=True, # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
date_compile_auc = '01/01/2009', # Optional: useful when computing the drift for a time that is not now
datadrift_file = "house_price_auc.csv" # Optional: name of the csv file that contains the performance history of data drift
)
The variable BsmtCond has mismatching unique values:
['Poor -Severe cracking, settling, or wetness'] | []
The variable Condition1 has mismatching unique values:
[] | ['Adjacent to East-West Railroad']
The variable Condition2 has mismatching unique values:
['Adjacent to arterial street'] | []
The variable Electrical has mismatching unique values:
[] | ['60 AMP Fuse Box and mostly knob & tube wiring (poor)']
The variable ExterCond has mismatching unique values:
['Excellent'] | []
The variable ExterQual has mismatching unique values:
['Fair'] | []
The variable Exterior1st has mismatching unique values:
['Brick Common', 'Cinder Block'] | ['Stone', 'Imitation Stucco']
The variable Exterior2nd has mismatching unique values:
['Brick Common', 'Cinder Block'] | ['Other']
The variable Functional has mismatching unique values:
['Major Deductions 2'] | []
The variable GarageCond has mismatching unique values:
['Excellent'] | ['Good']
The variable GarageQual has mismatching unique values:
[] | ['Poor']
The variable GarageType has mismatching unique values:
[] | ['More than one type of garage']
The variable LotConfig has mismatching unique values:
[] | ['Frontage on 3 sides of property']
The variable MSSubClass has mismatching unique values:
['1-Story w/Finished Attic All Ages'] | []
The variable Neighborhood has mismatching unique values:
['Northpark Villa', 'Bluestem'] | ['Veenker']
The variable RoofMatl has mismatching unique values:
[] | ['Metal', 'Wood Shakes']
The variable RoofStyle has mismatching unique values:
['Mansard'] | []
The variable SaleCondition has mismatching unique values:
[] | ['Adjoining Land Purchase']
The variable SaleType has mismatching unique values:
[] | ['Other']
The variable Utilities has mismatching unique values:
['Electricity and Gas Only'] | []
The computed AUC on the X_test used to build datadrift_classifier is equal to: 0.5405695039804042
[26]:
SD.plot.generate_historical_datadrift_metric() # works if date_compile_auc and/or datadrift_file are filled
Compile Drift et generate report for Year 2010¶
[27]:
house_df_2010 = house_df.loc[house_df['YrSold'] == 2010]
y_df_2010=house_df_2010['SalePrice'].to_frame()
X_df_2010=house_df_2010[house_df_2010.columns.difference(['SalePrice','YrSold'])]
[28]:
SD = SmartDrift(df_current=X_df_2010,
df_baseline=X_df_learning,
deployed_model=regressor, # Optional: put in perspective result with importance on deployed model
encoding=encoder # Optional: if deployed_model and encoder to use this model
)
[29]:
SD.compile(full_validation=True, # Optional: to save time, leave the default False value. If True, analyze consistency on modalities between columns.
date_compile_auc = '01/01/2010', # Optional: useful when computing the drift for a time that is not now
datadrift_file = "house_price_auc.csv" # Optional: name of the csv file that contains the performance history of data drift
)
The variable Condition1 has mismatching unique values:
["Within 200' of East-West Railroad"] | []
The variable Electrical has mismatching unique values:
[] | ['60 AMP Fuse Box and mostly knob & tube wiring (poor)']
The variable ExterCond has mismatching unique values:
['Poor'] | []
The variable ExterQual has mismatching unique values:
['Fair'] | []
The variable Exterior1st has mismatching unique values:
['Asphalt Shingles'] | ['Stone', 'Imitation Stucco']
The variable Exterior2nd has mismatching unique values:
['Asphalt Shingles', 'Brick Common'] | ['Other', 'Stone']
The variable Functional has mismatching unique values:
[] | ['Major Deductions 1']
The variable GarageCond has mismatching unique values:
[] | ['Poor', 'Good']
The variable GarageQual has mismatching unique values:
[] | ['Good', 'Excellent', 'Poor']
The variable GarageType has mismatching unique values:
[] | ['More than one type of garage']
The variable Heating has mismatching unique values:
[] | ['Gas hot water or steam heat', 'Wall furnace']
The variable HouseStyle has mismatching unique values:
[] | ['Two and one-half story: 2nd level finished', 'One and one-half story: 2nd level unfinished', 'Two and one-half story: 2nd level unfinished']
The variable LotConfig has mismatching unique values:
[] | ['Frontage on 3 sides of property']
The variable LotShape has mismatching unique values:
[] | ['Irregular']
The variable MSSubClass has mismatching unique values:
['1-Story w/Finished Attic All Ages'] | ['2-1/2 Story All Ages', '1-1/2 Story - Unfinished All Ages']
The variable MSZoning has mismatching unique values:
[] | ['Residential High Density']
The variable Neighborhood has mismatching unique values:
['Northpark Villa'] | ['Veenker']
The variable RoofMatl has mismatching unique values:
[] | ['Wood Shingles', 'Metal', 'Gravel & Tar']
The variable RoofStyle has mismatching unique values:
['Mansard', 'Shed'] | ['Flat']
The variable SaleCondition has mismatching unique values:
[] | ['Adjoining Land Purchase']
The variable SaleType has mismatching unique values:
['Contract 15% Down payment regular terms'] | ['Contract Low Down', 'Contract Low Down payment and low interest', 'Other']
The variable Street has mismatching unique values:
['Gravel'] | []
The computed AUC on the X_test used to build datadrift_classifier is equal to: 0.6978632478632478
[30]:
SD.plot.generate_historical_datadrift_metric() # works if date_compile_auc and/or datadrift_file are filled
