In this article, we will look into the GCP-What-If Tool and focus on the use of the tool to better understand your machine learning models. Here we will talk about how to run it on various platforms and model types.
The What-If Tool is a visual interface designed to help you understand your data sets and the output of your machine learning models. You can run with minimal code from many different platforms, including Jupyter Notebooks, Kollab, TensorBoard, and Cloud AI Platform Notebooks. It is primarily used for the below-listed reasons:
- Available on many platforms
- Supports What-if Analysis
- Visualizes Model Performance
Analysis using the What-If Tool can be helpful during training data collection, model creation, and post-training evaluation. The What-If Tool supports TensorFlow models out of the box and can support models built with any other framework with just a few lines of code. You can also use the What-If Tool with your models deployed on the Cloud AI Platform.
Let’s start with an example in Kollab. Here we’ve trained a TensorFlow model on this Iowa Housing data set from Kaggle. It’s a binary classification model that predicts whether a house is worth more or less than $160,000.
We can instantiate the What-If Tool with a few lines of code by passing it a set of test data points along with the ground truth labels for those examples. The What-If Tool has three tabs with different features.
And the initial view we’re dropped into is the data point editor. Here we can see our model’s prediction for each data point we’ve sent to the What-If Tool. The blue points indicate houses our model classified as above $160,000. In the data point editor, we can also inspect individual data points, change feature values, create custom visualizations, and much more.
The next tab, Performance and Fairness, shows a couple of common evaluation metrics for our model. It also lets us slice data by different features and apply various strategies to optimize for fairness.
The final tab, Features, shows us how balanced our data set is. For each feature, we can see the range of values for the data we’ve sent to the What-If Tool. You can load the Features tab with only a data set even before you’ve trained a model.
We can also use the What-If Tool for models deployed on the Cloud AI Platform. Here are a few models we’ve already deployed on the AI Platform.
We’ll open up an AI Platform Notebook instance here to analyze this XGBoost model.
Then we create an instance of the Wit Config Builder, this time using the set AI Platform model method, passing in my Google Cloud project ID, model name, and model version. The Wit Config Builder contains more methods to customize the What-If Tool for your use case.
Notice that the two of your models are trained on the same data set. We can even use the What-If Tool to compare them. All we need to do is add a line to our Wit Config Builder to tell it which model we want to compare. Now you can compare the performance of both models using various capabilities in the What-If Tool.
The What-If Tool can be used for both classification models and regression models.