model_garden/notebooks/community/experiments/vertex_ai_model_experimentation.ipynb

In [ ]:

# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Track, compare, manage experiments with Vertex AI Experiments

Run in Colab

View on GitHub

Open in Vertex AI Workbench

Overview

The goal when developing a model for a problem is to identify the best model for that particular use case. To this end, Vertex AI Experiments enables you to track, analyze, compare, and search across different ML frameworks (for example, TensorFlow, PyTorch, scikit-learn) and training environments.

Vertex AI Experiments enables you to track

• steps of an experiment run, for example, preprocessing, training
• inputs, for example, algorithm, parameters, datasets
• outputs of those steps, for example, models, checkpoints, metrics

Objective

In this tutorial, you learn how to use Vertex AI Experiments to cover model experimentation.

This tutorial uses the following Google Cloud ML services and resources:

• Cloud Storage
• Vertex AI

The steps performed include:

1. Create Experiment
2. Create Run
3. Log Parameters
4. Log Metrics
5. Track Artifacts and Executions
6. Track Vertex AI Pipeline

Dataset

The dataset is the Heart Disease DataSet. This database contains 14 attributes.

The "goal" field refers to the presence of heart disease in the patient.

Costs

This tutorial uses billable components of Google Cloud:

• Vertex AI
• Cloud Storage

Learn about Vertex AI pricing, and Cloud Storage pricing, and use the Pricing Calculator to generate a cost estimate based on your projected usage.

Set up your local development environment

If you are using Colab or Vertex AI Workbench Notebooks, your environment already meets all the requirements to run this notebook. You can skip this step.

NOTE: This notebook has been tested in the following environment:

• Python version = 3.9

Otherwise, make sure your environment meets this notebook's requirements. You need the following:

• The Google Cloud SDK

The Google Cloud guide to Setting up a Python development environment and the Jupyter installation guide provide detailed instructions for meeting these requirements. The following steps provide a condensed set of instructions:

1. Install and initialize the Cloud SDK.

2. Install Python 3.

3. Install virtualenv and create a virtual environment that uses Python 3. Activate the virtual environment.

4. To install Jupyter, run pip3 install jupyter on the command-line in a terminal shell.

5. To launch Jupyter, run jupyter notebook on the command-line in a terminal shell.

6. Open this notebook in the Jupyter Notebook Dashboard.

Installation

Install the following packages required to execute this notebook.

In [ ]:

import os

# The Vertex AI Workbench Notebook product has specific requirements
IS_WORKBENCH_NOTEBOOK = os.getenv("DL_ANACONDA_HOME")
IS_USER_MANAGED_WORKBENCH_NOTEBOOK = os.path.exists(
    "/opt/deeplearning/metadata/env_version"
)

# Vertex AI Notebook requires dependencies to be installed with '--user'
USER_FLAG = ""
if IS_WORKBENCH_NOTEBOOK:
    USER_FLAG = "--user"

! pip3 install --upgrade google-cloud-aiplatform kfp fsspec gcsfs {USER_FLAG} -q --no-warn-conflicts

Restart the kernel

After you install the additional packages, you need to restart the notebook kernel so it can find the packages.

In [ ]:

# Automatically restart kernel after installs
import os

if not os.getenv("IS_TESTING"):
    # Automatically restart kernel after installs
    import IPython

    app = IPython.Application.instance()
    app.kernel.do_shutdown(True)

Before you begin

Set up your Google Cloud project

The following steps are required, regardless of your notebook environment.

1. Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.

2. Make sure that billing is enabled for your project.

3. Enable the Vertex AI API.

4. If you are running this notebook locally, you need to install the Cloud SDK.

5. Enter your project ID in the cell below. Then run the cell to make sure the Cloud SDK uses the right project for all the commands in this notebook.

Note: Jupyter runs lines prefixed with ! as shell commands, and it interpolates Python variables prefixed with $ into these commands.

Set your project ID

If you don't know your project ID, you may be able to get your project ID using gcloud.

In [ ]:

PROJECT_ID = "[your-project-id]"  # @param {type:"string"}

In [ ]:

if PROJECT_ID == "" or PROJECT_ID is None or PROJECT_ID == "[your-project-id]":
    # Get your GCP project id from gcloud
    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null
    PROJECT_ID = shell_output[0]
    print("Project ID:", PROJECT_ID)

In [ ]:

! gcloud config set project $PROJECT_ID

Region

You can also change the REGION variable, which is used for operations throughout the rest of this notebook. Below are regions supported for Vertex AI. It is recommended that you choose the region closest to you.

• Americas: us-central1
• Europe: europe-west4
• Asia Pacific: asia-east1

You may not use a multi-regional bucket for training with Vertex AI. Not all regions provide support for all Vertex AI services.

Learn more about Vertex AI regions.

In [ ]:

REGION = "[your-region]"  # @param {type: "string"}

if REGION == "[your-region]":
    REGION = "us-central1"

UUID

If you are in a live tutorial session, you might be using a shared test account or project. To avoid name collisions between users on resources created, you create a uuid for each instance session, and append it onto the name of resources you create in this tutorial.

In [ ]:

import random
import string


# Generate a uuid of a specifed length(default=8)
def generate_uuid(length: int = 8) -> str:
    return "".join(random.choices(string.ascii_lowercase + string.digits, k=length))


UUID = generate_uuid()

Authenticate your Google Cloud account

If you are using Vertex AI Workbench Notebooks, your environment is already authenticated.

If you are using Colab, run the cell below and follow the instructions when prompted to authenticate your account via oAuth.

Otherwise, follow these steps:

1. In the Cloud Console, go to the Create service account key page.

2. Click Create service account.

3. In the Service account name field, enter a name, and click Create.

4. In the Grant this service account access to project section, click the Role drop-down list. Type "Vertex AI" into the filter box, and select Vertex AI Administrator. Type "Storage Object Admin" into the filter box, and select Storage Object Admin.

5. Click Create. A JSON file that contains your key downloads to your local environment.

6. Enter the path to your service account key as the GOOGLE_APPLICATION_CREDENTIALS variable in the cell below and run the cell.

In [ ]:

# If you are running this notebook in Colab, run this cell and follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

import os
import sys

# If on Vertex AI Workbench, then don't execute this code
IS_COLAB = "google.colab" in sys.modules
if not os.path.exists("/opt/deeplearning/metadata/env_version") and not os.getenv(
    "DL_ANACONDA_HOME"
):
    if "google.colab" in sys.modules:
        from google.colab import auth as google_auth

        google_auth.authenticate_user()

    # If you are running this notebook locally, replace the string below with the
    # path to your service account key and run this cell to authenticate your GCP
    # account.
    elif not os.getenv("IS_TESTING"):
        %env GOOGLE_APPLICATION_CREDENTIALS '[your-service-account-key-path]'

Get your project number

Now that the project ID is set, you get your corresponding project number.

In [ ]:

shell_output = ! gcloud projects list --filter="PROJECT_ID:'{PROJECT_ID}'" --format='value(PROJECT_NUMBER)'
PROJECT_NUMBER = shell_output[0]
print("Project Number:", PROJECT_NUMBER)

Create a Cloud Storage bucket

The following steps are required, regardless of your notebook environment.

Set the name of your Cloud Storage bucket below. It must be unique across all Cloud Storage buckets.

In [ ]:

BUCKET_NAME = "[your-bucket-name]"  # @param {type:"string"}
BUCKET_URI = f"gs://{BUCKET_NAME}"

In [ ]:

if BUCKET_NAME == "" or BUCKET_NAME is None or BUCKET_NAME == "[your-bucket-name]":
    BUCKET_NAME = PROJECT_ID + "-aip-" + UUID
    BUCKET_URI = f"gs://{BUCKET_NAME}"

Only if your bucket doesn't already exist: Run the following cell to create your Cloud Storage bucket.

In [ ]:

! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI

Finally, validate access to your Cloud Storage bucket by examining its contents:

In [ ]:

! gcloud storage ls --all-versions --long $BUCKET_URI

Service Account

If you do not want to use your project's Compute Engine service account, set SERVICE_ACCOUNT to another service account ID.

In [ ]:

SERVICE_ACCOUNT = "[your-service-account]"  # @param {type:"string"}

In [ ]:

if (
    SERVICE_ACCOUNT == ""
    or SERVICE_ACCOUNT is None
    or SERVICE_ACCOUNT == "[your-service-account]"
):
    # Get your service account from gcloud
    if not IS_COLAB:
        shell_output = !gcloud auth list 2>/dev/null
        SERVICE_ACCOUNT = shell_output[2].replace("*", "").strip()

    else:  # IS_COLAB:
        shell_output = ! gcloud projects describe  $PROJECT_ID
        project_number = shell_output[-1].split(":")[1].strip().replace("'", "")
        SERVICE_ACCOUNT = f"{project_number}-compute@developer.gserviceaccount.com"

    print("Service Account:", SERVICE_ACCOUNT)

Set service account access

Run the following commands to grant your service account access to {TODO; i.e., read and write pipeline artifacts} in the bucket that you created in the previous step. You only need to run this step once per service account.

In [ ]:

! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectCreator

! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer

Import libraries

In [ ]:

import kfp.v2.compiler as compiler
import kfp.v2.dsl as dsl
import pandas as pd
import tensorflow as tf
from google.cloud import aiplatform as vertex_ai
from kfp.v2.dsl import Metrics, Model, Output, component
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.layers import IntegerLookup, Normalization, StringLookup

Define constants

In [ ]:

EXPERIMENT_NAME = "[your-experiment-name]"  # @param {type:"string"}

In [ ]:

if EXPERIMENT_NAME == "[your-experiment-name]" or EXPERIMENT_NAME is None:
    EXPERIMENT_NAME = "my-experiment-" + UUID

Helpers

Below you have the following helper functions:

• dataframe_to_dataset to convert a Pandas dataframe to a tf.data.Dataset
• encode_numerical_feature to create a normalizer and encode a numerical feature.
• encode_categorical_feature to encode a categorical feature.

In [ ]:

def dataframe_to_dataset(dataframe):
    """
    Convert a Pandas dataframe to a tf.data.Dataset.
    Args:
        dataframe: Pandas dataframe
    Returns:
        tf.data.Dataset
    """
    dataframe = dataframe.copy()
    labels = dataframe.pop("target")
    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
    return ds


def encode_numerical_feature(feature, name, dataset):
    """
    Create a normalizer and encode a numerical feature.
    Args:
        feature: the feature to encode
        name: the name of the feature
        dataset: tf.data.Dataset
    Returns:
        the encoded feature
    """
    # Create a Normalization layer for our feature
    normalizer = Normalization()

    # Prepare a Dataset that only yields our feature
    feature_ds = dataset.map(lambda x, y: x[name])
    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))

    # Learn the statistics of the data
    normalizer.adapt(feature_ds)

    # Normalize the input feature
    encoded_feature = normalizer(feature)
    return encoded_feature


def encode_categorical_feature(feature, name, dataset, is_string):
    """
    Encode a categorical feature.
    Args:
        feature: the feature to encode
        name: the name of the feature
        dataset: tf.data.Dataset
        is_string: whether the feature is a string
    Returns:
        the encoded feature
    """
    lookup_class = StringLookup if is_string else IntegerLookup
    # Create a lookup layer which will turn strings into integer indices
    lookup = lookup_class(output_mode="binary")

    # Prepare a Dataset that only yields our feature
    feature_ds = dataset.map(lambda x, y: x[name])
    feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))

    # Learn the set of possible string values and assign them a fixed integer index
    lookup.adapt(feature_ds)

    # Turn the string input into integer indices
    encoded_feature = lookup(feature)
    return encoded_feature

Initialize Vertex AI SDK for Python

Initialize the Vertex AI SDK for Python for your project and corresponding bucket.

In [ ]:

vertex_ai.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)

Create TensorBoard instance using Vertex AI TensorBoard

You can upload your TensorBoard logs by first creating a Vertex AI TensorBoard instance.

Notice that if you did not activate yet, Vertex AI TensorBoard charges a monthly fee of $300 per unique active user.

Learn more about TensorBoard overview.

In [ ]:

vertex_ai_tb = vertex_ai.Tensorboard.create()

In [ ]:

vertex_ai.init(experiment=EXPERIMENT_NAME, experiment_tensorboard=vertex_ai_tb)

Model Experimentation and Formalization with Vertex AI Experiments

Vertex AI enables users to track the steps (for example, preprocessing, training) of an experiment run, and track inputs (for example, algorithm, parameters, datasets) and outputs (for example, models, checkpoints, metrics) of those steps.

To better understanding how parameters and metrics are stored and organized, the following concepts are explained:

1. Experiments describe a context that groups your runs and the artifacts you create into a logical session. For example, in this notebook you create an Experiment and log data to that experiment.

2. Run represents a single path/avenue that you executed while performing an experiment. A run includes artifacts that you used as inputs or outputs, and parameters that you used in this execution. An Experiment can contain multiple runs.

You can use the Vertex AI SDK for Python to track metrics and parameters models trained locally for each experiment across several experiment runs.

Initiate an experiment and run an experiment run

You define several experiment configurations, run experiments and track them in Vertex AI Experiments

In [ ]:

RUN_NAME = "run-1"
my_run = vertex_ai.start_run(RUN_NAME)

Prepare the data for model training

Track the dataset and data configuration

In [ ]:

file_url = "http://storage.googleapis.com/download.tensorflow.org/data/heart.csv"
dataset_artifact = vertex_ai.Artifact.create(
    schema_title="system.Dataset",
    resource_id=f"{EXPERIMENT_NAME}-heart-data",
    uri=file_url,
    display_name="heart data",
)

Initiate experiment configuration

In [ ]:

params = dict(
    dataset_uri=dataset_artifact.uri,
    dataset_fraction_split=0.2,
    dataset_batch=32,
    random_state=1337,
)

vertex_ai.log_params(params)
vertex_ai.get_experiment_df()

Read the dataset

In [ ]:

dataframe = pd.read_csv(file_url)
dataframe.head()

Create training, test and validation datasets and track them in Vertex ML Metadata for data lineage

Run an execution to prepare training data and track resulting datasets as part of Experiment lineage.

In [ ]:

with vertex_ai.start_execution(
    schema_title="system.ContainerExecution", display_name=f"{RUN_NAME} data split"
) as exc:
    exc.assign_input_artifacts([dataset_artifact])

    # Train, test and validation split
    val_dataframe = dataframe.sample(
        frac=params["dataset_fraction_split"], random_state=params["random_state"]
    )
    test_dataframe = val_dataframe.sample(
        frac=params["dataset_fraction_split"], random_state=params["random_state"]
    )
    train_dataframe = dataframe.drop(val_dataframe.index)

    train_uri = f"{BUCKET_URI}/{EXPERIMENT_NAME}/{RUN_NAME}/data/heart_train.csv"
    test_uri = f"{BUCKET_URI}/{EXPERIMENT_NAME}/{RUN_NAME}/data/heart_test.csv"
    val_uri = f"{BUCKET_URI}/{EXPERIMENT_NAME}/{RUN_NAME}/data/heart_val.csv"

    # Materialize data
    train_dataframe.to_csv(train_uri)
    val_dataframe.to_csv(val_uri)
    test_dataframe.to_csv(val_uri)

    # Create Vertex AI Datasets
    train_metadata = vertex_ai.Artifact.create(
        schema_title="system.Dataset", uri=train_uri, display_name="train split"
    )
    val_metadata = vertex_ai.Artifact.create(
        schema_title="system.Dataset", uri=val_uri, display_name="val split"
    )
    test_metadata = vertex_ai.Artifact.create(
        schema_title="system.Dataset", uri=test_uri, display_name="test split"
    )

    exc.assign_output_artifacts([train_metadata, val_metadata, test_metadata])

#### Feature Engineering

Trasform dataframe into TF dataset

In [ ]:

train_ds = (
    dataframe_to_dataset(train_dataframe)
    .batch(params["dataset_batch"])
    .shuffle(buffer_size=len(train_dataframe))
)
val_ds = dataframe_to_dataset(val_dataframe).batch(params["dataset_batch"])
test_ds = dataframe_to_dataset(test_dataframe).batch(params["dataset_batch"])

Generate features

The steps performed in this part includes categorical and integer features encoding.

In [ ]:

# Categorical features encoded as integers
sex = keras.Input(shape=(1,), name="sex", dtype="int64")
cp = keras.Input(shape=(1,), name="cp", dtype="int64")
fbs = keras.Input(shape=(1,), name="fbs", dtype="int64")
restecg = keras.Input(shape=(1,), name="restecg", dtype="int64")
exang = keras.Input(shape=(1,), name="exang", dtype="int64")
ca = keras.Input(shape=(1,), name="ca", dtype="int64")

# Categorical feature encoded as string
thal = keras.Input(shape=(1,), name="thal", dtype="string")

# Numerical features
age = keras.Input(shape=(1,), name="age")
trestbps = keras.Input(shape=(1,), name="trestbps")
chol = keras.Input(shape=(1,), name="chol")
thalach = keras.Input(shape=(1,), name="thalach")
oldpeak = keras.Input(shape=(1,), name="oldpeak")
slope = keras.Input(shape=(1,), name="slope")

all_inputs = [
    sex,
    cp,
    fbs,
    restecg,
    exang,
    ca,
    thal,
    age,
    trestbps,
    chol,
    thalach,
    oldpeak,
    slope,
]

# Integer categorical features
sex_encoded = encode_categorical_feature(sex, "sex", train_ds, False)
cp_encoded = encode_categorical_feature(cp, "cp", train_ds, False)
fbs_encoded = encode_categorical_feature(fbs, "fbs", train_ds, False)
restecg_encoded = encode_categorical_feature(restecg, "restecg", train_ds, False)
exang_encoded = encode_categorical_feature(exang, "exang", train_ds, False)
ca_encoded = encode_categorical_feature(ca, "ca", train_ds, False)

# String categorical features
thal_encoded = encode_categorical_feature(thal, "thal", train_ds, True)

# Numerical features
age_encoded = encode_numerical_feature(age, "age", train_ds)
trestbps_encoded = encode_numerical_feature(trestbps, "trestbps", train_ds)
chol_encoded = encode_numerical_feature(chol, "chol", train_ds)
thalach_encoded = encode_numerical_feature(thalach, "thalach", train_ds)
oldpeak_encoded = encode_numerical_feature(oldpeak, "oldpeak", train_ds)
slope_encoded = encode_numerical_feature(slope, "slope", train_ds)

all_features = layers.concatenate(
    [
        sex_encoded,
        cp_encoded,
        fbs_encoded,
        restecg_encoded,
        exang_encoded,
        slope_encoded,
        ca_encoded,
        thal_encoded,
        age_encoded,
        trestbps_encoded,
        chol_encoded,
        thalach_encoded,
        oldpeak_encoded,
    ]
)

Build and train the model

Track model configuration

In [ ]:

params.update(n_units=32, activation="relu", dropout_rate=0.5)

vertex_ai.log_params(params)
vertex_ai.get_experiment_df()

Build the model

In [ ]:

x = layers.Dense(params["n_units"], activation=params["activation"])(all_features)
x = layers.Dropout(params["dropout_rate"])(x)
output = layers.Dense(1, activation="sigmoid")(x)
model = keras.Model(all_inputs, output)
model.compile("adam", "binary_crossentropy", metrics=["accuracy"])

Train the model and track metrics

In [ ]:

with vertex_ai.start_execution(
    schema_title="system.ContainerExecution", display_name=f"{RUN_NAME} train"
) as exc:

    exc.assign_input_artifacts([train_metadata, val_metadata, test_metadata])

    params.update(epochs=10)
    vertex_ai.log_params(params)
    history = model.fit(train_ds, epochs=params["epochs"], validation_data=val_ds)
    for i in range(history.params["epochs"]):
        vertex_ai.log_time_series_metrics(
            dict(
                train_loss=history.history["loss"][i],
                train_accuracy=history.history["accuracy"][i],
                val_loss=history.history["val_loss"][i],
                val_accuracy=history.history["val_accuracy"][i],
            )
        )

    metrics = model.evaluate(test_ds, return_dict=True)
    vertex_ai.log_metrics(
        dict(
            loss=metrics["loss"],
            accurancy=metrics["accuracy"],
        )
    )

    model_uri = f"{BUCKET_URI}/{EXPERIMENT_NAME}/{RUN_NAME}/model/"
    model.save(model_uri)

    model_metadata = vertex_ai.Artifact.create(
        schema_title="system.Model", uri=model_uri, display_name="trained heart model"
    )

    exc.assign_output_artifacts([model_metadata])

Visualizing experiment results

Plot metrics

In [ ]:

vertex_ai.get_experiment_df()

Plot metrics per epoch

In [ ]:

my_run.get_time_series_data_frame()

Visualize experiment in Cloud Console

In [ ]:

print("Vertex AI Experiments:")
print(
    f"https://console.cloud.google.com/ai/platform/experiments/experiments?folder=&organizationId=&project={PROJECT_ID}"
)

In [ ]:

vertex_ai.end_run()

Formalize your experiment in a Vertex AI Pipeline

Create a custom component

The section below create a custom trainer component that wraps the experiment code in a pipeline component

In [ ]:

@component(packages_to_install=["tensorflow", "pandas"])
def tabular_trainer(
    dataset_uri: str,
    dataset_fraction_split: float,
    dataset_batch: int,
    random_state: int,
    n_units: int,
    activation: str,
    dropout_rate: float,
    epochs: int,
    metrics: Output[Metrics],
    model_metadata: Output[Model],
):

    import pandas as pd
    import tensorflow as tf
    from tensorflow import keras
    from tensorflow.keras import layers
    from tensorflow.keras.layers import (IntegerLookup, Normalization,
                                         StringLookup)

    dataframe = pd.read_csv(dataset_uri)
    dataframe.head()

    val_dataframe = dataframe.sample(
        frac=dataset_fraction_split, random_state=random_state
    )
    test_dataframe = val_dataframe.sample(
        frac=dataset_fraction_split, random_state=random_state
    )
    train_dataframe = dataframe.drop(val_dataframe.index)

    def dataframe_to_dataset(dataframe):
        """
        Convert a Pandas dataframe to a tf.data.Dataset.
        Args:
            dataframe: Pandas dataframe
        Returns:
            tf.data.Dataset
        """
        dataframe = dataframe.copy()
        labels = dataframe.pop("target")
        ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
        return ds

    train_ds = (
        dataframe_to_dataset(train_dataframe)
        .batch(dataset_batch)
        .shuffle(buffer_size=len(train_dataframe))
    )
    val_ds = dataframe_to_dataset(val_dataframe).batch(dataset_batch)
    test_ds = dataframe_to_dataset(test_dataframe).batch(dataset_batch)

    def encode_numerical_feature(feature, name, dataset):
        """
        Create a normalizer and encode a numerical feature.
        Args:
          feature: the feature to encode
          name: the name of the feature
          dataset: tf.data.Dataset
        Returns:
          the encoded feature
        """
        # Create a Normalization layer for our feature
        normalizer = Normalization()

        # Prepare a Dataset that only yields our feature
        feature_ds = dataset.map(lambda x, y: x[name])
        feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))

        # Learn the statistics of the data
        normalizer.adapt(feature_ds)

        # Normalize the input feature
        encoded_feature = normalizer(feature)
        return encoded_feature

    def encode_categorical_feature(feature, name, dataset, is_string):
        """
        Encode a categorical feature.
        Args:
          feature: the feature to encode
          name: the name of the feature
          dataset: tf.data.Dataset
          is_string: whether the feature is a string
        Returns:
          the encoded feature
        """
        lookup_class = StringLookup if is_string else IntegerLookup
        # Create a lookup layer which will turn strings into integer indices
        lookup = lookup_class(output_mode="binary")

        # Prepare a Dataset that only yields our feature
        feature_ds = dataset.map(lambda x, y: x[name])
        feature_ds = feature_ds.map(lambda x: tf.expand_dims(x, -1))

        # Learn the set of possible string values and assign them a fixed integer index
        lookup.adapt(feature_ds)

        # Turn the string input into integer indices
        encoded_feature = lookup(feature)
        return encoded_feature

    # Categorical features encoded as integers
    sex = keras.Input(shape=(1,), name="sex", dtype="int64")
    cp = keras.Input(shape=(1,), name="cp", dtype="int64")
    fbs = keras.Input(shape=(1,), name="fbs", dtype="int64")
    restecg = keras.Input(shape=(1,), name="restecg", dtype="int64")
    exang = keras.Input(shape=(1,), name="exang", dtype="int64")
    ca = keras.Input(shape=(1,), name="ca", dtype="int64")

    # Categorical feature encoded as string
    thal = keras.Input(shape=(1,), name="thal", dtype="string")

    # Numerical features
    age = keras.Input(shape=(1,), name="age")
    trestbps = keras.Input(shape=(1,), name="trestbps")
    chol = keras.Input(shape=(1,), name="chol")
    thalach = keras.Input(shape=(1,), name="thalach")
    oldpeak = keras.Input(shape=(1,), name="oldpeak")
    slope = keras.Input(shape=(1,), name="slope")

    all_inputs = [
        sex,
        cp,
        fbs,
        restecg,
        exang,
        ca,
        thal,
        age,
        trestbps,
        chol,
        thalach,
        oldpeak,
        slope,
    ]

    # Integer categorical features
    sex_encoded = encode_categorical_feature(sex, "sex", train_ds, False)
    cp_encoded = encode_categorical_feature(cp, "cp", train_ds, False)
    fbs_encoded = encode_categorical_feature(fbs, "fbs", train_ds, False)
    restecg_encoded = encode_categorical_feature(restecg, "restecg", train_ds, False)
    exang_encoded = encode_categorical_feature(exang, "exang", train_ds, False)
    ca_encoded = encode_categorical_feature(ca, "ca", train_ds, False)

    # String categorical features
    thal_encoded = encode_categorical_feature(thal, "thal", train_ds, True)

    # Numerical features
    age_encoded = encode_numerical_feature(age, "age", train_ds)
    trestbps_encoded = encode_numerical_feature(trestbps, "trestbps", train_ds)
    chol_encoded = encode_numerical_feature(chol, "chol", train_ds)
    thalach_encoded = encode_numerical_feature(thalach, "thalach", train_ds)
    oldpeak_encoded = encode_numerical_feature(oldpeak, "oldpeak", train_ds)
    slope_encoded = encode_numerical_feature(slope, "slope", train_ds)

    all_features = layers.concatenate(
        [
            sex_encoded,
            cp_encoded,
            fbs_encoded,
            restecg_encoded,
            exang_encoded,
            slope_encoded,
            ca_encoded,
            thal_encoded,
            age_encoded,
            trestbps_encoded,
            chol_encoded,
            thalach_encoded,
            oldpeak_encoded,
        ]
    )

    x = layers.Dense(n_units, activation=activation)(all_features)
    x = layers.Dropout(dropout_rate)(x)
    output = layers.Dense(1, activation="sigmoid")(x)
    model = keras.Model(all_inputs, output)
    model.compile("adam", "binary_crossentropy", metrics=["accuracy"])

    model.fit(train_ds, epochs=epochs, validation_data=val_ds)

    m = model.evaluate(test_ds, return_dict=True)
    metrics.metadata.update(
        dict(
            loss=m["loss"],
            accurancy=m["accuracy"],
        )
    )

    model.save(model_metadata.uri)

Build the ML pipeline

Build and compile the pipeline including the trainer component

In [ ]:

@dsl.pipeline(name="simple-tabular-pipeline")
def pipeline(
    dataset_uri: str,
    dataset_fraction_split: float,
    dataset_batch: int,
    random_state: int,
    n_units: int,
    activation: str,
    dropout_rate: float,
    epochs: int,
):

    tabular_trainer(
        dataset_uri=dataset_uri,
        dataset_fraction_split=dataset_fraction_split,
        dataset_batch=dataset_batch,
        random_state=random_state,
        n_units=n_units,
        activation=activation,
        dropout_rate=dropout_rate,
        epochs=epochs,
    )


compiler.Compiler().compile(pipeline_func=pipeline, package_path="pipeline.json")

Submit the experiment pipeline run

In [ ]:

job = vertex_ai.PipelineJob(
    display_name="my pipeline run",
    template_path="pipeline.json",
    job_id=f"pipeline-{RUN_NAME}",
    pipeline_root=BUCKET_URI,
    parameter_values={**params},
)

job.submit(experiment=EXPERIMENT_NAME)

Check the pipeline experiment status

In [ ]:

vertex_ai.get_experiment_df()

In [ ]:

job.wait()

In [ ]:

vertex_ai.get_experiment_df()

Visualize experiment in Cloud Console

In [ ]:

print("Vertex AI Experiments:")
print(
    f"https://console.cloud.google.com/ai/platform/experiments/experiments?folder=&organizationId=&project={PROJECT_ID}"
)

Cleaning up

To clean up all Google Cloud resources used in this project, you can delete the Google Cloud project you used for the tutorial.

Otherwise, you can delete the individual resources you created in this tutorial.

In [ ]:

# Delete pipeline
job.delete()

# Delete experiment
exp = vertex_ai.Experiment(EXPERIMENT_NAME)
exp.delete(delete_backing_tensorboard_runs=True)

# Delete Tensorboard
vertex_ai_tb.delete()

# Delete Artifacts
artifacts_list = vertex_ai.Artifact.list()
for artifact in artifacts_list:
    vertex_ai.Artifact.delete(artifact)

# Delete Contexts
context_list = vertex_ai.Context.list()
for context in context_list:
    vertex_ai.Context.delete(context)


# Delete Cloud Storage objects that were created
delete_bucket = False

if delete_bucket or os.getenv("IS_TESTING"):
    ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}