From 88cf9b3278a6f31e9e81d7aef4da57efb61abd76 Mon Sep 17 00:00:00 2001 From: Andrew Ferlitsch Date: Tue, 17 Oct 2023 10:42:09 -0700 Subject: [PATCH] update indices (#2382) --- notebooks/official/automl/README.md | 4 +- notebooks/official/bigquery_ml/README.md | 16 ++-- notebooks/official/custom/README.md | 2 +- notebooks/official/experiments/README.md | 18 +++- notebooks/official/explainable_ai/README.md | 17 ++++ notebooks/official/feature_store/README.md | 35 ++++++- notebooks/official/generative_ai/README.md | 93 ++++++++++++++++++ notebooks/official/matching_engine/README.md | 55 +++++++++++ notebooks/official/migration/README.md | 6 +- notebooks/official/ml_metadata/README.md | 2 + notebooks/official/model_evaluation/README.md | 12 +-- notebooks/official/pipelines/README.md | 39 ++++---- notebooks/official/prediction/README.md | 73 ++++++++++++++ notebooks/official/sdk/README.md | 48 ++++++++++ notebooks/official/tabnet/README.md | 15 +++ .../official/tabular_workflows/README.md | 2 +- notebooks/official/training/README.md | 95 +++++++++++++++++-- notebooks/official/workbench/README.md | 31 +----- 18 files changed, 481 insertions(+), 82 deletions(-) create mode 100644 notebooks/official/generative_ai/README.md diff --git a/notebooks/official/automl/README.md b/notebooks/official/automl/README.md index 6bdef03c4..159716bc5 100644 --- a/notebooks/official/automl/README.md +++ b/notebooks/official/automl/README.md @@ -49,8 +49,8 @@ The steps performed are: - Make a batch prediction with the BigQuery ML model. - Create a Vertex AI `Dataset` resource. - Train the Vertex AI Forecasting model. -- View the Model evaluation. -- Make a batch prediction with the Model. +- View the Vertex AI Model Evaluation results. +- Make a batch prediction with the Vertex AI Forecasting model. ``` diff --git a/notebooks/official/bigquery_ml/README.md b/notebooks/official/bigquery_ml/README.md index ee79973c5..9d31ed562 100644 --- a/notebooks/official/bigquery_ml/README.md +++ b/notebooks/official/bigquery_ml/README.md @@ -2,21 +2,19 @@ [Online prediction with BigQuery ML](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery_ml/bqml-online-prediction.ipynb) ``` -Learn how to train and deploy a churn prediction model for real-time inference, with the data in BigQuery and model trained using BigQuery ML, registered to Vertex AI Model Registry, and deployed to an endpoint on Vertex AI for online predictions. +In this tutorial, you fetch the required data from a public BigQuery dataset and prepare it for training. The steps performed include: -- Using Python & SQL to query the public data in BigQuery -- Preparing the data for modeling -- Training a classification model using BigQuery ML and registering it to Vertex AI Model Registry -- Inspecting the model on Vertex AI Model Registry -- Deploying the model to an endpoint on Vertex AI -- Making sample online predictions to the model endpoint +- Query and fetch the data from the public BigQuery dataset. +- Prepare the data for training. +- Train a churn classification model using BigQuery ML. +- Save the trained model to Vertex AI Model Registry. +- Deploy the model to a Vertex AI Endpoint. +- Make online prediction requests to the endpoint. ``` -   Learn more about [BigQuery ML](https://cloud.google.com/vertex-ai/docs/beginner/bqml). - [Get started with BigQuery ML Training](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery_ml/get_started_with_bqml_training.ipynb) diff --git a/notebooks/official/custom/README.md b/notebooks/official/custom/README.md index 35fbc1664..b7fc729a1 100644 --- a/notebooks/official/custom/README.md +++ b/notebooks/official/custom/README.md @@ -93,7 +93,7 @@ The steps performed include:    Learn more about [Vertex AI TensorBoard Profiler](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-profiler). -[Get started with Vertex AI Training for XGBoost](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/get_started_vertex_training.ipynb) +[Get started with Vertex AI Training for XGBoost](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/custom/get_started_vertex_training_xgboost.ipynb) ``` Learn how to use `Vertex AI Training` for training a XGBoost custom model. diff --git a/notebooks/official/experiments/README.md b/notebooks/official/experiments/README.md index 2e19dff96..6f6c894cf 100644 --- a/notebooks/official/experiments/README.md +++ b/notebooks/official/experiments/README.md @@ -74,6 +74,22 @@ The steps performed include:    Learn more about [Vertex AI TensorBoard](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview). +[Custom training autologging - Local script](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_custom_training_autologging_local_script.ipynb) + +``` +Learn how to autolog paramenters and metrics of an ML experiment running on Vertex AI training by leveraging the integration with Vertex AI Experiments. + +The steps performed include: + +- Formalize model experiment in a script +- Run model traning using local script on Vertex AI Training +- Check out ML experiment parameters and metrics in Vertex AI Experiments + +``` + +   Learn more about [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments). + + [Get started with Vertex AI Experiments](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_vertex_experiments.ipynb) ``` @@ -107,7 +123,7 @@ The steps performed include:    Learn more about [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training). -[Autologging](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/autologging.ipynb) +[Autologging](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/get_started_with_vertex_experiments_autologging.ipynb) ``` Learn how to use `Vertex AI Autologging`. diff --git a/notebooks/official/explainable_ai/README.md b/notebooks/official/explainable_ai/README.md index c8b5d3e23..4ef02c409 100644 --- a/notebooks/official/explainable_ai/README.md +++ b/notebooks/official/explainable_ai/README.md @@ -169,3 +169,20 @@ The steps performed include:    Learn more about [Vertex AI Prediction](https://cloud.google.com/vertex-ai/docs/predictions/get-predictions). + +[Explaining text classification with Vertex Explainable AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/explainable_ai/xai_text_classification_feature_attributions.ipynb) + +``` +Learn how to configure feature-based explanations using **sampled Shapley method** on a TensorFlow text classification model for online predictions with explanations. + +The steps performed include: + +- Build and train a TensorFlow text classification model +- Upload model for deployment +- Deploy model for online prediction +- Make online prediction with explanations + +``` + +   Learn more about [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview). + diff --git a/notebooks/official/feature_store/README.md b/notebooks/official/feature_store/README.md index 62781b1f7..90cf788e7 100644 --- a/notebooks/official/feature_store/README.md +++ b/notebooks/official/feature_store/README.md @@ -15,6 +15,38 @@ The steps performed include:    Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore). +[Online feature serving and fetching of BigQuery data with Vertex AI Feature Store](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/online_feature_serving_and_fetching_bigquery_data_with_feature_store.ipynb) + +``` +Learn how to create and use an online feature store instance to host and serve data in `BigQuery` with `Vertex AI Feature Store` in an end to end workflow of feature values serving and fetching user journey. + +The steps performed include: + +- Provision an online feature store instance to host and serve data. +- Register a `BigQuery` view with the online feature store instance and set up the sync job. +- Use the online server to fetch feature values for online prediction. + +``` + +   Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/overview). + + +[Online feature serving and vector retrieval of BigQuery data with Vertex AI Feature Store](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/online_feature_serving_and_vector_retrieval_bigquery_data_with_feature_store.ipynb) + +``` +Learn how to create and use an online feature store instance to host and serve data in `BigQuery` with `Vertex AI Feature Store` in an end to end workflow of features serving and vector retrieval user journey. + +The steps performed include: + +- Provision an online feature store instance to host and serve data. +- Create an online feature store instance to serve a `BigQuery` table. +- Use the online server to search nearest neighbors. + +``` + +   Learn more about [Vertex AI Feature Store](https://cloud.google.com/vertex-ai/docs/featurestore/overview). + + [Using Vertex AI Feature Store with Pandas Dataframe](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/feature_store/sdk-feature-store-pandas.ipynb) ``` @@ -22,7 +54,8 @@ Learn how to use `Vertex AI Feature Store` with pandas Dataframe. The steps performed include: -- Ingest Feature values from Pandas DataFrame into Feature Store's Entity types. +- Create Featurestore, entity types and features. +- Ingest feature values from Pandas DataFrame into Feature Store's Entity types. - Read Entity feature values from Online Feature Store into Pandas DataFrame. - Batch serve feature values from your Feature Store into Pandas DataFrame. diff --git a/notebooks/official/generative_ai/README.md b/notebooks/official/generative_ai/README.md new file mode 100644 index 000000000..74484c92f --- /dev/null +++ b/notebooks/official/generative_ai/README.md @@ -0,0 +1,93 @@ + +[Vertex AI LLM Evaluation & Batch Inference](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/batch_eval_llm.ipynb) + +``` +Learn to use Vertex AI to evaluate a large language model. + +The steps performed include: + +- Create Vertex AI Pipeline job using a predefined template for bulk inference. +- Execute the pipeline using Vertex AI Pipelines. +- Produce prediction results against a model for a given dataset. + +``` + +   Learn more about [Overview of Generative AI support on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview). + + +[Vertex AI LLM Reinforcement Learning from Human Feedback](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/genertive_ai/rlhf_tune_llm.ipynb) + +``` +In this tutorial, you will use `Vertex AI RLHF` to tune and deploy a large language model model. + +The steps performed include: + +- Set the number of model tuning steps. +- Create Vertex AI Pipeline job using a predefined template for tuning. +- Execute the pipeline using `Vertex AI Pipelines`. +- Perform online prediction with the tuned model. + +``` + + +[Semantic Search using Embeddings](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/text_embedding_api_semantic_search_with_scann.ipynb) + +``` +In this tutorial, we demonstrate how to create an embedding generated from text and perform a semantic search. + +The steps performed include: +- Installation and imports +- Create embedding dataset +- Create an index +- Query the index + +``` + +   Learn more about [text embedding](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings). + + +[Text Embedding New API](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/text_embedding_new_api.ipynb) + +``` +Learn how to call text embedding latest APIs on two +new models, textembedding-gecko@latest and textembedding-gecko-multilingual@latest: + +The steps performed include: + +- Installation and imports +- Generate embeddings + +``` + +   Learn more about [text embedding api](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings#api_changes_to_models_released_in_or_after_august_2023). + + +[Vertex AI Tuning a PEFT model](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/tune_peft.ipynb) + +``` +Learn to use `Vertex AI LLM` to tune and deploy a PEFT large language model. + +The steps performed include: + +- Get the Vertex AI LLM model. +- Tune the model. + - This will automatically create a Vertex AI endpoint and deploy the model to it. +- Make a prediction using `Vertex AI LLM`. +- Make a prediction using `Vertex AI Prediction` + +``` + + +[Using the Vertex AI SDK with Large Language Models](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/generative_ai/vertex_sdk_llm_snippets.ipynb) + +``` +Learn how to provide text input to Large Language Models available on Vertex AI to test, tune, and deploy generative AI language models. + +The steps performed include: + +- Use the predict endpoints of Vertex AI PaLM API to receive generative AI responses to a message. +- Use the text embedding endpoint to receive a vector representation of a message. +- Perform prompt tuning of an LLM, based on input/output training data. + +``` + diff --git a/notebooks/official/matching_engine/README.md b/notebooks/official/matching_engine/README.md index 289d0629d..ee03c03d3 100644 --- a/notebooks/official/matching_engine/README.md +++ b/notebooks/official/matching_engine/README.md @@ -1,4 +1,21 @@ +[Using Vertex AI Multimodal Embeddings and Vector Search](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_create_multimodal_embeddings.ipynb) + +``` +Learn how to encode custom text embeddings, create an Approximate Nearest Neighbor index, and query against indexes. + +The steps performed include: + +* Convert an image dataset to embeddings +* Create an index +* Upload embeddings to the index +* Create an index endpoint +* Deploy the index to the index endpoint +* Perform an online query + +``` + + [Using Vertex AI Matching Engine for StackOverflow Questions](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_create_stack_overflow_embeddings.ipynb) ``` @@ -16,6 +33,27 @@ The steps performed include:    Learn more about [Vertex AI Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview). +[Using Vertex AI Vector Search and Vertex AI Embeddings for Text for StackOverflow Questions](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_create_stack_overflow_embeddings_vertex.ipynb) + +``` +Learn how to encode text embeddings, create an Approximate Nearest Neighbor index, and query against indexes. + +The steps performed include: + +* Convert a BigQuery dataset to embeddings +* Create an index +* Upload embeddings to the index +* Create an index endpoint +* Deploy the index to the index endpoint +* Perform an online query + +``` + +   Learn more about [Vertex AI Vector Search](https://cloud.google.com/vertex-ai/docs/matching-engine/overview). + +   Learn more about [Vertex AI Embeddings for Text](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings). + + [Using Vertex AI Matching Engine for Text-to-Image Embeddings](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_create_text_to_image_embeddings.ipynb) ``` @@ -33,6 +71,23 @@ The steps performed include:    Learn more about [Vertex AI Matching Engine](https://cloud.google.com/vertex-ai/docs/matching-engine/overview). +[Using Vertex AI Vector Search for Text-to-Image Embeddings](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_create_text_to_image_embeddings.ipynb) + +``` +Learn how to encode custom text embeddings, create an Approximate Nearest Neighbor index, and query against indexes. + +The steps performed include: + +* Create ANN index +* Create an index endpoint with VPC Network +* Deploy ANN index +* Perform online query + +``` + +   Learn more about [Vertex AI Vector Search](https://cloud.google.com/vertex-ai/docs/matching-engine/overview). + + [Create Vertex AI Matching Engine index](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/matching_engine/sdk_matching_engine_for_indexing.ipynb) ``` diff --git a/notebooks/official/migration/README.md b/notebooks/official/migration/README.md index fcbc18a7d..570139deb 100644 --- a/notebooks/official/migration/README.md +++ b/notebooks/official/migration/README.md @@ -37,7 +37,7 @@ The steps performed include:    Learn more about [Object detection for image data](https://cloud.google.com/vertex-ai/docs/training-overview#object_detection_for_images). -[AutoML Video Object Tracking](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/UJ15 Vertex SDK AutoML Object Tracking.ipynb) +[AutoML Video Object Tracking](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/sdk-automl-object-tracking-batch-prediction.ipynb) ``` Learn to use `AutoML` to train a video model and use `Vertex AI Batch Prediction` to do batch predictions. @@ -119,7 +119,7 @@ The steps performed include the following:    Learn more about [Entity extraction for text data](https://cloud.google.com/vertex-ai/docs/training-overview#entity_extraction_for_text). -[AutoML Text Sentiment Analysis](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/UJ8 Vertex SDK AutoML Text Sentiment Analysis.ipynb) +[AutoML Text Sentiment Analysis](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/sdk-automl-text-sentiment-analysis-batch-prediction.ipynb) ``` The objective of this notebook is to build a AutoML Text Sentiment Analysis model. @@ -140,7 +140,7 @@ The steps performed include the following:    Learn more about [Sentiment analysis for text data](https://cloud.google.com/vertex-ai/docs/training-overview#sentiment_analysis_for_text). -[AutoML Video Classification](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/UJ14 Vertex SDK AutoML Video Classification.ipynb) +[AutoML Video Classification](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/migration/sdk-automl-video-classification-batch-prediction.ipynb) ``` Learn to use `AutoML` to train a video model and use `Vertex AI Batch Prediction` to do batch predictions. diff --git a/notebooks/official/ml_metadata/README.md b/notebooks/official/ml_metadata/README.md index 22bd4c21a..bb3853abd 100644 --- a/notebooks/official/ml_metadata/README.md +++ b/notebooks/official/ml_metadata/README.md @@ -14,6 +14,8 @@ The steps performed include:    Learn more about [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training). +   Learn more about [Vertex AI Experiments](https://cloud.google.com/vertex-ai/docs/experiments/intro-vertex-ai-experiments). + [Track parameters and metrics for locally trained models](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/ml_metadata/sdk-metric-parameter-tracking-for-locally-trained-models.ipynb) diff --git a/notebooks/official/model_evaluation/README.md b/notebooks/official/model_evaluation/README.md index 60ae19ccb..ead3ef7cb 100644 --- a/notebooks/official/model_evaluation/README.md +++ b/notebooks/official/model_evaluation/README.md @@ -45,22 +45,22 @@ The steps performed include: [AutoML text classification pipelines using google-cloud-pipeline-components](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/automl_text_classification_model_evaluation.ipynb) ``` -Learn how to use `Vertex AI Pipelines` and `Google Cloud Pipeline Components` to build an `AutoML` text classification model. +Learn how to use `Vertex AI Pipelines` and `Google Cloud Pipeline Components` to build and evaluate an `AutoML` text classification model. The steps performed include: - Create a Vertex AI `Dataset`. -- Train a Automl Text Classification model on the `Dataset` resource. +- Train an Automl Text Classification model on the `Dataset` resource. - Import the trained `AutoML model resource` into the pipeline. - Run a `Batch Prediction` job. -- Evaulate the AutoML model using the `Classification Evaluation Component`. -- Import the classification metrics to the AutoML model resource. +- Evaluate the AutoML model using the `Classification Evaluation Component`. +- Import the evaluation metrics to the AutoML model resource. ```    Learn more about [Vertex AI Model Evaluation](https://cloud.google.com/vertex-ai/docs/evaluation/introduction). -   Learn more about [AutoML Text](https://cloud.google.com/vertex-ai/docs/text-data/classification/prepare-data). +   Learn more about [Classification on text data](https://cloud.google.com/vertex-ai/docs/training-overview#classification_for_text). [Evaluating batch prediction results from AutoML Video classification model](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/automl_video_classification_model_evaluation.ipynb) @@ -84,7 +84,7 @@ The steps performed include:    Learn more about [Classification for video data](https://cloud.google.com/vertex-ai/docs/training-overview#classification_for_videos). -[Evaluating BatchPrediction results from a Custom Tabular classification model](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb) +[Evaluating BatchPrediction results from a custom tabular classification model](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/model_evaluation/custom_tabular_classification_model_evaluation.ipynb) ``` In this tutorial, you train a scikit-learn RandomForest model, save it in Vertex AI Model Registry and learn how to evaluate it through a Vertex AI pipeline job using `google_cloud_pipeline_components`. diff --git a/notebooks/official/pipelines/README.md b/notebooks/official/pipelines/README.md index d4e818c4e..3ea1ee602 100644 --- a/notebooks/official/pipelines/README.md +++ b/notebooks/official/pipelines/README.md @@ -240,26 +240,6 @@ The steps performed include:    Learn more about [BigQuery ML components](https://cloud.google.com/vertex-ai/docs/pipelines/bigqueryml-component). -[Loan eligibility prediction using `google-cloud-pipeline-components` and Spark ML](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_dataproc_tabular.ipynb) - -``` -Learn how to build a Vertex AI pipeline and train a random-forest model using Spark ML for loan-eligibility classification problem. - -The steps performed include: - -* Use the `DataprocPySparkBatchOp` to preprocess data. -* Create a Vertex AI dataset resource on the training data. -* Train a random forest model using PySpark. -* Build a Vertex AI pipeline and run the training job. -* Use the Spark serving image in order to deploy a Spark model on Vertex AI Endpoint. - -``` - -   Learn more about [Vertex AI Pipelines](https://cloud.google.com/vertex-ai/docs/pipelines/introduction). - -   Learn more about [Dataproc components](https://cloud.google.com/vertex-ai/docs/pipelines/dataproc-component). - - [Model train, upload, and deploy using Google Cloud Pipeline Components](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/google_cloud_pipeline_components_model_train_upload_deploy.ipynb) ``` @@ -301,6 +281,25 @@ The steps performed include:    Learn more about [Vertex AI Model components](https://cloud.google.com/vertex-ai/docs/pipelines/model-endpoint-component). +[Vertex AI Pipelines with KFP 2.x](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/kfp2_pipeline.ipynb) + +``` +Learn to use `Vertex AI Pipelines` and KFP 2. + +The steps performed include: + +- Create a KFP pipeline: + - Create a `BigQuery Dataset` resource. + - Export the dataset. + - Train an XGBoost `Model` resource. + - Create an `Endpoint` resource. + - Deploys the `Model` resource to the `Endpoint` resource. +- Compile the KFP pipeline. +- Execute the KFP pipeline using `Vertex AI Pipelines` + +``` + + [Lightweight Python function-based components, and component I/O](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/pipelines/lightweight_functions_component_io_kfp.ipynb) ``` diff --git a/notebooks/official/prediction/README.md b/notebooks/official/prediction/README.md index 2eabc0ec8..5e9bbe3b3 100644 --- a/notebooks/official/prediction/README.md +++ b/notebooks/official/prediction/README.md @@ -15,6 +15,61 @@ The steps performed include:    Learn more about [Vertex AI Batch Prediction](https://cloud.google.com/vertex-ai/docs/tabular-data/classification-regression/get-batch-predictions). +[Get started with Custom Prediction Routine (CPR)](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/get_started_with_cpr.ipynb) + +``` +Learn how to use Custom Prediction Routine for `Vertex AI Predictions`. + +The steps performed include: + +- Write a custom data preprocessor. +- Train the model. +- Build a custom scikit-learn serving container with custom data preprocessing using the Custom Prediction Routine model server. + - Test the model serving container locally. + - Upload and deploy the model serving container to Vertex AI Endpoint. + - Make a prediction request. +- Build a custom scikit-learn serving container with custom predictor (post-processing) using the Custom Prediction Routine model server. + - Implement custom predictor. + - Test the model serving container locally. + - Upload and deploy the model serving container to Vertex AI Endpoint. + - Make a prediction request. +- Build a custom scikit-learn serving container with custom predictor and HTTP request handler using the Custom Prediction Routine model server. + - Implement a custom handler. + - Test the model serving container locally. + - Upload and deploy the model serving container to Vertex AI Endpoint. + - Make a prediction request. +- Customize the Dockerfile for a custom scikit-learn serving container with custom predictor and HTTP request handler using the Custom Prediction Routine model server. + - Implement a custom Dockerfile. + - Test the model serving container locally. + - Upload and deploy the model serving container to Vertex AI Endpoint. + - Make a prediction request. + +``` + +   Learn more about [Custom prediction routines](https://cloud.google.com/vertex-ai/docs/predictions/custom-prediction-routines). + + +[Vertex AI LLM and streaming prediction](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/llm_streaming_prediction.ipynb) + +``` +Learn how to use Vertex AI LLM to download pretrained LLM model, make predictions and finetuning the model. + +The steps performed include: + +- Load a pretrained text generation model. +- Make a non-streaming prediction +- Load a pretrained text generation model, which supports streaming. +- Make a streaming prediction +- Load a pretrained chat model. +- Do a local interactive chat session. +- Do a batch prediction with a text generation model. +- Do a batch prediction with a text embedding model. + +``` + +   Learn more about [Vertex AI Language Models](https://cloud.google.com/python/docs/reference/aiplatform/latest/vertexai.language_models.TextGenerationModel#vertexai_language_models_TextGenerationModel_predict_streaming). + + [Serving PyTorch image models with prebuilt containers on Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/pytorch_image_classification_with_prebuilt_serving_containers.ipynb) ``` @@ -51,3 +106,21 @@ The steps performed include: ``` + +[Vertex AI SDK 2.0 Vertex AI Remote Prediction](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/prediction/sdk2_remote_prediction.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Download and split the dataset +- Perform transformations as a Vertex AI remote training. +- For scikit-learn, PyTorch, TensorFlow, PyTorch Lightning + - Train the model remotely. + - Uptrain the pretrained model remotely. + - Evaluate both the pretrained and uptrained model. + - Make a prediction remotely + +``` + diff --git a/notebooks/official/sdk/README.md b/notebooks/official/sdk/README.md index 3509650e8..6b8c88399 100644 --- a/notebooks/official/sdk/README.md +++ b/notebooks/official/sdk/README.md @@ -40,3 +40,51 @@ The steps performed include:    Learn more about [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training). + +[Train a pytorch model with Vertex AI SDK 2.0 and Bigframes](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/sdk/sdk2_bigframes_pytorch.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Initialize a dataframe from a BigQuery table and split the dataset +- Perform transformations as a Vertex AI remote training. +- Train the model remotely and evaluate the model locally + +``` + +   Learn more about [bigframes](https://cloud.google.com/bigquery/docs/). + + +[Train a scikit-learn model with Vertex AI SDK 2.0 and Bigframes](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/sdk/sdk2_bigframes_sklearn.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Initialize a dataframe from a BigQuery table and split the dataset +- Perform transformations as a Vertex AI remote training. +- Train the model remotely and evaluate the model locally + +``` + +   Learn more about [bigframes](https://cloud.google.com/bigquery/docs/). + + +[Train a Tensorflow Keras model with Vertex AI SDK 2.0 and Bigframes](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/sdk/sdk2_bigframes_tensorflow.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Initialize a dataframe from a BigQuery table and split the dataset +- Perform transformations as a Vertex AI remote training. +- Train the model remotely and evaluate the model locally + +``` + +   Learn more about [bigframes](https://cloud.google.com/bigquery/docs/). + diff --git a/notebooks/official/tabnet/README.md b/notebooks/official/tabnet/README.md index be02d944f..47edce378 100644 --- a/notebooks/official/tabnet/README.md +++ b/notebooks/official/tabnet/README.md @@ -35,6 +35,21 @@ The steps performed include:    Learn more about [Tabular Workflow for TabNet](https://cloud.google.com/vertex-ai/docs/tabular-data/tabular-workflows/tabnet). +[Train a TabNet model using Vertex AI Remote Training with Vertex AI SDK 2.0](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/sdk2_remote_tabnet_training.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Download and split the dataset +- Ingest the data in a Dataframe and perform transformations. +- Train a tabular classification model. +- Train a tabular regression model. + +``` + + [Vertex AI TabNet](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabnet/tabnet_vertex_tutorial.ipynb) ``` diff --git a/notebooks/official/tabular_workflows/README.md b/notebooks/official/tabular_workflows/README.md index 7a7b89a57..117281c06 100644 --- a/notebooks/official/tabular_workflows/README.md +++ b/notebooks/official/tabular_workflows/README.md @@ -20,7 +20,7 @@ The steps performed are: [TabNet Pipeline](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/tabular_workflows/tabnet_on_vertex_pipelines.ipynb) ``` -Learn how to create two classification models using Vertex AI TabNet Tabular Workflows. +Learn how to create classification models on tabular data using two of the Vertex AI TabNet Tabular Workflows. The steps performed include: diff --git a/notebooks/official/training/README.md b/notebooks/official/training/README.md index fa99e810c..596eb04ee 100644 --- a/notebooks/official/training/README.md +++ b/notebooks/official/training/README.md @@ -1,4 +1,20 @@ +[Distributed Vertex AI Hyperparameter Tuning](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/distributed_hyperparameter_tuning.ipynb) + +``` +In this notebook, you create a custom trained model from a Python script in a Docker container. + +The steps performed include: + +- Training using a Python package. +- Report accuracy when hyperparameter tuning. +- Save the model artifacts to Cloud Storage using GCSFuse. + +``` + +   Learn more about [Vertex AI Hyperparameter Tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview). + + [Get started with Vertex AI Distributed Training](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/get_started_with_vertex_distributed_training.ipynb) ``` @@ -33,6 +49,23 @@ The steps performed include:    Learn more about [Vertex AI Hyperparameter Tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview). +[Vertex AI Hyperparameter Tuning for XGBoost](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/hyperparameter_tuning_xgboost.ipynb) + +``` +Learn how to use `Vertex AI Hyperparameter Tuning` for training a XGBoost custom model. + +The steps performed include: + +- Training using a Python package. +- Report accuracy when hyperparameter tuning. +- Save the model artifacts to Cloud Storage using GCSFuse. +- Create a `Vertex AI Model` resource. + +``` + +   Learn more about [Vertex AI Hyperparameter Tuning](https://cloud.google.com/vertex-ai/docs/training/hyperparameter-tuning-overview). + + [PyTorch image classification multi-node distributed data parallel training on cpu using Vertex training with custom container](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_gloo_vertex_training_with_custom_container.ipynb) ``` @@ -43,7 +76,7 @@ The steps performed include: - Setting up your GCP project : Setting up the PROJECT_ID, REGION & SERVICE_ACCOUNT - Creating a cloud storage bucket - Building Custom Container using Artifact Registry and Docker -- Create a Vertex AI tensorboard instance to store your Vertex AI experiment +- Create a Vertex AI TensorBoard instance to store your Vertex AI experiment - Run a Vertex AI SDK CustomContainerTrainingJob ``` @@ -51,22 +84,20 @@ The steps performed include:    Learn more about [Vertex AI Training](https://cloud.google.com/vertex-ai/docs/training/custom-training). -[PyTorch Image Classification Multi-Node Distributed Data Parallel Training on GPU using Vertex AI Training with Custom Container](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb) +[PyTorch image classification multi-node NCCL distributed data parallel training on cpu using Vertex training with custom container](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/multi_node_ddp_nccl_vertex_training_with_custom_container.ipynb) ``` Learn how to create a distributed PyTorch training job using Vertex AI SDK for Python and custom containers. The steps performed include: -- Setting up your GCP project : Setting up the PROJECT_ID, REGION & SERVICE_ACCOUNT -- Creating a cloud storage bucket - Building Custom Container using Artifact Registry and Docker -- Create a Vertex AI Tensorboard Instance to store your Vertex AI experiment +- Create a Vertex AI tensorboard instance to store your Vertex AI experiment - Run a Vertex AI SDK CustomContainerTrainingJob ``` -   Learn more about [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training). +   Learn more about [Vertex AI Training](https://cloud.google.com/vertex-ai/docs/training/custom-training). [Training, tuning and deploying a PyTorch text sentiment classification model on Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/pytorch-text-sentiment-classification-custom-train-deploy.ipynb) @@ -89,10 +120,58 @@ The steps performed include:    Learn more about [Custom training](https://cloud.google.com/vertex-ai/docs/training/custom-training). -[Create a distributed custom training job](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/xgboost_data_parallel_training_on_cpu_using_dask.ipynb) +[Train PyTorch model on Vertex AI with data from Cloud Storage](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/pytorch_gcs_data_training.ipynb) ``` -Learn how to create a distributed training job using Vertex AI SDK for Python. +Learn how to create a training job using PyTorch and a dataset stored on Cloud Storage. + +The steps performed include: + +- Write a custom training script that creates your train & test datasets and trains the model. +- Run a Vertex AI SDK `CustomTrainingJob` + +``` + +   Learn more about [PyTorch integration in Vertex AI](https://cloud.google.com/vertex-ai/docs/start/pytorch). + + +[Vertex AI SDK 2.0 Vertex AI Remote Hyperparameter Tuning for OSS ML frameworks](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/sdk2_remote_hyperparameter_tuning.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Download and split the dataset +- Perform transformations as a Vertex AI remote training. +- For scikit-learn, PyTorch, TensorFlow, PyTorch Lightning, Tabnet + - Tune the model remotely. + - Get the best model. + +``` + + +[Vertex AI SDK 2.0 Vertex AI Remote Training for OSS ML frameworks](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/sdk2_remote_training.ipynb) + +``` +Learn to use `Vertex AI SDK 2. + +The steps performed include: + +- Download and split the dataset +- Perform transformations as a Vertex AI remote training. +- For scikit-learn, PyTorch, TensorFlow, PyTorch Lightning + - Train the model remotely. + - Uptrain the pretrained model remotely. + - Evaluate both the pretrained and uptrained model. + +``` + + +[Distributed XGBoost training with Dask](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/training/xgboost_data_parallel_training_on_cpu_using_dask.ipynb) + +``` +Learn how to create a distributed training job using XGBoost with Dask. The steps performed include: diff --git a/notebooks/official/workbench/README.md b/notebooks/official/workbench/README.md index 26cee0f0b..a4acde546 100644 --- a/notebooks/official/workbench/README.md +++ b/notebooks/official/workbench/README.md @@ -147,10 +147,6 @@ The steps performed include: ``` -   Learn more about [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction). - -   Learn more about [Vertex AI Training](https://cloud.google.com/vertex-ai/docs/training/custom-training). - [Predictive Maintenance using Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/predictive_maintainance/predictive_maintenance_usecase.ipynb) @@ -242,7 +238,7 @@ The steps performed are:    Learn more about [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction). -   Learn more about [Dataproc](https://cloud.google.com/vertex-ai/docs/pipelines/dataproc-component). +   Learn more about [Dataproc Serverless for Spark](https://cloud.google.com/dataproc-serverless/docs/guides/bigquery-connector-spark-example). [SparkML with Dataproc and BigQuery](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/spark/spark_ml.ipynb) @@ -271,28 +267,3 @@ The steps performed are:    Learn more about [Dataproc](https://cloud.google.com/vertex-ai/docs/pipelines/dataproc-component). - -[Telecom subscriber churn prediction on Vertex AI](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/workbench/subscriber_churn_prediction/telecom-subscriber-churn-prediction.ipynb) - -``` -This tutorial shows you how to do exploratory data analysis, preprocess data, train, deploy and get predictions from a churn prediction model on a tabular churn dataset. - -The steps performed include: - -- Load data from a Cloud Storage path -- Perform exploratory data analysis (EDA) -- Preprocess the data -- Train a scikit-learn model -- Evaluate the scikit-learn model -- Save the model to a Cloud Storage path -- Create a model and an endpoint in Vertex AI -- Deploy the trained model to an endpoint -- Generate predictions and explanations on test data from the hosted model -- Undeploy the model resource - -``` - -   Learn more about [Vertex AI Workbench](https://cloud.google.com/vertex-ai/docs/workbench/introduction). - -   Learn more about [Vertex Explainable AI](https://cloud.google.com/vertex-ai/docs/explainable-ai/overview). -