model_garden/notebooks/official/experiments

Build Vertex AI Experiment lineage for custom training

Learn how to integrate preprocessing code in a Vertex AI experiments.

The steps performed include:

- Execute module for preprocessing data
  - Create a dataset artifact
  - Log parameters
-  Execute module for training the model
  - Log parameters
  - Create model artifact
  - Assign tracking lineage to dataset, model and parameters

   Learn more about Vertex AI Experiments.

   Learn more about Vertex ML Metadata.

Track parameters and metrics for locally trained models

Learn how to use Vertex AI Experiments to compare and evaluate model experiments.

The steps performed include:

- log the model parameters
- log the loss and metrics on every epoch to TensorBoard
- log the evaluation metrics

   Learn more about Vertex AI Experiments.

Compare pipeline runs with Vertex AI Experiments

Learn how to use `Vertex AI Experiments` to log a pipeline job and compare different pipeline jobs.

The steps performed include:

* Formalize a training component
* Build a training pipeline
* Run several Pipeline jobs and log their results
* Compare different Pipeline jobs

   Learn more about Vertex AI Experiments.

   Learn more about Vertex AI Pipelines.

Delete Outdated Experiments in Vertex AI TensorBoard

Learn how to delete outdated TensorBoard Experiments to avoid unnecessary storage costs.

The steps performed include:

- How to delete the TB Experiment with a predefined key-value label pair `<label_key, label_value>`

- How to delete the TB Experiments created before the  `create_time`

- How to delete the TB Experiments created before the  `update_time`

   Learn more about Vertex AI TensorBoard.

Custom training autologging - Local script

Learn how to autolog paramenters and metrics of an ML experiment running on Vertex AI training by leveraging the integration with Vertex AI Experiments.

The steps performed include:

- Formalize model experiment in a script
- Run model traning using local script on Vertex AI Training
- Check out ML experiment parameters and metrics in Vertex AI Experiments

   Learn more about Vertex AI Experiments.

Get started with Vertex AI Experiments

Learn how to use `Vertex AI Experiments` when training with `Vertex AI`.

The steps performed include:

- Local (notebook) Training
    - Create an experiment
    - Create a first run in the experiment
    - Log parameters and metrics
    - Create artifact lineage
    - Visualize the experiment results
    - Execute a second run
    - Compare the two runs in the experiment
- Cloud (`Vertex AI`) Training
    - Within the training script:
        - Create an experiment
        - Log parameters and metrics
        - Create artifact lineage
    - Create a `Vertex AI Training` custom job
    - Execute the custom job
    - Visualize the experiment results

   Learn more about Vertex AI Experiments.

   Learn more about Vertex ML Metadata.

   Learn more about Custom training.

Autologging

Learn how to use `Vertex AI Autologging`.

The steps performed include:

- Enable autologging in the Vertex AI SDK.
- Train scikit-learn model and see the resulting experiment run with metrics and parameters autologged to Vertex AI Experiments without setting an experiment run.
- Train Tensorflow model, check autologged metrics and parameters to Vertex AI Experiments by manually setting an experiment run with `aiplatform.start_run()` and `aiplatform.end_run()`.
- Disable autologging in the Vertex AI SDK, train a PyTorch model and check that none of the parameters or metrics are logged.