* fix, chore, refactor: fixes the issue with job creation request, refactors to use the latest SDKs, follows the new template, sentence corrections
* chore: removes future tense, minor sentence corrections
* chore: addresses the review comments
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore, refactor: follows new template, removes IS_TESTING in clean up, contracts words and some steps, spell correct, K80 to T4
* chore: addresses review comments
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore, fix, refactor: template fixes, replace K80 with T4, remove IS_TESTING, reword some sentences and headings
* fix, refactor, feat, chore: replace K80 with T4, fix the TF version, template based fixes, remove local files in the clean up step
* chore: addresses review comments
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore, feat: template fixes, adds steps to remove training job, local files, and remove future tense and contract long words
* fix: replaces REGION with LOCATION while creating the bucket
* refactor: reduces the budget_milli_node_hours to 1000 from 8000
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* fix, chore, refactor, feat: replaces K80 to T4, replaces gcr pushes to artifact registry pushes, template based fixes, clean up for local files, remove IS_TESTING
* fix: defines the IS_COLAB step before the condition
* fix: REGION is replaced by LOCATION while creating the artifact registry
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* refactor(egen): Refactored code according to the template guidelines
* fix, refactor: fixes the compatible tf version and removes unnecessary import in the clean up step
* fix: adds matplotlib in the installation step
---------
Co-authored-by: rohith-egen <rohith.alla@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore: restructures according to template, contracts text and cells, organizes headings
* refactor: removes the IS_TESTING conditions for steps involving redis instance
* chore: addresses review comments
* fix: adds back the IS_TESTING conditions for redis commands to skip in the test environment
* chore: removes duplicate comment
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore, fix: restructures as per template, rewords some sentences, removes sudo in the option docker run
* feat: adds clean up step for local files
* chore: reverts lowercase to camelcase for artifact registry and other review comments
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore: replaces K80 with T4, Cloud ML with Vertex AI, REGION with LOCATION, restructures from template, & removes future tense
* fix: fixes the docker tag command for colab
* chore, fix: addresses the review comments, tf is pinned to 2.15.1 as the latest tf causes issues
* chore: replaces of with or
* chore: removes the collapsed license comment
* fix: removes the test env specific package update step and contracts the installations into one step to keep tf as 2.15
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore: restructures according to the template, removes future tense, removes IS_TESTING, unused imports
* chore, fix: addresses the review comments, converts npy array to list for running in py-3.9
* chore: adds --it's-- in the sentence
* chore: Getting started --> Get started
* chore: removes the collapsed license comment
* fix: extracts list from numpy objects instead of Dataset object
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* refactor, chore, fix: replaces K80 with T4, fixes to follow the template, fixes to utilize the defined accelerators while training
* chore: addresses the review comments
---------
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* refactor: removes boilerplate code, future tenses, and fixes heading styles
* chore: replaces REGION with LOCATION to match the notebook template
* chore: markdown heading fixes according to guidelines
---------
Co-authored-by: rohith-egen <rohith.alla@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* chore: updates the notebook according to the latest template
* chore: linter test
* refactor: removes IS_TESTING, and os import
* ran linter test using linter.sh
---------
Co-authored-by: rohith-egen <rohith.alla@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* fix: n1-standard-8 changed to n1-standard-16 and Tesla K80 changed to Tesla T4 + refactored code according to the template
* refactor: keeping the machine types same. Original issue with the K80 accelerators as they're no longer supported
---------
Co-authored-by: rohith-egen <rohith.alla@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
* feat: boilerplate reduction 59
* fix: bucket
* spark runtime
* dataproc runtine -> 1.1.20
* updates the build steps + clean up steps
* ran linter test
* adds gcloud components update step for test env + adds needed services in before you begin section
* ran linter test
* moves the gcloud components update to the start + updates experiment fetching
* ran linter test
* updates the aiplatform sdk to the latest
* ran linter test
* removes gcloud components update and adds try except at get_dataframe() method
* ran linter test
* initializes experiment + adds gcloud update + removes try catch
* ran linter test
---------
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* fix: boilerplate reduction 41
* fix: install
* fix: TFDV version
* fix: increase wait time
* increase wait time
* replace INPUT_GS_PATH with TRAINING_DATASET while copying data in GCS + removes future tense + clean up step for batch job + moves the learn more section above clean up section
* ran linter test
---------
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* fix: boilerplate reduction 88
* fix: install
* fix: reduce dataset size
* fix: > 24hrs
* fix: reduce dataset for testing
* fix: reduce dataset for testing
* updates sklearn and fixes the version, elaborates some existing descriptions and adds updates based on the template
* ran linter test
* changes FILE_NAME to LOCAL_FILE_NAME and adds a clean up step for the local saved model
* ran linter test
---------
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* fix: issue 2125
* removes duplicate parameters, reduces max_steps to 100, fixes grammar and updates realted to the writing guidelines
* removes f from the string parameter
* ran linter test
* sets max_steps to 20 and adds lines in the cleanup step to remove the pipeline jobs
* ran linter test
---------
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds the preprocessing steps on the original dataset + updates for boiler plate reduction
* ran linter test
* adds project-id and ticks in the sql queries
* ran linter test
* upgrade: boilerplate reduction 38
* fixes the pipeline-root-path typo, adds dataflow api dependency in the before-you-begin section, restructures the sections according to the template, fixes grammar and headings
* ran linter test
---------
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds the custom tabular classification evaluation notebook
* removes json import
* ran linter test
* fixes the model serving code and adds image
* ran linter test
* adds notebook to the codeowners file
* addresses the review comments: updates text, adds headings
* ran linter test
* boiler-plate reduction, addresses the review comments, adds a constant for table id
* ran linter test
* updates the installation step
* ran linter test
* removes the --user flag while installation
* ran linter test
Co-authored-by: Soheila Zangeneh <49654056+soheilazangeneh@users.noreply.github.com>
Checklist for moving the notebook to the main repo:
- [x] Use the [notebook template](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebook_template.ipynb) as a starting point.
- [x] Follow the style and grammar rules outlined in the above notebook template.
- [x] Verify the notebook runs successfully in Colab since the automated tests cannot guarantee this even when it passes.
- [x] Passes all the required automated checks. You can locally test for formatting and linting with these [instructions](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/docs/contributing.md#code-quality-checks).
- [ ] You have consulted with a tech writer to see if tech writer review is necessary. If so, the notebook has been reviewed by a tech writer, and they have approved it.
- [x] This notebook has been added to the [CODEOWNERS](https://togithub.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/docs/CODEOWNERS) file under `# Official Notebooks` section, pointing to the author or the author's team.
- [x] The Jupyter notebook cleans up any artifacts it has created (datasets, ML models, endpoints, etc) so as not to eat up unnecessary resources.
* adds the custom-model-bq-io-batch-prediction-pipeline notebook to the official folder
* removes the unused libraries and variables
* ran linter test
* adds a raise exception statement to prevent auto-test check as this notebook is just to serve as a template
* resolves review comments: fixes names, removes trailing comma, fixes link format
* removes the raise excpetion step
* ran linter test
* cleans up code, explains sections and renames the notebook
* removes unused variable batch_predict_task
* removes unnecessary imports
* ran linter test
* updates the dependencies
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* upgrades gcpc to latest, adds minor textual corrections
* ran linter test
* minor textual corrections
* ran linter test
* fixes sentence cases, unnecessary capitalizations, article corrections and sentence corrections
* ran linter test
* renames the ground_truth_column argument in classificationeval component to target_field_name as per version 1.0.26 and updates the reference link
* ran linter test
* upgrades gcpc to latest, adds minor textual corrections
* ran linter test
* minor textual corrections
* ran linter test
* fixes sentence cases, unnecessary capitalizations, article corrections and sentence corrections
* ran linter test
* renames the ground_truth_column argument in classificationeval component to target_field_name as per version 1.0.26 and updates the reference link
* ran linter test
* fixes the prediction_score_column unsupported error, adds class_labels field, adds the targetfieldremover component, minor structural/textual updates
* ran linter test & updates the image
* removes unnecessary newline
* ran linter test
Co-authored-by: Soheila Zangeneh <49654056+soheilazangeneh@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates featurestore name with uuid to avoid name collisions, elaboration, textual corrections
* ran linter test
* addresses review comments: adds headings + textual corrections
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* cleans up the notebook,replaces docker with cloud build, textual edits still in progress
* cleans up the notebook
* ran linter test
* changes tf train/serve version to 2.9
* ran linter test
* adds '=' to fix a typo
* ran linter test
* updates the opencv installation package
* ran linter test
* updates the opencv installation dependencies
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* resolves the shell-output issue + updates the structure based on the template
* ran linter test
* fixes issues from review: textual updates, delete_bucket=False
* ran linter test
Co-authored-by: Ivan Cheung <ivans.mailbox@gmail.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* moves the sentiment_analysis notebook from community to official folder after making the updates
* removes unused modules
* ran linter test
* updates the dataset's GCS links and notebook links in the heading
* ran linter test
* fixes the typo(=)
* ran linter test
* removes wait() calls and IS_TESTING condition
* ran linter test
Co-authored-by: Ivan Cheung <ivans.mailbox@gmail.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds the updated predictive-maintenance (managed)notebook from community to official folder
* ran linter test
* resubmitting during phase2
* ran linter test
* addresses the review comments: updates based on the new template, sets delete_bucket to False
* ran linter test
* replaces timestamp with uuid
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* fixes tensorflow version, replaces timestamp with uuid, adds service-account and minor textual changes
* removes second defnition of random library
* ran linter test
* uncomments the user flag and updates the installation command
* ran linter test
* removes the extra backslash
* ran linter test
* updates the installation step to fix long running compatibility checks
* ran linter test
* updates the METADATA path during installation steps
* ran linter test
* adds google-api-core version in the installation
* ran linter test
* removes METADATA step during installation
* ran linter test
* updates google api-core & auth versions
* ran linter test
* fixes tensorflow version, replaces timestamp with uuid, adds service-account and minor textual changes
* removes second defnition of random library
* ran linter test
* uncomments the user flag and updates the installation command
* ran linter test
* removes the extra backslash
* ran linter test
* updates the installation step to fix long running compatibility checks
* ran linter test
* updates the METADATA path during installation steps
* ran linter test
* adds google-api-core version in the installation
* ran linter test
* removes METADATA step during installation
* ran linter test
* updates google api-core & auth versions
* ran linter test
* fixes the issues from review: cell descriptions, parameter definitions, tense changes, 3rd person --> 2nd person, list model after pipeline run
* ran linter test
* removes the METADATA hack and updates the installations
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* fixing links to open notebook - main and images
* linter test changes
* fixes the papermill execution error(hard-coded bucket link was the cause)
* ran linter test
* adds minor textual changes
* ran linter test
* fixes issues from review: future tense, copyright year, section placement, latest sdk methods, new updates from the template
* ran linter test
Co-authored-by: Manuel Amunategui <manuel.amunategui@springml.com>
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds the missing delete_bucket variable, adds steps to configure SERVICE_ACCOUNT
* removes unnecessary random import
* ran linter test
* removes the src folder dependency to run on Colab, adds the pipeline.wait step, updates the cleanup steps
* ran linter test
* fixed issues from review: section posistions, tense changes, section descriptions, template updates
* ran linter test
* fixes exception(reg-test), replaces timestamp with uuid, minor changes
* ran linter test
* resolved review comments: license year, Vertex AI SDK, dataset after objective and future tense
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the configuring steps, replaces timestamp with uuid, expands the imports
* ran linter test
* separates the vertex-ai and bigquery initialization steps
* adds comment to cell_24
* adds blank line to cell_24:7:1
* adds blank line to cell_24:7:1
* ran linter test
* fixes aiplatform+bigquery installation compatibility issue
* ran linter test
* fixes installation dependencies
* ran linter test
* fixes the issues from the review: future tense, section positions, updates from the latest template
* ran linter test
* fixes the issues from the review: Code formatting, delete redundant cells, resource name changes
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* replaces timestamp with uuid #create *task #tag1 replace the TIMESTAMP with uuid in other official notebooks
* ran linter test
* updates the uuid code
* fixes the comment style highlighted through linter-test
* ran linter test
* adds length argument to uuid function defaulted to 8
* ran linter test
* updates and adds the telecom-subscriber-churn-prediction notebook to official and removes from the community
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds manual-scaling config and explanation to the notebook
* ran linter test after installing linter requirement updates
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* adds the ml_ops/stage2/get_Started_bqml_training notebook to official and removes the same from community folder
* ran linter test
* updates the textual content
* ran linter test
* moves the updated stage2/get-started-bqml notebook back to the communit folder
* ran linter test
* updates the header according to the template
* ran linter test
* adds colab part and minor changes
* ran linter test
* retains the newly added code lost in conflicts
* ran linter test
* converts vertex to vertex ai
* ran linter test
* moves deletion of temporary BQ table outside delete_storage condition
* ran linter test
* adds bigquery-storage dependency to the notebook tested on Colab
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the get-started-vertex-experiments notebook in the community folder
* ran linter test
* adds the costs section
* ran linter test
* adds colab part and minor changes
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the get-started-automl-training notebook
* ran linter test
* adds --user flag during installation step
* ran linter test
* updates the clean up step
* ran linter test
* adds colab part and minor changes
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
* adds the updated mlops-stage1-get_started_bq_datasets notebook to the official branch and removes it from the community branch
* removes second instance of create_bigquery_dataset() function
* ran linter test successfully
* adds costs section
* ran linter test successfully
* updates the dependency installation step and GCS bucket explanation
* ran linter test
* adds pyarrow to the installations
* ran linter test
* removes unnecessary installations + adds silent install + moves the notebook back from official to community folder + adds IS_TESTING condition during clean-up
* ran linter test
* resolves the move up?? comment and builtin comment
* ran linter test
* updates textual content about package installation
* ran linter test
* resolves the future-tense and dependency installations comments
* ran linter test
* updates the header according to template
* ran linter test
* adds Colab part and minor changes
* ran linter test
* updates the enable apis step in setup project section
* ran linter test
* changes vertex to vertex ai
* ran linter test
* moves temporary BQ table deletion outside the delete_storage condition
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the get-started-automl-pipelines in the mlops/stage3 folder inside community folder
* replaces the unused variable deploy_op with _
* removes the unused Model import
* adds the costs section
* ran linter test
* adds Colab part to the notebook
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the mlops/stage3/get_started_with_kubeflow_pipelines.ipynb notebook
* fixes unused variables
* fixes conflicting function names
* ran linter test
* adds colab changes
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates: adds delete-batch code + adds colab part + updates textual content
* sets delete_bucket to False as default
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates get-started-featurestore notebook in mlops/stage2
* ran linter test
* adds the colab changes and minor textual changes
* ran linter test
* adds the colab changes to the notebook and minor textual changes
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the mlops/stage3/get_started_with_kubeflow_pipelines.ipynb notebook
* fixes unused variables
* fixes conflicting function names
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the get-started-automl-pipelines in the mlops/stage3 folder inside community folder
* replaces the unused variable deploy_op with _
* removes the unused Model import
* adds the costs section
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
* updates the get-started-automl-training notebook
* ran linter test
* adds --user flag during installation step
* ran linter test
* updates the clean up step
* ran linter test
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
* adds the updated mlops-stage1-get_started_bq_datasets notebook to the official branch and removes it from the community branch
* removes second instance of create_bigquery_dataset() function
* ran linter test successfully
* adds costs section
* ran linter test successfully
* updates the dependency installation step and GCS bucket explanation
* ran linter test
* adds pyarrow to the installations
* ran linter test
* removes unnecessary installations + adds silent install + moves the notebook back from official to community folder + adds IS_TESTING condition during clean-up
* ran linter test
* resolves the move up?? comment and builtin comment
* ran linter test
* updates textual content about package installation
* ran linter test
* resolves the future-tense and dependency installations comments
* ran linter test
* updates the header according to template
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
* adds the ml_ops/stage2/get_Started_bqml_training notebook to official and removes the same from community folder
* ran linter test
* updates the textual content
* ran linter test
* moves the updated stage2/get-started-bqml notebook back to the communit folder
* ran linter test
* updates the header according to the template
* ran linter test
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
* adds automl-text-sentiment-analysis-online notebook
* adds the cleaned up automl-text-sentiment-analysis notebook after running linter test
* adds textual content on what the dataset predicts in the dataset section
* ran the linter test after the update
* adds textual content on what the dataset predicts in the dataset section
* ran the linter test after the update
* corrects the IMPORT_FILE parameter in the notebook
* ran linter test after update
* deletes the source file from the community/sdk folder
* updates the colab, git & workbench links in the notebook
* ran linter test
* updates the license year to 2022 and simplifies the clean-up step for bucket-deletion
* ran linter test
* adds TESTING env condition while deleting the buckets
* ran linter test successfully
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
* adds the automl-video-action-recognition-notebook
* ran linter test
* fixes the dag variable by replacing with job
* ran linter test
* fixes the dag variable by replacing with job
* ran linter test
* corrects the IMPORT_FILE parameter in the notebook
* ran linter test after update
* updates the colab, git & vertex-ai links
* ran linter test
* updates the license year to 2022 and simplifies the lean-up step for bucket created
* ran linter test
* removes the file from the community folder
* adds the TESTING env condition while deleting the buckets
* ran linter test successfully
* adds TESTING env condition while deleting the bucket
* ran linter test successfully
Co-authored-by: Andrew Ferlitsch <aferlitsch@google.com>
Co-authored-by: Karl Weinmeister <11586922+kweinmeister@users.noreply.github.com>
