Commit Graph
3027 Commits
Author SHA1 Message Date
Eric DongandGitHub 77e0230eb8 Update README.md 3 (#3103)
Add usage section
2024-06-17 14:51:12 +00:00
8895075d28 chore: renames 'matching_engine' to 'vector_search' and 'Matching Engine' to 'Vector Search' (#3096)
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-17 12:40:24 +00:00
Katie NguyenandGitHub f381ae9817 feat: add colab enterprise link format to script (#3097) 2024-06-17 12:37:28 +00:00
Mend RenovateandGitHub 0b623038fa chore(deps): update dependency flake8 to v7.1.0 (#3099) 2024-06-17 12:36:41 +00:00
7cd354436e chore, refactor, fix, feat(egen): template structure+K80GPU+contract words+clean up (#3085)
* chore, refactor, fix, feat: template structure, removes redundant code, contracts content, replaces K80 with T4, cleans up local files

* chore,chore, fix: addresses review comments+ sets replica=1,accelerator_count=4

* fix, chore: reduces the GPU count to 1, rewords the title

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-14 21:12:17 +00:00
Eric DongandGitHub d7c334353c Update README.md 2 (#3091)
Continued to update README.
2024-06-14 17:10:37 +00:00
kittyabsandGitHub ba80e59b1e Update tensorboard_vertex_ai_pipelines_integration.ipynb (#3036)
* Update tensorboard_vertex_ai_pipelines_integration.ipynb

added colab enterprise link and icon. Deleted <br> line, too

* Update tensorboard_vertex_ai_pipelines_integration.ipynb

fixed - added missing .ipynb
2024-06-14 14:23:43 +00:00
62b3f0af1b Update llama3 finetuning notebook (#3090)
* Update llama3 finetuning notebook

* Update model_garden_pytorch_llama3_finetuning.ipynb

fix linter issue

* Update model_garden_pytorch_llama3_finetuning.ipynb

Remove unnecessary metadata

---------

Co-authored-by: minwoopark <minwoopark@google.com>
2024-06-13 23:54:58 +00:00
bd10dede25 refactor(egen): follows new template+sentence and other minor corrections (#3073)
* chore: restructures according to template, contracts text and cells, organizes headings

* refactor: removes the IS_TESTING conditions for steps involving redis instance

* chore: addresses review comments

* fix: adds back the IS_TESTING conditions for redis commands to skip in the test environment

* chore: removes duplicate comment

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-13 20:52:43 +00:00
7e8bf8a126 fix, refactor, chore(egen): follows new template, adds dataflow enable step, refactor (#3089)
* Updated template of notebook

* fix, refactor, chore: follows new template, adds dataflow enable step, cleanup steps for files, removes IS_TESTING in cleanup, REGION==>LOCATION

* chore: addresses the review comments

---------

Co-authored-by: nileshspringml <nilesh.mahajan@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-13 20:36:50 +00:00
nileshspringmlandGitHub 847d38642c refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3086)
* Did required changes in notebook template

* Service account permission changed as we don't need admin level access for this notebook

* Fixed issue based on PR feedback
2024-06-13 20:31:07 +00:00
Gary WeiandGitHub 697a4c6e88 Add dreambooth-lora-sdxl task for SDXL base model in the dreambooth finetune Gradio notebook. (#3082)
* Add controlnet-canny to the Gradio playground, and some additional UX enhancement.

* Minor fixes.

* Minor fixes

* Add additional document regarding the list of supported models, and some UI enhancement.

* Minor update to the hyperlink.

* Rewrite the SD2.1 dreambooth finetune notebook.

* Add code owners.

* Some minor changes to the stable diffusion 2.1 and sd-xl notebooks.

* some additional minor fixes.

* additional fixes.

* Create a notebook to demonstrate dreambooth LoRA finetune for SD-XL model.

* minor updates

* add to the codeowner list.

* merge conflict.

* minor fix to the Gradio UI workshop notebook.

* Some minor updates to the SD2.1 deployment notebook.

* Minor update the `sd-xl` deployment notebook, based on the QA feedback.

* Add a few community models to the Gradio workshop.

* Switch `mediapipe-train` docker container from `vertex-ai-restricted` to `vertex-ai`, in the `mediapipe-train` notebooks.

* Create a notebook for model `instantx/instantid`.

* Update Gradio notebook to use the latest Gradio version and fix some bugs.

1. Update Gradio version to 4.29.0, as it complains 3.50.0 is too old.
2. Uninstall nest-asyncio and uvloop as a workaround to b/339301920 and https://github.com/gradio-app/gradio/issues/8238#issuecomment-2101066984.

* Resolve merge conflict.

* minor updates.

* minor updates.

* Merge some SD notebook in g3 and github.

* Remove the unused variable in the controlnet notebook.

* minor updates.

* include the SD1.5 dreambooth notebook.

* Include the sd1.5 dreambooth notebook.

* Improve the stable diffusion dreambooth tuning CUJ in the Gradio notebook.

* minor update.

* Add dreambooth-lora-sdxl task for SDXL base model in the dreambooth finetune Gradio notebook.
2024-06-13 14:01:39 +00:00
f5c17c0700 refactor, chore, feat(egen): contracts content, steps & template based fixes (#3075)
* refactor, chore, feat: removes/contracts long explanations, template based fixes, adds step to remove locally generated files

* fix: adds the missing import, replaces code markdown with bold style at some places

* fix: replaces REGION with LOCATION

* chore: addresses review comments

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-12 22:37:30 +00:00
0e662eca2b refactor, chore(egen): Uses gcloud builds for building and pushing image to artifact registry, Removes boilerplate, heading fixes, and other corrections from template (#3074)
* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to notebook template

* <refactor,chore> refactored notebook according to template

* <refactor,chore> refactored notebook according to template

* fix for  docker repository creation in PR test environment

* <included IS_TESTING condition for docker repository

* lint fix

* Apply suggested edits from @kittyabs review

---------

Co-authored-by: SumanthKasula99 <sumanth.kasula@egen.ai>
2024-06-12 17:28:01 +00:00
Eric DongandGitHub d339690a4b Update README.md (#3088)
Update the overview with link to generative-ai repo
2024-06-12 17:08:54 +00:00
Mend RenovateandGitHub 134a6221e6 chore(deps): update dependency pyupgrade to v3.16.0 (#3066) 2024-06-11 19:14:25 +00:00
kittyabsandGitHub e79c4b5caa Update get_started_with_pytorch_rov.ipynb (#3034)
Update colab enterprise logo
2024-06-11 19:13:18 +00:00
Kathy YuandGitHub 535a6232d3 Remove legacy Llama 2 notebook. (#3083) 2024-06-11 19:11:11 +00:00
KCFindstrandGitHub f49e09698e Fix broken links in model garden notebooks (#3077)
* Fix broken links in notebooks

PiperOrigin-RevId: 641077957

* Revert llama3 deployment notebook change
2024-06-11 19:10:16 +00:00
Gary WeiandGitHub 792e3b16a5 Update the diffusers serving docker image to 20240605_1400_RC00 to resolve vulnerabilities. (#3079)
* Create a Gradio notebook for the new InstantId model.

* Add dreambooth finetune to the stable diffusion Gradio workshop notebook.

* Update the image generation Gradio notebook to support Dreambooth finetuning.

* linter update

* linter update

* minor fix to the instant-id notebook.

* Minor fix to the stable diffusion gradio notebook.

* Split the 'instant-id' deployment notebook prediction into two sections.

* add `deployment_source` to the notebook.

* Switch to `pytorch-diffusers-serve-opt` container to for diffusion lora serving.

* add the dreambooth_lora notebook.

* minor update.

* Parameterize the "show_debug_logs" to facilitate automatic test of the Gradio notebooks.

* Lint format.

* minor updates

* Delete the two deprecated SD1.5 and 2.1 notebooks, as they were no longer referenced on any model cards.

* Sync Colab notebooks between g3 and github.

* format changes

* format update.

* Improve the SDXL-dreambooth-lora finetune notebook CUJ.

* Update the diffusers serving docker image version to `20240605_1400_RC00` to resolve vulnerabilities.

* Add dreambooth-lora-sdxl task for SDXL base model in the dreambooth finetune Gradio notebook.

* Add dreambooth-lora-sdxl task for SDXL base model in the dreambooth finetune Gradio notebook.

* Add dreambooth-lora-sdxl task for SDXL base model in the dreambooth finetune Gradio notebook.
2024-06-11 19:09:30 +00:00
Kaushik KoiladaandGitHub fbcf3b4cc8 chore, refactor(Egen): restructure from template, remove boilerplate, and fix notebook (#3068)
* chore: updates copyright text, adds colab enterprise and for

* chore: removes boilerplate and changes colab authentication and get started section

* refactor: removes IS_TESTING and other test code

* refactor: removes IS_TESTING and other test code

* chore: clear all outputs and linter reformatting

* chore: removes code added for testing and other fixes

* chore: runs lint

* fix: fixes testing induced bug

* chore: review comments with header and copyright changes addressed
2024-06-11 18:27:36 +00:00
ac8f7ad196 chore, fix, feat(egen): follows template + fix docker run + adds clean up (#3072)
* chore, fix: restructures as per template, rewords some sentences, removes sudo in the option docker run

* feat: adds clean up step for local files

* chore: reverts lowercase to camelcase for artifact registry and other review comments

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-11 18:06:50 +00:00
Kaushik KoiladaandGitHub 4e21510eeb refactor, chore, fix (Egen) : removes boilerplate and fixes notebook pytorch_train_deploy_models_with_prebuilt_containers.ipynotebook (#3070)
* chore: updates the license

* chore: formats run in buttons and adds colab enterprise link

* chore: removes code font for names of products

* refactor: removes boilerplate and edits aiplatform initialization cell order

* fix: changes naming of app to deal with setup changing tar.gz file to canonical name and gsutil command not able to find it

* chore: linter run

* chore: edit all the future tense sentences

* chore: updates review comments
2024-06-11 18:01:11 +00:00
Rohith AllaandGitHub 04059ebadb refactor(egen): Refactored the Custom tabular bq managed dataset code as per template guidelines (#3053)
* refactor: Refactored custom-tabular-bq-managed-dataset code according to the template notebook

* refactor: Included markdown code in the beginning of the license cell to make it collapsible

* chore: linter test

* refactor, chore: Updated markdown according to the updated notebook template and performed linter test

* chore: Grouped all the imports present in the notebook

* chore: linter test

* refactor, chore(egen): REGION is replaced with LOCATION, linter test
2024-06-11 17:53:12 +00:00
Kaushik KoiladaandGitHub acb4fbb684 refactor, fix, chore(Egen): changes and refactors the get_started_vertex_training_xgboost.ipynb notebook (#3069)
* chore: updates license information, adds colab enterprise, and reformats run buttons

* chore: 'will' replaced appropriately

* chore: updates region and removes boilerplate

* fix: corrects the argument at pip install

* refactor: refactoring the cells for end to end functionality

* chore: changes verbiage in bucket creation

* chore: linter test done

* refactor: rearrage cell order for aiplatform initialization

* chore: review comments addressed

* chore: review comments addressed
2024-06-11 17:48:44 +00:00
Rohith AllaandGitHub 32934925f6 refactor(egen): Refactored custom batch prediction feature filter code according to template guidelines (#3052)
* refactor: Refactored custom_batch_prediction_feature_filter code according to the template notebook

* refactor: Updated markdown according to the updated notebook template and did linter test

* chore: Modified code according to updated template

* chore: linter test

* refactor, chore(egen): REGION is replaced with LOCATION, linter test
2024-06-11 17:40:16 +00:00
Rohith AllaandGitHub daab623cac refactor, chore: Updated markdown according to the updated notebook template and performed linter test (#3063) 2024-06-07 23:12:01 +00:00
Rohith AllaandGitHub dad35742de fix, refactor(egen): Replaced tf2-gpu.2-5 with tf2-gpu.2-6, Refactored the code as per template guidelines (#3064)
* refactor: Updated markdown according to the updated notebook template

* refactor, chore: Added colab enterprise logo url and performed  linter test

* refactor, chore: Reverted back the max trail count and parallel trail count values, performed linter test
2024-06-07 21:30:06 +00:00
Rohith AllaandGitHub f7b49ebb40 refactor(egen): Refactored the Autosxs check alignment against human preference data code as per template guidelines (#3062)
* refactor, chore: Updated markdown according to the updated notebook template and performed linter test

* Revert "refactor, chore: Updated markdown according to the updated notebook template and performed linter test"

This reverts commit c91fa09431.

* refactor, chore: Updated markdown according to the updated notebook template and performed linter test
2024-06-07 21:24:53 +00:00
nileshspringmlandGitHub 95ffea1915 refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3058)
* Did all the required changes in notebook

* Added below comments:
# @title Copyright & License (click to expand)

* Upadated the template of notebook

* Fixed issue based on the feedback given on PR
2024-06-07 21:20:07 +00:00
nileshspringmlandGitHub 7e6122a50d refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3046)
* refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template

* Added Below line in comments:
# @title Copyright & License (click to expand)

* Fixed issue in notebook based on feedback given by reviewer in PR

* Fixed issue based on feedback given on PR
2024-06-07 21:15:20 +00:00
7afdeb05f5 chore, fix(egen): corrections from template, guidelines, & fixes docker tag command (#3031)
* chore: replaces K80 with T4, Cloud ML with Vertex AI, REGION with LOCATION, restructures from template, & removes future tense

* fix: fixes the docker tag command for colab

* chore, fix: addresses the review comments, tf is pinned to 2.15.1 as the latest tf causes issues

* chore: replaces of with or

* chore: removes the collapsed license comment

* fix: removes the test env specific package update step and contracts the installations into one step to keep tf as 2.15

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-07 21:10:09 +00:00
2df29ec118 chore, fix(egen): restructuring from template, markdown corrections & dataset object fix (#3028)
* chore: restructures according to the template, removes future tense, removes IS_TESTING, unused imports

* chore, fix: addresses the review comments, converts npy array to list for running in py-3.9

* chore: adds --it's-- in the sentence

* chore: Getting started -->  Get started

* chore: removes the collapsed license comment

* fix: extracts list from numpy objects instead of Dataset object

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-07 20:58:54 +00:00
nileshspringmlandGitHub c2cd389201 refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3020)
* Completed code fixing for file prediction/pytorch_image_classification_with_prebuilt_serving_containers.ipynb. Please note that we need to execute file end to end.

* Added Colab Enterprice link and testead once again with different environment as previously it was throwing an expecption of  libraries

* Added below comment:
# @title Copyright & License (click to expand)

* Fixed issue in notebook based on feedback given by reviewer in PR

* Fixed issue based on feedback give on PR
2024-06-07 20:54:10 +00:00
nileshspringmlandGitHub b30a230d1d refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3019)
* Fix code of file prediction/get_started_with_raw_predict.ipynb

* Added link of Colab Enterprise

* Updated create bucket command because it requried those changes to execute in google colab notebook

* Removed IS_TESTING environment variable

* Added below comment in notebook:
# @title Copyright & License (click to expand)

* Fixed issue in notebook based on feedback given by reviewer in PR

* Fixed issue based on the feedback given on PR
2024-06-07 20:51:24 +00:00
Aiden010200andGitHub c5be708d45 Update func of computing resources recycling (#3043)
* Upload examples of kfp v2

* Upload run experiment example.

* Upload batch prediction job sample.

* Update recycling of computing resources

Recycling computing resources after predictions.

* Upload missing file

Add delete endpoint func to recycle resources.
2024-06-07 12:39:33 +00:00
Gary WeiandGitHub bc577d9f79 Improve the SDXL-dreambooth-lora finetune notebook CUJ. (#3055)
* Create a Gradio notebook for the new InstantId model.

* Add dreambooth finetune to the stable diffusion Gradio workshop notebook.

* Update the image generation Gradio notebook to support Dreambooth finetuning.

* linter update

* linter update

* minor fix to the instant-id notebook.

* Minor fix to the stable diffusion gradio notebook.

* Split the 'instant-id' deployment notebook prediction into two sections.

* add `deployment_source` to the notebook.

* Switch to `pytorch-diffusers-serve-opt` container to for diffusion lora serving.

* add the dreambooth_lora notebook.

* minor update.

* Parameterize the "show_debug_logs" to facilitate automatic test of the Gradio notebooks.

* Lint format.

* minor updates

* Delete the two deprecated SD1.5 and 2.1 notebooks, as they were no longer referenced on any model cards.

* Sync Colab notebooks between g3 and github.

* format changes

* format update.

* Improve the SDXL-dreambooth-lora finetune notebook CUJ.
2024-06-07 12:36:14 +00:00
KCFindstrandGitHub 1fcafd3364 Fix accelerator_type variable name case in quota checks (#3061)
PiperOrigin-RevId: 640634353
2024-06-07 12:34:21 +00:00
be31756dac refactor, chore(egen): Claude documentation link update, Text explanations for required cells and other corrections from template (#3027)
* <refactor> refactored notebook according to template

* <refactor> refactored notebook according to template

* <refactor> refactored notebook according to template

* <refactor>: refactored notebook according to new notebook template

* <refactor>: refactored notebook according to new notebook template

* <refactor>: refactored notebook according to new notebook template

---------

Co-authored-by: SumanthKasula99 <sumanth.kasula@egen.ai>
2024-06-06 23:03:34 +00:00
868e49224b refactor(egen): Removes boilerplate, heading fixes, and other corrections from template (#3026)
* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to notebook template

* <refactor>: refactored code according to new notebook template

* <refactor>: refactored code according to new notebook template

* <refactor>: refactored code according to new notebook template

* <refactor>: refactored notebook according to new notebook template

* <refactor>: refactored notebook according to new notebook template

---------

Co-authored-by: SumanthKasula99 <sumanth.kasula@egen.ai>
2024-06-06 23:01:07 +00:00
nileshspringmlandGitHub 32caaaa7d9 Updated notebook template (#3059)
* Updated notebook template based on google team comment on PR

* Removed unwated comment line from notebook
2024-06-06 13:12:57 +00:00
Rohith AllaandGitHub 521f58b7e6 refactor(egen): Refactored Text embedding api semantic search with scann code according to template guidelines (#3024)
* <refactor>: Refactored text_embedding_api_semantic_search_with_scann code according to the notebook template

* chore: linter test

* refactor: Modified colab enterprise logo url and made necessary changes according to the updated notebook template

* chore: linter test
2024-06-05 20:59:54 +00:00
030024cc51 refactor, chore, fix(egen): GPU usage fixes, template & markdown fixes (#3050)
* refactor, chore, fix: replaces K80 with T4, fixes to follow the template, fixes to utilize the defined accelerators while training

* chore: addresses the review comments

---------

Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-05 20:51:27 +00:00
Rohith AllaandGitHub dc38b9e0f9 refactor(egen): Refactored text embedding new api code according to the template guidelines (#3051)
* refactor: Refactored text_embedding_new_api code according to the template notebook

* refactor: Modified colab enterprise logo url and made necessary changes according to the updated notebook template
2024-06-05 20:12:57 +00:00
Amy WuandGitHub cd95c35cc2 remove data science package samples since they are deprecated (#3056) 2024-06-04 23:49:20 +00:00
4e04f7d166 Template vertex ai (#3054)
* Update build_model_experimentation_lineage_with_prebuild_code.ipynb

added colab enterprise logo and link

* Update build_model_experimentation_lineage_with_prebuild_code.ipynb

hope I fixed the JSON issue

* Update notebook_template.ipynb

- changed "Getting Started" to "Get started" to be in compliance with style guide
- added "for Python" to "Vertex AI SDK" to be in compliance with product guidelines

* fix: reset changes

---------

Co-authored-by: Katie Nguyen <21978337+katiemn@users.noreply.github.com>
2024-06-04 18:02:17 +00:00
bc14b3fa05 refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3045)
* <refactor>: refactored code according to notebook template

* <refactor> refactored notebook according to template

---------

Co-authored-by: SumanthKasula99 <sumanth.kasula@egen.ai>
2024-06-04 08:20:36 +00:00
db9c182716 refactor, chore(egen): Removes boilerplate, heading fixes, and other corrections from template (#3021)
* refactor: removes boilerplate code, future tenses, and fixes heading styles

* chore: replaces REGION with LOCATION to match the notebook template

* chore: markdown heading fixes according to guidelines

---------

Co-authored-by: rohith-egen <rohith.alla@egen.ai>
Co-authored-by: krishr2d2 <krishna.movva@egen.ai>
2024-06-03 23:48:13 +00:00
Rohith AllaandGitHub c240cf6a34 refactor(egen): Refactored the code as per template guidelines (#3015)
* refactored code according to template notebook

* refactored code according to template notebook
2024-06-03 23:46:29 +00:00
lee1premiumandGitHub 78d2d46f80 fix: Comment on Vertex AI workbench. (#3049)
* fix: Comment on Vertex AI workbench.

* fix: Comment on Vertex AI workbench.
2024-06-03 20:04:53 +00:00