Compare commits

...
Author SHA1 Message Date
hyper-param 5b5873872a fix: Fix json response parsing error 2026-01-20 21:59:16 +00:00
Sam-DecigaandGitHub 5afa83dd25 feat: MongoDB voyage-4 launch (#4419)
* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch

* feat: MongoDB voyage-4 launch
2026-01-16 16:26:05 -05:00
Sam-DecigaandGitHub 8cab85d6ad feat: MongoDB voyage-multimodal-3.5 launch (#4420)
* feat: MongoDB voyage-multimodal-3.5 launch

* feat: MongoDB voyage-multimodal-3.5 launch
2026-01-16 14:57:11 -05:00
Eric DongandGitHub a7f3940635 refactor: Update Github icon (#4418) 2026-01-13 13:36:20 -05:00
Mend RenovateandGitHub 5bb1a75a48 chore(deps): update dependency pyupgrade to v3.21.2 (#4359) 2026-01-13 18:26:35 +00:00
Damodar PanigrahiandGitHub 8fe4985aa8 feat: WeatherNext2 Initial Updates (#4417) 2026-01-13 13:24:48 -05:00
intentsolutions.ioandGitHub 996b6534d9 Add ADK inline source deployment tutorial for Agent Engine (#4393)
* Add ADK inline source deployment tutorial for Agent Engine

* fix: address Gemini review feedback

- Change model from gemini-2.0-flash to gemini-1.5-flash-001
- Improve exception handling with ZoneInfoNotFoundError

* fix: address Gemini code review feedback

- Use specific ZoneInfoNotFoundError exception instead of generic Exception
- Define REQUIREMENTS variable once and reuse to avoid duplication
- Keep generic Exception as fallback for unexpected errors

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* style: fix notebook formatting via official linter
2026-01-13 08:56:07 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1061ae5348 Bump urllib3 (#4416)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.6.0 to 2.6.3.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.6.0...2.6.3)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-12 20:01:22 +00:00
Eric DongandGitHub 4fac3a630f refactor: reformat notebook template (#4414)
* refactor: reformat notebook template

* Update Python version to 3.12

* Fix an import

* Update licience year
2026-01-12 13:15:30 -05:00
Eric DongandGitHub 1f2db2903c refactor: Updated Github icons (#4413) 2026-01-12 10:10:34 -05:00
Sam-DecigaandGitHub 912a52de70 feat:MongoDB Voyage 3.5-Lite (#4403)
* feat:MongoDB Voyage 3.5-Lite

* chore:apply linter

* Update voyage-3.5-lite.ipynb

Fixing MODEL_NAME
2026-01-06 21:19:17 -05:00
Sam-DecigaandGitHub 5e29090e86 Model Deprecation (#4409)
Haiku 3.5 Model Deprecation
2026-01-05 16:30:23 -05:00
Mend RenovateandGitHub cd8fcd1839 Update actions/checkout action to v6 (#4374) 2026-01-05 14:20:36 +00:00
Mend RenovateandGitHub e51075ec4b chore(deps): update dependency black to v25.12.0 (#4360) 2026-01-05 14:18:08 +00:00
Mend RenovateandGitHub 9709c0dddb chore(deps): update dependency isort to v7 (#4289) 2026-01-05 14:14:26 +00:00
Mend RenovateandGitHub a21ae41762 Update dependency python to 3.14 (#4398) 2026-01-05 14:13:11 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
ff5ff7b609 Bump urllib3 (#4386)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:12:43 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
23748f443e Bump urllib3 (#4387)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.5.0...2.6.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.6.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-01-05 14:12:18 +00:00
Vertex MG TeamandCopybara-Service fc6b2167de ComfyUI tutorial notebook
PiperOrigin-RevId: 851380881
2026-01-02 10:20:16 -08:00
e79a45358c Migrate gsutil usage to gcloud storage (#4299)
* Migrate gsutil usage to gcloud storage

* changes for 4299

* Apply automated linter fixes

* update

* remove model_garden changes

* revert to main

* revert model garden file

* Update model_garden_weather_prediction_on_vertex.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:37:14 -05:00
20d19fb11c Migrate gsutil usage to gcloud storage (#4331)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Changes for 4331

* Changes for 4331

* Removed changes model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:36:46 -05:00
9c9f7a6e2a Migrate gsutil usage to gcloud storage (#4334)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* changes for 4334

* fix linter issue\ for 4334

* manual changes

* Restore gcloud migration code

* Remove changes for model garden

* update

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:36:21 -05:00
ede41c2115 Migrate gsutil usage to gcloud storage (#4335)
* Migrate gsutil usage to gcloud storage

* remoed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-31 08:35:53 -05:00
ca53786c04 Migrate gsutil usage to gcloud storage (#4322)
* Migrate gsutil usage to gcloud storage

* Changes for 4322

* fix linter issue for 4322

* removed changes for model garden

* Update model_garden_pytorch_gemma_peft_finetuning_hf.ipynb

* Update model_garden_pytorch_gemma_peft_finetuning_hf.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:49:46 -05:00
419f8310c9 Migrate gsutil usage to gcloud storage (#4316)
* Migrate gsutil usage to gcloud storage

* Changes for 4316

* Changes for 4316

* fix linter issue for 4316

* removed changes for model garden

* removed changes for model garden

* Update model_garden_tfvision_image_classification.ipynb

* Update model_garden_tfvision_image_classification.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:49:33 -05:00
996b690e03 Migrate gsutil usage to gcloud storage (#4317)
* Migrate gsutil usage to gcloud storage

* Changes for 4317

* Changes for 4317

* fix linter issue for 4317

* removed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:48:30 +00:00
5b9d04d63a Migrate gsutil usage to gcloud storage (#4318)
* Migrate gsutil usage to gcloud storage

* Changes for 4318

* fix linter issue for 4318

* removed changes for model garden:

* Update model_garden_gemma2_deployment_on_vertex.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:47:49 -05:00
9ed3c2f83d Migrate gsutil usage to gcloud storage (#4320)
* Migrate gsutil usage to gcloud storage

* Changes for 4320

* fix linter issue for 4320

* removed chnages for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:47:17 -05:00
5fc93c8bdb Migrate gsutil usage to gcloud storage (#4328)
* Migrate gsutil usage to gcloud storage

* PR changes for 4328

* Fix Linter issue for 4328

* changes removed model garden

* Update model_garden_jax_fvlm.ipynb

* Update model_garden_jax_fvlm.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:46:52 -05:00
3cf46226e9 Migrate gsutil usage to gcloud storage (#4321)
* Migrate gsutil usage to gcloud storage

* Changes for 4321

* fix the linter issue for br 4321

* removed changes for model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:46:04 -05:00
439f6a0cae Migrate gsutil usage to gcloud storage (#4329)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Linter fiex the issues for 4329

* Revert "Linter fiex the issues for 4329"

This reverts commit a1ed9c6f59.

* Revert "Manual Changes"

This reverts commit 8ca9b56c5b.

* changes removed model garden

* Update model_garden_gemma2_finetuning_on_vertex.ipynb

* Update model_garden_pytorch_llama3_3_finetuning.ipynb

* Update model_garden_pytorch_llama3_3_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:45:30 -05:00
993898bb71 Migrate gsutil usage to gcloud storage (#4330)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Removed blank line and spaces Manual Changes

* fix the Linter issue for 4330

* Changes for model garden

* Update model_garden_pytorch_llama3_1_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:45:00 +00:00
9e9e639375 Migrate gsutil usage to gcloud storage (#4323)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Migrate gsutil usage to gcloud storage

* Manual Changes

* Fix linter issue for 4323

* removed changes for model garden

* Update model_garden_axolotl_gpt_oss_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:44:34 -05:00
633cf6a799 Migrate gsutil usage to gcloud storage (#4327)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Migrate gsutil usage to gcloud storage

* Changes for 4327

* fixing linting erro

* removed changes of model garden

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:44:05 -05:00
8b618bc455 Migrate gsutil usage to gcloud storage (#4326)
* Migrate gsutil usage to gcloud storage

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Revert "Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'"

This reverts commit 175eaa4fe8.

* Manual Changes-Updated the cell by replacing 'gsutil copy' with the correct 'gcloud storage cp'

* Changes for 4326

* Changes for 4326

* Linter fix issue for 4326

* removed model garden changes

* Update model_garden_movinet_action_recognition.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:43:31 -05:00
6bbe3bcfe0 Migrate gsutil usage to gcloud storage (#4336)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* removed model garden changes

* Update model_garden_llama3_1_finetuning_with_workbench.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:42:44 -05:00
Margubur RahmanandGitHub 814827ac19 Migrate gsutil usage to gcloud storage (#4332) 2025-12-25 10:42:12 -05:00
Margubur RahmanandGitHub 7a613785b9 Migrate gsutil usage to gcloud storage (#4333) 2025-12-25 15:41:38 +00:00
100243e90a Migrate gsutil usage to gcloud storage (#4338)
* Migrate gsutil usage to gcloud storage

* Manual Changes

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
2025-12-25 15:40:46 +00:00
6424515b03 Migrate gsutil usage to gcloud storage (#4339)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Fix: Updated gcloud storage command without formatting

* Manual Changes

* Manual Changes

* Revert "Manual Changes"

This reverts commit a7a7bda0f9.

* Manual Changes

* Revert "Manual Changes"

This reverts commit a7a7bda0f9.

* Manual Changes

* Revert "Manual Changes"

This reverts commit 71c777d5f1.

* Manual Changes

* Manual changes

* Changes for 4339

* Changes for 4339

* Changes for 4339

* Fix: Applied linter formatting and resolved style issues

* gcloud to gsutilchanges for  4339

* removed gsutil to gcloud migration

* Manual changes

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 15:40:00 +00:00
8d22b221b4 Migrate gsutil usage to gcloud storage (#4337)
* Migrate gsutil usage to gcloud storage

* Manual Changes

* Manual Changes

* Manual Changes

* Manual Changes

* Revert "Manual Changes"

This reverts commit 3ef23f1456.

* Manaul Changes

* Changes for 4337

* Fix: Resolved linter errors and formatted notebooks

* Revert "Fix: Resolved linter errors and formatted notebooks"

This reverts commit 7534760fe1.

* Revert "Changes for 4337"

This reverts commit f497568138.

* Changes for 4337

* gcloud to gsutil migration

* removed changes for model garden

* Update model_garden_axolotl_qwen3_finetuning.ipynb

* Update model_garden_axolotl_qwen3_finetuning.ipynb

---------

Co-authored-by: bhandarivijay <bhandarivijay@google.com>
Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-25 10:39:31 -05:00
2a8ad7cdbb Migrate gsutil usage to gcloud storage (#4319)
* Migrate gsutil usage to gcloud storage

* changes for 4319

* changes for 4319

* Apply automated linter fixes

* remove unused import

* removed model_garden changes

* Update model_garden_pytorch_mixtral_peft_tuning.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:58 -05:00
3239b301f2 Migrate gsutil usage to gcloud storage (#4315)
* Migrate gsutil usage to gcloud storage

* changes for 4315

* Apply automated linter fixes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:28 -05:00
acb10d14b8 Migrate gsutil usage to gcloud storage (#4314)
* Migrate gsutil usage to gcloud storage

* changes for 4314

* Apply automated linter fixes

* remove unused imports

* removed model_garden changes

* updates

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:41:00 -05:00
778d145970 Migrate gsutil usage to gcloud storage (#4313)
* Migrate gsutil usage to gcloud storage

* changes for 4313

* Apply automated linter fixes

* remove model_garden updates

* remove model_garden updates

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:40:34 -05:00
4d00356f4b Migrate gsutil usage to gcloud storage (#4312)
* Migrate gsutil usage to gcloud storage

* changes for 4312

* Apply automated linter fixes

* remove model_garden folder changes

* remove model_garden folder changes

* Update model_garden_pytorch_llama2_peft_hyperparameter_tuning.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:40:01 -05:00
0654305994 Migrate gsutil usage to gcloud storage (#4311)
* Migrate gsutil usage to gcloud storage

* changes for 4311

* Apply automated linter fixes

* removed unused imports

* remove model_garden changes

* Update model_garden_pytorch_zipnerf.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:39:27 -05:00
c53f392c5a Migrate gsutil usage to gcloud storage (#4310)
* Migrate gsutil usage to gcloud storage

* changes for 4310

* Apply automated linter fixes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:38:53 -05:00
f4b56e92ae Migrate gsutil usage to gcloud storage (#4308)
* Migrate gsutil usage to gcloud storage

* changes for 4308

* added = in command

* Apply automated linter fixes

* removed model_garden folder changes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:38:24 -05:00
Margubur RahmanGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gurusai-voletigemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
45ec1cf18a Migrate gsutil usage to gcloud storage (#4307)
* Migrate gsutil usage to gcloud storage

* changes for 4307

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Apply automated linter fixes

* Update sdk_vector_search_for_indexing.ipynb

* Update sdk_vector_search_for_indexing.ipynb

* update

* Update model_garden_pytorch_falcon_instruct_quantization.ipynb

* Update model_garden_pipeline_templates_t5x.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-24 14:37:45 +00:00
f6c8bcf937 Migrate gsutil usage to gcloud storage (#4301)
* Migrate gsutil usage to gcloud storage

* changes for 4301

* linter changes

* Revert "linter changes"

This reverts commit 6665133b2c.

* Apply automated linter fixes

* Update training-multi-class-classification-model-for-ads-targeting-usecase.ipynb

* Update training-multi-class-classification-model-for-ads-targeting-usecase.ipynb

* removed model_garden folder changes

* Update model_garden_mediapipe_object_detection.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-24 09:37:11 -05:00
Rayan DasoriyaandCopybara-Service 9e590d5a9f Update notebooks based on latest deployment options
PiperOrigin-RevId: 847555297
2025-12-21 19:32:54 -08:00
46e0ea4f1c Migrate gsutil usage to gcloud storage (#4309)
* Migrate gsutil usage to gcloud storage

* changes for 4309

* changes for 4309

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-20 11:34:42 -05:00
300fce6b9f Migrate gsutil usage to gcloud storage (#4305)
* Migrate gsutil usage to gcloud storage

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-20 11:34:10 -05:00
Vertex MG TeamandCopybara-Service 52e3066c38 No public description
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845978812
2025-12-19 10:42:55 -08:00
27ebf52198 Migrate gsutil usage to gcloud storage (#4296)
* Migrate gsutil usage to gcloud storage

* updates

* update

* Update llm_streaming_prediction.ipynb

* revert change

* removed model_garden folder changes

* removed model_garden folder changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:57:20 -05:00
ca7d4e153e Migrate gsutil usage to gcloud storage (#4297)
* Migrate gsutil usage to gcloud storage

* changes for 4297

* Apply automated linter fixes

* removed changes from model_garden

* removed changes from model_garden

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:56:49 -05:00
5b6c766629 Migrate gsutil usage to gcloud storage (#4298)
* Migrate gsutil usage to gcloud storage

* update

* linter changes for 4298

* Revert "linter changes for 4298"

This reverts commit c7a00a9710.

* Linter fixes

* update

* update

* Update distributed_hyperparameter_tuning.ipynb

* removed changes in model_garden folder

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 11:56:15 -05:00
Margubur RahmanandGitHub 5efa51206f Migrate gsutil usage to gcloud storage (#4324) 2025-12-19 15:11:28 +00:00
Margubur RahmanandGitHub e604a4d43e Migrate gsutil usage to gcloud storage (#4341) 2025-12-19 15:10:52 +00:00
23af5373ec Migrate gsutil usage to gcloud storage (#4295)
* Migrate gsutil usage to gcloud storage

* removed notes

* Apply automated linter fixes

* Revert "Apply automated linter fixes"

This reverts commit 3f6c4c0d36.

* linter changes

* update

* update

* update

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 10:09:05 -05:00
7577c0b1fc Migrate gsutil usage to gcloud storage (#4292)
* Migrate gsutil usage to gcloud storage

* Manual change

* update

* Update NotebookProcessors.py

* Update NotebookProcessors.py

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 15:07:50 +00:00
b648f9e73b Migrate gsutil usage to gcloud storage (#4306)
* Migrate gsutil usage to gcloud storage

* changes for 4306

* Apply automated linter fixes

* revert gsutil to gcloud

* Update model_garden_pytorch_stable_diffusion_custom.ipynb

* Update model_garden_pytorch_stable_diffusion_xl_lcm.ipynb

* Update model_garden_pytorch_deployed_model_agent_engine.ipynb

* Update model_garden_pytorch_blip_vqa.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-19 15:07:21 +00:00
Margubur RahmanandGitHub 966bbc49a7 Migrate gsutil usage to gcloud storage (#4340) 2025-12-18 13:50:35 -05:00
f5d341ae45 Migrate gsutil usage to gcloud storage (#4303)
* Migrate gsutil usage to gcloud storage

* changes for 4303

* Apply automated linter fixes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:49:38 -05:00
70770a50c7 Migrate gsutil usage to gcloud storage (#4302)
* Migrate gsutil usage to gcloud storage

* changes for 4302

* changes for 4302

* removed note

* linter changes

* Revert "linter changes"

This reverts commit a9544e8251.

* Apply automated linter fixes

* Update lightweight_functions_component_io_kfp.ipynb

* Update lightweight_functions_component_io_kfp.ipynb

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:48:39 -05:00
Matej AleksandrovandCopybara-Service f181c39cbf No public description
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845941350
2025-12-18 10:04:35 -08:00
Rayan DasoriyaandCopybara-Service a5637f87f2 Add notebook for T5Gemma 2 local inference
PiperOrigin-RevId: 846315479
2025-12-18 10:03:26 -08:00
0103299084 Migrate gsutil usage to gcloud storage (#4304)
* Migrate gsutil usage to gcloud storage

* changes for 4304

* Apply automated linter fixes

* remove unused import

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 14:00:23 +00:00
1867536d76 Migrate gsutil usage to gcloud storage (#4294)
* Migrate gsutil usage to gcloud storage

* updates

* removed notes added by agent

* removed note

* linter changes

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:59:10 +00:00
0edae683e7 Migrate gsutil usage to gcloud storage (#4293)
* Migrate gsutil usage to gcloud storage

* update

---------

Co-authored-by: gurusai-voleti <gvoleti@google.com>
2025-12-18 13:58:03 +00:00
Matej AleksandrovandCopybara-Service 8471b5cb6f No public description
PiperOrigin-RevId: 845941350
2025-12-17 15:25:49 -08:00
Aaron DietzandGitHub 87f540ac53 Update notebook_template_review.py (#4405)
Minor change to help us update references to "custom training" to specify "serverless training"
2025-12-17 22:08:00 +00:00
Sam-DecigaandGitHub babeba9f02 feat:NVIDIA Nemotron Nano v2 12B VL - 2025-12-TBD (#4391)
* feat:NVIDIA Nemotron Nano v2 12B VL - 2025-12-TBD

* refactor:Reformat Notebook

* refactor:Reformat Notebook
2025-12-17 18:49:58 +00:00
Vertex MG TeamandCopybara-Service 85c649dd26 Add Llama 3.3 TPU7x deployment notebook.
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 845481841
2025-12-16 16:50:45 -08:00
Vertex MG TeamandCopybara-Service b7135ae1f0 Add Llama 3.3 TPU7x deployment notebook.
PiperOrigin-RevId: 845481841
2025-12-16 16:38:07 -08:00
Vertex MG TeamandCopybara-Service 2f5119a266 Allows the user to select spot VM for deployment
PiperOrigin-RevId: 845064375
2025-12-15 21:23:11 -08:00
Vertex MG TeamandCopybara-Service 23e64ca76f fix: Update TimesFM 2.0 deployment notebook to use GCS path as MODEL_ID
PiperOrigin-RevId: 844826585
2025-12-15 10:25:10 -08:00
gurusai-voletiandGitHub 0ba5a62cc9 Fix ci workflow to use python 3.13 to avoid linter issues (#4397)
* update

* use python 3.13
2025-12-15 13:54:49 +00:00
Damodar PanigrahiandGitHub 0be2c6fd0c feat: authenticate using sa (#4394) 2025-12-12 15:38:25 -05:00
Vertex MG TeamandCopybara-Service cef4928c49 Allows the user to select spot VM for deployment
PiperOrigin-RevId: 843098639
2025-12-11 01:03:35 -08:00
Vertex MG TeamandCopybara-Service 9bb8107110 Add notebook for using Deepseek 3.2 model on Vertex AI.
PiperOrigin-RevId: 842763792
2025-12-10 09:42:39 -08:00
Vertex MG TeamandCopybara-Service b075990d88 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 842339077
2025-12-09 12:04:54 -08:00
Ravi DalalGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
6a83c4c695 Updated notebook comment for custom vllm container image (#4385)
* updated comment for custom vllm container image

* Update notebooks/official/prediction/vertexai_serving_vllm/vertexai_serving_vllm_cpu_llama3_2_3B.ipynb

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-05 20:26:32 +00:00
Damodar PanigrahiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
820c0f8db4 Update Use Case description and API change to accept GCP auth token (#4384)
* pass auth_token in create_http_client

* lint on the notebook

* feat:Removed the last update date

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/cloudai_alphagenome_vai_quickstart.ipynb

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-05 18:30:23 +00:00
Damodar PanigrahiGitHubgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
4ab197a4ba AlphaGenome GCP API with quickstart.ipynb and README.md (#4378)
* AlphaGenome GCP API  with quickstart.ipynb and README.md

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update notebooks/community/alphagenome/README.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update cloudai_alphagenome_vai_quickstart.ipynb

lint errors

* Update cloudai_alphagenome_vai_quickstart.ipynb

lint errors

* lint errors

* lint errors

* lint import order

* lint errors

* lint import order

* lint import

* Update cloudai_alphagenome_vai_quickstart.ipynb format

* Update cloudai_alphagenome_vai_quickstart.ipynb remove hardcoded url

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-11-27 18:56:07 +00:00
Damodar PanigrahiandGitHub 2a5877fbd1 Update CODEOWNERS (#4380)
* Update CODEOWNERS

* Update CODEOWNERS
2025-11-27 18:30:33 +00:00
Vertex MG TeamandCopybara-Service 090e1d9fee Add dynamic model loading/unloading and instructions to model co-hosting notebook.
PiperOrigin-RevId: 837273935
2025-11-26 15:17:30 -08:00
Eric DongandGitHub 2e049d4830 feat: Add new supported model for Claude (#4377)
* feat: Add new supported model for Claude

* Fix elif

* Remove unused endpoint
2025-11-24 16:56:19 -05:00
Vertex MG TeamandCopybara-Service e3320d2126 Update vLLM docker URI in model co-hosting notebook.
PiperOrigin-RevId: 836250727
2025-11-24 09:14:59 -08:00
Vertex MG TeamandCopybara-Service 5fc0e03ca3 Use separate regions for training, evaluation, and deployment in Llama 3.1 finetuning notebook
PiperOrigin-RevId: 835030738
2025-11-20 20:31:36 -08:00
Vertex MG TeamandCopybara-Service ff2a16237d Minor typo fixes and prints
PiperOrigin-RevId: 834789027
2025-11-20 09:15:31 -08:00
Vertex MG TeamandCopybara-Service d26f081642 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 833620797
2025-11-17 20:35:49 -08:00
Vertex MG TeamandCopybara-Service 8a0a39176c Some minor updates and refactoring
PiperOrigin-RevId: 832558814
2025-11-14 20:22:08 -08:00
Vertex MG TeamandCopybara-Service 646532ea69 Fixed the deployment quota check and modified the documentation.
PiperOrigin-RevId: 832174331
2025-11-13 23:14:25 -08:00
Vertex MG TeamandCopybara-Service 9e96a3da67 MiniMax-M2 deployment notebook
PiperOrigin-RevId: 831880572
2025-11-13 08:58:34 -08:00
Vertex MG TeamandCopybara-Service db34e1fbd5 Add multi-model benchmark utility and benchmark results to model co-hosting tutorial notebook.
PiperOrigin-RevId: 831592927
2025-11-12 16:54:24 -08:00
Sam-DecigaandGitHub 82308acbac Update anthropic_claude_3_intro.ipynb - Sonnet 3.7 Deprecation (#4364)
Given information above. Approved.
2025-11-12 12:32:57 -05:00
Vertex MG TeamandCopybara-Service ee0ba75d1e Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 830644861
2025-11-10 16:31:45 -08:00
Vertex MG TeamandCopybara-Service d61aedc721 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 829629618
2025-11-07 17:08:47 -08:00
Vertex MG TeamandCopybara-Service 19f7f94af5 DeepSeek-OCR deployment notebook
PiperOrigin-RevId: 827772392
2025-11-03 21:08:50 -08:00
Vertex MG TeamandCopybara-Service 4e5ce9b226 Add single-model multi-replica & multi-model model co-hosting tutorial notebook.
PiperOrigin-RevId: 826614269
2025-10-31 13:46:14 -07:00
Vertex MG TeamandCopybara-Service 99938244f4 Add DWS to the 8B model in the Eval section
PiperOrigin-RevId: 825943414
2025-10-30 02:42:42 -07:00
Vertex MG TeamandCopybara-Service 0cc7be4a6a Added remote sensing deployment notebook
PiperOrigin-RevId: 825002026
2025-10-28 06:13:52 -07:00
Vertex MG TeamandCopybara-Service b6bde41850 Deepseek deployment v3_2 notebook
PiperOrigin-RevId: 824868883
2025-10-27 23:39:38 -07:00
Vertex MG TeamandCopybara-Service 52444a0933 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 824690270
2025-10-27 14:57:44 -07:00
Vertex MG TeamandCopybara-Service bab9c398fd Add new variants to qwen3-vl
PiperOrigin-RevId: 823383168
2025-10-24 00:02:05 -07:00
Vertex MG TeamandCopybara-Service 447affcc93 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 823276006
2025-10-23 18:49:58 -07:00
Bhaskar GoyalandGitHub 6132c37be0 Initiate Deprecation for Mistral Large (24.11) and Codestral (25.01) (#4347) 2025-10-23 16:38:09 +00:00
Vertex MG TeamandCopybara-Service 8d7f59aeec Update vLLM TPU deployment container image URI.
PiperOrigin-RevId: 822769821
2025-10-22 15:49:05 -07:00
Vertex MG TeamandCopybara-Service bd327ad424 Qwen3-VL deployment notebook
PiperOrigin-RevId: 822441900
2025-10-21 23:36:55 -07:00
Vertex MG TeamandCopybara-Service 3b1fbdb382 Update image in text+image chat completions requests in MG notebooks.
PiperOrigin-RevId: 821912015
2025-10-20 19:56:47 -07:00
Vertex MG TeamandCopybara-Service 954043a729 No public description
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 821735608
2025-10-20 12:10:35 -07:00
Vertex MG TeamandCopybara-Service 16ef9ee80e Add notebook for deploying GPT OSS models on G4 (RTX Pro 6000).
PiperOrigin-RevId: 821735608
2025-10-20 11:42:04 -07:00
Bhaskar GoyalandGitHub 79301b4a4d <feature> - Add Codestral 2 Model (#4291) 2025-10-16 22:05:58 +00:00
Vertex MG TeamandCopybara-Service 0b38d02e6f Add vLLM TPU deployment notebook for qwen3
PiperOrigin-RevId: 820273514
2025-10-16 09:41:13 -07:00
kthytangandGitHub c52ff25ba4 Haiku 4.5 update to anthropic_claude_intro.ipynb (#4300)
* Haiku 4.5 update to anthropic_claude_intro.ipynb

* Update anthropic_claude_intro.ipynb
2025-10-15 20:53:05 +00:00
Vertex MG TeamandCopybara-Service 424400bace Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 819784960
2025-10-15 09:16:29 -07:00
Mend RenovateandGitHub b1dfac2043 chore(deps): update actions/setup-python action to v6 (#4245) 2025-10-10 14:32:20 +00:00
Mend RenovateandGitHub b81ffcddab chore(deps): update python docker tag to v3.14 (#4284) 2025-10-10 14:29:19 +00:00
Mend RenovateandGitHub 21d8f144aa chore(deps): update dependency pyupgrade to v3.21.0 (#4287) 2025-10-10 14:29:11 +00:00
Vertex MG TeamandCopybara-Service 065a674305 Minor fixes for ollama deployment notebook
PiperOrigin-RevId: 817513895
2025-10-10 00:21:23 -07:00
haomengchaoandGitHub 571d498d08 feat: add notebook for VirtueAI model in Model Garden (#4280)
* feat: add virtueai's notebook for Model Garden

* feat: add virtueai's notebook for Model Garden with fixes

* feat: fix endpoint place holder to pass the test

* feat: fix endpoint place holder to pass the test

* feat: fix endpoint place holder to pass the test

* feat: fix a typo
2025-10-09 22:15:24 +00:00
Bhaskar GoyalandGitHub e936882123 <feature>: Medium 3 launch (#4279) 2025-10-09 16:42:11 +00:00
Vertex MG TeamandCopybara-Service f754f99052 Increase the dws max_wait_duration to 90 minutes
PiperOrigin-RevId: 817042685
2025-10-09 00:11:15 -07:00
Vertex MG TeamandCopybara-Service aa5523a5e9 Delete Gemma 3 peft finetuning notebooks.
PiperOrigin-RevId: 816744576
2025-10-08 09:39:54 -07:00
Vertex MG TeamandCopybara-Service 81393ede1a Weekly update the hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 815926264
2025-10-06 16:17:53 -07:00
Vertex MG TeamandCopybara-Service 5a1c0222da Qwen Image Deployment Notebook
PiperOrigin-RevId: 815781924
2025-10-06 10:19:30 -07:00
Vertex MG TeamandCopybara-Service 1f9326bd56 Weekly update the vllm container version.
PiperOrigin-RevId: 814830089
2025-10-03 14:24:00 -07:00
Vertex MG TeamandCopybara-Service a94cae2e79 Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 813495515
2025-09-30 17:28:38 -07:00
kthytangandGitHub cb861713c8 Update anthropic_claude_intro.ipynb (#4277) 2025-09-29 17:17:00 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
3a55087789 Bump urllib3 (#4257)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.4.0 to 2.5.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.4.0...2.5.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.5.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-29 16:59:52 +00:00
Vertex MG TeamandCopybara-Service d359b21f3e Weekly update the vllm/hf-tei/hf-inference-toolkit containers.
PiperOrigin-RevId: 811502904
2025-09-25 14:30:34 -07:00
Vertex MG TeamandCopybara-Service c7d4123b25 Print project and region information
PiperOrigin-RevId: 810501296
2025-09-23 10:54:22 -07:00
Vertex MG TeamandCopybara-Service 07a8bb2d0c Migrate Phi-4 notebook to use Model Garden SDK
PiperOrigin-RevId: 810268843
2025-09-22 20:52:05 -07:00
Vertex MG TeamandCopybara-Service f6b6f365b6 Migrate Ollama deploy notebook to use Model Garden SDK
PiperOrigin-RevId: 810150526
2025-09-22 14:13:49 -07:00
Vertex MG TeamandCopybara-Service fec825f9e5 Use dictionary for the deletion of multiple endpoints
PiperOrigin-RevId: 810066401
2025-09-22 10:26:45 -07:00
Vertex MG TeamandCopybara-Service 1e9bf72097 Blip2 notebook refactoring
PiperOrigin-RevId: 810050386
2025-09-22 09:47:46 -07:00
Vertex MG TeamandCopybara-Service 3fa1cf99ec Make 4b as the default model_version in the Gemma3 notebook
PiperOrigin-RevId: 809821706
2025-09-21 19:09:19 -07:00
Vertex MG TeamandCopybara-Service 2bcaf8abde Allow the user to enter Region
PiperOrigin-RevId: 808435945
2025-09-18 00:17:02 -07:00
Vertex MG TeamandCopybara-Service 759495a1f8 Migrate LaMa notebook to use Model Garden SDK
PiperOrigin-RevId: 808001560
2025-09-16 23:19:01 -07:00
Vertex MG TeamandCopybara-Service 9925e62c4b feat: Refactor to use deploy SDK.
PiperOrigin-RevId: 807803299
2025-09-16 12:35:20 -07:00
Vertex MG TeamandCopybara-Service 40ade71b35 Weekly update the vllm serving container version to 20250911_0916_RC01.
PiperOrigin-RevId: 807413330
2025-09-15 15:45:51 -07:00
Vertex MG TeamandCopybara-Service 65173071a5 Weekly update the serving container version for hf-inference-toolkit and hf-tei containers.
PiperOrigin-RevId: 807413263
2025-09-15 15:44:23 -07:00
Vertex MG TeamandCopybara-Service a514bb51c2 Allow the user to enter Region
PiperOrigin-RevId: 807182772
2025-09-15 04:21:01 -07:00
Rayan DasoriyaandCopybara-Service a6dc1b0f6d Fix notebook issues
PiperOrigin-RevId: 806407972
2025-09-12 13:33:39 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
75236998c7 Bump torch (#4239)
Bumps [torch](https://github.com/pytorch/pytorch) from 2.2.0 to 2.8.0.
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](https://github.com/pytorch/pytorch/compare/v2.2.0...v2.8.0)

---
updated-dependencies:
- dependency-name: torch
  dependency-version: 2.8.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-09-10 23:33:41 +00:00
MarkandGitHub 9c8a7808bf chore: Replace 'prediction' with 'inference' per urgent rebranding request (#4231) 2025-09-10 23:32:32 +00:00
Holt SkinnerandGitHub 86ce1576d2 Delete notebooks/official/model_evaluation/automl_video_classification_model_evaluation.ipynb (#4256) 2025-09-10 23:32:08 +00:00
Holt Skinner c2b743bfeb Removed Deprecated notebooks 2025-09-10 18:31:40 -05:00
Vertex MG TeamandCopybara-Service b04575d746 Fix axolotl gcs output path.
PiperOrigin-RevId: 805417425
2025-09-10 10:26:55 -07:00
Vertex MG TeamandCopybara-Service b90d16885e Weekly update the serving container version for hf-inference-toolkit and hf-tei containers.
PiperOrigin-RevId: 805081873
2025-09-09 15:12:41 -07:00
Holt SkinnerandGitHub 828f1a26fa Delete notebooks/official/pipelines/google_cloud_pipeline_components_automl_text.ipynb (#4254)
b/442906902
2025-09-09 20:43:03 +00:00
Holt SkinnerandGitHub 915e5edf0a chore: Remove AutoML Notebooks for deprecated Text and Video features (#4253)
* chore: Remove AutoML Notebooks for deprecated Text and Video features

* Remove remaining Text/Video samples
2025-09-09 16:49:06 +00:00
Vertex MG TeamandCopybara-Service 5d4c1be285 Migrate llava notebook to use Model Garden SDK
PiperOrigin-RevId: 804765256
2025-09-09 00:16:12 -07:00
Vertex MG TeamandCopybara-Service 275bfb8f69 Weekly update the vllm serving container version to 20250905_0916_RC01.
PiperOrigin-RevId: 804514266
2025-09-08 11:21:48 -07:00
Vertex MG TeamandCopybara-Service 93cb0ca3d9 Migrate blip image captioning notebook to use Model Garden SDK
PiperOrigin-RevId: 803091009
2025-09-04 10:49:28 -07:00
Vertex MG TeamandCopybara-Service 41d60d052d Add EmbeddingGemma local inference notebook.
PiperOrigin-RevId: 803038284
2025-09-04 08:30:34 -07:00
Vertex MG TeamandCopybara-Service d872cdcf7e Migrate Paligemma2 notebook to use Model Garden SDK
PiperOrigin-RevId: 802861534
2025-09-03 22:35:04 -07:00
Vertex MG TeamandCopybara-Service ada5e4a854 chore: fix google-auth and requests package version.
PiperOrigin-RevId: 802270767
2025-09-02 13:33:33 -07:00
Vertex MG TeamandCopybara-Service ecb32b099d Migrate Stable Diffusion Upscaler notebook to use Model Garden SDK
PiperOrigin-RevId: 801824204
2025-09-01 08:55:58 -07:00
Vertex MG TeamandCopybara-Service f8d09e8e9b Migrate Stable Diffusion XL Lightning notebook to use Model Garden SDK
PiperOrigin-RevId: 801782973
2025-09-01 06:06:41 -07:00
Vertex MG TeamandCopybara-Service 007df88fba Migrate Llama4 notebook to use Model Garden SDK
PiperOrigin-RevId: 801662862
2025-08-31 22:27:06 -07:00
Vertex MG TeamandCopybara-Service c70f3ef9a3 Weekly update vllm/hf-tei/hf-inference-toolkit container image versions.
PiperOrigin-RevId: 801034279
2025-08-29 14:39:29 -07:00
Vertex MG TeamandCopybara-Service 505e101452 Bug fix for Wan2.2
PiperOrigin-RevId: 800419261
2025-08-28 05:11:01 -07:00
Vertex MG TeamandCopybara-Service b77d51b58b Migrate QWEN3 notebook to use Model Garden SDK
PiperOrigin-RevId: 800346334
2025-08-28 00:57:59 -07:00
Rayan DasoriyaandCopybara-Service deaa1ccf2b Update quota check to use gcloud beta quotas info describe command
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 800261134
2025-08-27 19:19:54 -07:00
Vertex MG TeamandCopybara-Service f6e38860aa Update H100/H200 region recommendations in DeepSeek deployment notebook.
PiperOrigin-RevId: 799800908
2025-08-26 18:29:57 -07:00
Vertex MG TeamandCopybara-Service 4937e382b1 Bug fix for Wan2.1
PiperOrigin-RevId: 799116301
2025-08-25 07:40:20 -07:00
Vertex MG TeamandCopybara-Service 0fe2770947 Weekly update the vLLM container image version.
PiperOrigin-RevId: 798391915
2025-08-22 16:59:25 -07:00
Vertex MG TeamandCopybara-Service 80d7ee67d6 Weekly update the container version for hf-inference-toolkit and hf-tei.
PiperOrigin-RevId: 797920954
2025-08-21 14:45:09 -07:00
Vertex MG TeamandCopybara-Service f17e2d6c8d Migrate gpt-oss deploy notebook to use Model Garden SDK
PiperOrigin-RevId: 797824747
2025-08-21 10:40:45 -07:00
Vertex MG TeamandCopybara-Service ae1cd0ed08 Migrate llama3.3 deployment notebook to use Model Garden SDK
PiperOrigin-RevId: 797622523
2025-08-20 23:34:08 -07:00
denisj3030andGitHub a68b491edc dep35 (#4227) 2025-08-20 20:02:14 +00:00
Vertex MG TeamandCopybara-Service d39bed012a Migrate Controlnet notebook to use Model Garden SDK
PiperOrigin-RevId: 797349160
2025-08-20 09:37:09 -07:00
Vertex MG TeamandCopybara-Service f8c93de1c4 Migrate BLIP2 deploy notebook to use Model Garden SDK
PiperOrigin-RevId: 797131322
2025-08-19 20:34:06 -07:00
Vertex MG TeamandCopybara-Service 0b5d97aa3a use A100 machine by default for Llama3.1 fast deployment
PiperOrigin-RevId: 796788128
2025-08-19 02:51:44 -07:00
Vertex MG TeamandCopybara-Service 87f0785ed1 Migrate Segment Anything Model(SAM) notebook to use Model Garden SDK
PiperOrigin-RevId: 796693389
2025-08-18 20:55:11 -07:00
Vertex MG TeamandCopybara-Service 09c8ea616b Weekly update the vLLM container image version.
PiperOrigin-RevId: 796581324
2025-08-18 14:40:14 -07:00
Vertex MG TeamandCopybara-Service 6c11241cf7 Migrate BLIP2 VQA(Visual Question Answering) notebook to use Model Garden SDK
PiperOrigin-RevId: 796316856
2025-08-18 01:35:17 -07:00
Vertex MG TeamandCopybara-Service 9b7dc2e4fd Migrate BLIP2 VQA(Visual Question Answering) notebook to use Model Garden SDK
PiperOrigin-RevId: 795738195
2025-08-15 22:43:11 -07:00
Vertex MG TeamandCopybara-Service 9b3d43b1a6 Use correct notebook_util.
PiperOrigin-RevId: 794874566
2025-08-13 22:18:11 -07:00
Dustin LuongandCopybara-Service 57ee4e2eab Update Qwen3 deployment notebook with new variants and corrected model names.
PiperOrigin-RevId: 794407429
2025-08-12 22:41:45 -07:00
Dustin LuongandCopybara-Service 4d736d7992 Use correct notebook_util.
PiperOrigin-RevId: 794396544
2025-08-12 22:00:22 -07:00
Dustin LuongandCopybara-Service 69e92a650d Add gpt-oss-20b finetuning with lora on Vertex notebook.
PiperOrigin-RevId: 794317611
2025-08-12 16:57:36 -07:00
Vertex MG TeamandCopybara-Service 4ed979eec4 Migrate Stable Diffusion XL 1.0 notebook to use Model Garden SDK
PiperOrigin-RevId: 793954002
2025-08-11 23:22:40 -07:00
Vertex MG TeamandCopybara-Service 0344de8090 Wan Deployment Notebook
PiperOrigin-RevId: 793839574
2025-08-11 16:30:15 -07:00
Vertex MG TeamandCopybara-Service 83eca0f0cc Update the vLLM container image version.
PiperOrigin-RevId: 793738684
2025-08-11 11:47:34 -07:00
Vertex MG TeamandCopybara-Service 4619b272fa Migrate QwQ notebook to use Model Garden SDK.
PiperOrigin-RevId: 793694751
2025-08-11 10:05:17 -07:00
Vertex MG TeamandCopybara-Service 091fa33c01 Migrate Gemma3 notebook to use Model Garden SDK.
PiperOrigin-RevId: 793692575
2025-08-11 10:01:02 -07:00
Rayan DasoriyaandCopybara-Service 42a05c2b35 No public description
PiperOrigin-RevId: 793644258
2025-08-11 07:50:06 -07:00
Vertex MG TeamandCopybara-Service 8ac32fa42f Migrate Gemma3n notebook to use Model Garden SDK
PiperOrigin-RevId: 793583308
2025-08-11 04:09:41 -07:00
Rayan DasoriyaandCopybara-Service 0303057f11 Update the common util location in the notebooks
PiperOrigin-RevId: 793425712
2025-08-10 18:26:31 -07:00
Ravi DalalandGitHub 7ae13b346a added notebooks and dockerfiles for serving open models on vertexai using vllm custom containers (#4148)
* added notebooks and dockerfiles for serving open models on vertexai using vllm customer containers

* updated official codeowners

* fixed linting errors

* fixed linting errors

* moved notebooks

* ran linter

* fixed param type

* added some formatting

* added autoscaling configuration to model deployment

* fixed a heading

* moved notebooks and docker folder under prediction

* updated notebook repo paths

* switched to raw_predict to avoid code changes and rebuild

* removed linting errors

* fixed readme lint error

* fixed links

* removed dedicated_endpoint_enabled

* updated workdir path

* fixed cell type

* added license

* fixed linting error

* using cloud build container image build

* fixed linting issues

* fixes

* cloudbuild yaml

* fixed gemini review comments

* fixed linting errors

* fixed tpu_count type

* handled invalid device type

* optimized dockerfile run command

* updated dockerfile

* addressed review comments

* fixed linting errors

* added license to cloudbuild and dockerfile

* optimized image build code

* fixed image_name variable
2025-08-08 18:01:03 +00:00
Rayan DasoriyaandCopybara-Service 44390cbd99 Add a no-op message to check_quota if it fails.
MG_DOCKER_CODES_PIPER_ORIGIN_REV_ID: 792654121
2025-08-08 09:32:08 -07:00
Vertex MG TeamandCopybara-Service 58dab0b1bb E5 notebook
PiperOrigin-RevId: 791999324
2025-08-06 22:48:50 -07:00
Vertex MG TeamandCopybara-Service 8d36834fcd Add GPT OSS models deployment notebook.
PiperOrigin-RevId: 791864273
2025-08-06 15:17:54 -07:00
Dustin LuongandCopybara-Service 31ca3e36f4 Add Qwen3-30B-A3B instruct and thinking 2507 variants to notebook.
PiperOrigin-RevId: 791746627
2025-08-06 10:27:39 -07:00
Vertex MG TeamandCopybara-Service a72d7dc49f Fix custom dataset input for axolotl notebooks.
PiperOrigin-RevId: 791627090
2025-08-06 04:16:45 -07:00
denisj3030andGitHub 9efbd48233 cl41 (#4193) 2025-08-05 17:02:14 +00:00
Vertex MG TeamandCopybara-Service 166f0f8ce7 Update the vLLM container image version.
PiperOrigin-RevId: 790903846
2025-08-04 14:56:59 -07:00
Vertex MG TeamandCopybara-Service bd9f9675cf Add us-south1 region
PiperOrigin-RevId: 790762290
2025-08-04 08:35:53 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
42ec0e6ad9 Bump urllib3 (#4173)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.7 to 2.5.0.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.0.7...2.5.0)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-version: 2.5.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-01 14:29:14 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
fde3d98a2a Bump requests (#4174)
Bumps [requests](https://github.com/psf/requests) from 2.32.3 to 2.32.4.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.32.3...v2.32.4)

---
updated-dependencies:
- dependency-name: requests
  dependency-version: 2.32.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-01 14:28:53 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
b83f869a44 Bump protobuf (#4175)
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.20.3 to 4.25.8.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.20.3...v4.25.8)

---
updated-dependencies:
- dependency-name: protobuf
  dependency-version: 4.25.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-08-01 14:28:28 +00:00
Vertex MG TeamandCopybara-Service b918768776 add batch prediction
PiperOrigin-RevId: 789643160
2025-08-01 00:21:30 -07:00
Eric DongandGitHub a22aceb10c refactor: Batch update icon links for notebooks/community (#4188) 2025-07-31 15:41:21 +00:00
Dustin LuongandCopybara-Service c8b53a614e Fix text in TEI notebook to change instances of nomic-ai/nomic-embed-text-v1 to Qwen/Qwen3-Embedding-8B.
PiperOrigin-RevId: 789204281
2025-07-30 23:26:39 -07:00
Eric DongandGitHub 7be7d5be44 refactor: Batch update icon links for notebooks/official (#4187) 2025-07-31 00:08:25 +00:00
Eric DongandGitHub 02a030d6f1 fix: Update the icon links (#4186) 2025-07-30 20:37:13 +00:00
Dustin LuongandCopybara-Service fc8e9e4483 Set Qwen/Qwen3-Embedding-8B as example model in TEI notebook.
PiperOrigin-RevId: 788960826
2025-07-30 10:48:47 -07:00
Eliot LaidlawandGitHub fa019e051a Add CSM intro notebook (#4178)
* notebook

* linting

* add codeowner

* Add cleanup

* fixes

* Update copyright year
2025-07-30 16:36:51 +00:00
Dustin LuongandCopybara-Service 01d513eea3 Update the deployment notebook for Qwen3 models to include Qwen3-235B-Thinking-2507 and Qwen3-235B-Thinking-2507-FP8.
PiperOrigin-RevId: 788636454
2025-07-29 15:45:43 -07:00
Vertex MG TeamandCopybara-Service 928822cc0c Remove 8-bit mode for Qwen 2.5 finetuning notebook.
PiperOrigin-RevId: 788499530
2025-07-29 10:02:42 -07:00
Vertex MG TeamandCopybara-Service 1dfb4091ea notebook reformatting
PiperOrigin-RevId: 788430948
2025-07-29 06:34:26 -07:00
Dustin LuongandCopybara-Service ffb5ea7e88 Update HF TEI serving image to use new, FEDRamp compliant container.
PiperOrigin-RevId: 788203719
2025-07-28 16:36:17 -07:00
Vertex MG TeamandCopybara-Service dd2028d76c add batch prediction
PiperOrigin-RevId: 787943651
2025-07-28 04:02:27 -07:00
Vertex MG TeamandCopybara-Service 70daf2e605 Add Qwen3-Coder notebook
PiperOrigin-RevId: 787286540
2025-07-25 16:57:44 -07:00
654 changed files with 46912 additions and 33357 deletions
@@ -365,7 +365,7 @@ def process_and_execute_notebook(
# Use gcloud to get tail
try:
result.error_message = subprocess.check_output(
["gsutil", "cat", "-r", "-1000", log_file_uri], encoding="UTF-8"
["gcloud", "storage", "cat", "--range", "-1000", log_file_uri], encoding="UTF-8"
)
except Exception as error:
result.error_message = str(error)
+2 -2
View File
@@ -56,8 +56,8 @@ def execute_notebook(
print("\n=== DOWNLOAD EXECUTED NOTEBOOK ===\n")
print(f"Please debug the executed notebook by downloading the executed notebook:")
print("Option 1. Using gsutil. Run the following command in your terminal.")
print(f'\tgsutil cp "{output_file_or_uri}" .')
print("Option 1. Using gcloud storage. Run the following command in your terminal.")
print(f'\tgcloud storage cp "{output_file_or_uri}" .')
print("Option 2. Using this link.")
print(f"\thttps://storage.googleapis.com/{output_file_or_uri[5:]}")
+1 -1
View File
@@ -108,7 +108,7 @@ class VertexAIInstallProprocessor(Preprocessor):
if "google-cloud-aiplatform" not in content:
return content
return (
f"gsutil cp {self.vertex_ai_wheel} google-cloud-aiplatform.whl\n" +
f"gcloud storage cp {self.vertex_ai_wheel} google-cloud-aiplatform.whl\n" +
content.replace("google-cloud-aiplatform\n", "google-cloud-aiplatform.whl\n")
.replace("google-cloud-aiplatform ", "google-cloud-aiplatform.whl ")
)
+2 -2
View File
@@ -15,7 +15,7 @@ def download_file(bucket_name: str, blob_name: str, destination_file: str) -> st
remote_file_path = "".join(["gs://", "/".join([bucket_name, blob_name])])
subprocess.check_output(
["gsutil", "cp", remote_file_path, destination_file], encoding="UTF-8"
["gcloud", "storage", "cp", remote_file_path, destination_file], encoding="UTF-8"
)
return destination_file
@@ -27,7 +27,7 @@ def upload_file(
) -> str:
"""Copies a local file to a GCS path"""
subprocess.check_output(
["gsutil", "cp", local_file_path, remote_file_path], encoding="UTF-8"
["gcloud", "storage", "cp", local_file_path, remote_file_path], encoding="UTF-8"
)
return remote_file_path
+3 -3
View File
@@ -7,11 +7,11 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Set up Python
uses: actions/setup-python@v5
uses: actions/setup-python@v6
with:
python-version: '3.x'
python-version: '3.12'
- name: Fetch pull request branch
uses: actions/checkout@v4
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Fetch base main branch
+1 -1
View File
@@ -4,7 +4,7 @@
# 2. To lint specific notebooks:
# docker run -v ${PWD}:/setup/app gcr.io/python-docs-samples-tests/notebook_linter:latest notebooks/1.ipynb notebooks/2.ipynb
FROM python:3.13
FROM python:3.14
WORKDIR setup
+3 -3
View File
@@ -2,9 +2,9 @@ git+https://github.com/tensorflow/docs
ipython
jupyter
nbconvert
black==25.1.0
pyupgrade==3.20.0
isort==6.0.1
black==25.12.0
pyupgrade==3.21.2
isort==7.0.0
flake8==7.3.0
nbqa==1.9.1
@@ -148,7 +148,7 @@ implementation:
# Downloading the model archive from GCS
# TODO: Fix gsutil bugs (requires project ID, has auth issues) and use gsutil instead.
# gsutil cp "$model_archive_uri" "$model_archive_local_path"
# gcloud storage cp "$model_archive_uri" "$model_archive_local_path"
pip install google-cloud-storage
python -c '
import sys
@@ -24,12 +24,12 @@ implementation:
# Checking whether the URI points to a single blob, a directory or a URI pattern
# URI points to a blob when that URI does not end with slash and listing that URI only yields the same URI
if [[ "$uri" != */ ]] && (gsutil ls "$uri" | grep --fixed-strings --line-regexp "$uri"); then
if [[ "$uri" != */ ]] && (gcloud storage ls "$uri" | grep --fixed-strings --line-regexp "$uri"); then
mkdir -p "$(dirname "$output_path")"
gsutil -m cp -r "$uri" "$output_path"
gcloud storage cp --recursive "$uri" "$output_path"
else
mkdir -p "$output_path" # When source path is a directory, gsutil requires the destination to also be a directory
gsutil -m rsync -r "$uri" "$output_path" # gsutil cp has different path handling than Linux cp. It always puts the source directory (name) inside the destination directory. gsutil rsync does not have that problem.
gcloud storage rsync --recursive "$uri" "$output_path" # gsutil cp has different path handling than Linux cp. It always puts the source directory (name) inside the destination directory. gsutil rsync does not have that problem.
fi
- inputValue: GCS path
- outputPath: Data
@@ -1,3 +1,3 @@
torch==2.2.0
torch==2.8.0
torchvision==0.9.1
tensorboard==2.5.0
@@ -110,7 +110,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -192,7 +192,7 @@
},
"outputs": [],
"source": [
"! gsutil cp -r $gcs_output_uri_prefix/model ./model_server/"
"! gcloud storage cp --recursive $gcs_output_uri_prefix/model ./model_server/"
]
},
{
@@ -556,7 +556,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
},
{
@@ -412,7 +412,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
}
],
@@ -77,4 +77,4 @@ echo "After the job is completed successfully, model files will be saved at $JOB
# # Verify the model was exported
# echo "Verify the model was exported:"
# gsutil ls ${JOB_DIR}/
# gcloud storage ls ${JOB_DIR}/
@@ -34,4 +34,4 @@ RUN echo "service_envelope=json\n" "inference_address=http://0.0.0.0:${AIP_H
USER model-server
# run Torchserve HTTP serve to respond to prediction requests
CMD ["echo", "AIP_STORAGE_URI=${AIP_STORAGE_URI}", ";", "gsutil", "cp", "-r", "${AIP_STORAGE_URI}/${MODEL_NAME}.mar", "/home/model-server/model-store/", ";", "ls", "-ltr", "/home/model-server/model-store/", ";", "torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "${MODEL_NAME}=${MODEL_NAME}.mar", "--model-store", "/home/model-server/model-store"]
CMD ["echo", "AIP_STORAGE_URI=${AIP_STORAGE_URI}", ";", "gcloud", "storage", "cp", "--recursive", "${AIP_STORAGE_URI}/${MODEL_NAME}.mar", "/home/model-server/model-store/", ";", "ls", "-ltr", "/home/model-server/model-store/", ";", "torchserve", "--start", "--ts-config=/home/model-server/config.properties", "--models", "${MODEL_NAME}=${MODEL_NAME}.mar", "--model-store", "/home/model-server/model-store"]
@@ -67,4 +67,4 @@ echo "After the job is completed successfully, model files will be saved at $JOB
# # Verify the model was exported
# echo "Verify the model was exported:"
# gsutil ls ${JOB_DIR}/
# gcloud storage ls ${JOB_DIR}/
@@ -478,8 +478,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -498,8 +497,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -582,8 +580,7 @@
"outputs": [],
"source": [
"# Download the sample data into your RAW_DATA_PATH\n",
"! gsutil cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $RAW_DATA_PATH"
]
"! gcloud storage cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $RAW_DATA_PATH" ]
},
{
"cell_type": "code",
@@ -1621,9 +1618,7 @@
"! gcloud scheduler jobs delete $SIMULATOR_SCHEDULER_JOB --quiet\n",
"\n",
"# Delete Cloud Storage objects that were created.\n",
"! gsutil -m rm -r $PIPELINE_ROOT\n",
"! gsutil -m rm -r $TRAINING_ARTIFACTS_DIR"
]
"! gcloud storage rm --recursive $PIPELINE_ROOT\n", "! gcloud storage rm --recursive $TRAINING_ARTIFACTS_DIR" ]
}
],
"metadata": {
@@ -398,6 +398,7 @@
"if not IS_GOOGLE_CLOUD_NOTEBOOK:\n",
" if \"google.colab\" in sys.modules:\n",
" from google.colab import auth as google_auth\n",
"\n",
" google_auth.authenticate_user()\n",
"\n",
" # If you are running this notebook locally, replace the string below with the\n",
@@ -472,7 +473,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -492,7 +493,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -565,7 +566,7 @@
"outputs": [],
"source": [
"# Copy the sample data into your DATA_PATH\n",
"! gsutil cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $DATA_PATH"
"! gcloud storage cp \"gs://cloud-samples-data/vertex-ai/community-content/tf_agents_bandits_movie_recommendation_with_kfp_and_vertex_sdk/u.data\" $DATA_PATH"
]
},
{
@@ -579,11 +580,15 @@
"# Set hyperparameters.\n",
"BATCH_SIZE = 8 # @param {type:\"integer\"} Training and prediction batch size.\n",
"TRAINING_LOOPS = 5 # @param {type:\"integer\"} Number of training iterations.\n",
"STEPS_PER_LOOP = 2 # @param {type:\"integer\"} Number of driver steps per training iteration.\n",
"STEPS_PER_LOOP = (\n",
" 2 # @param {type:\"integer\"} Number of driver steps per training iteration.\n",
")\n",
"\n",
"# Set MovieLens simulation environment parameters.\n",
"RANK_K = 20 # @param {type:\"integer\"} Rank for matrix factorization in the MovieLens environment; also the observation dimension.\n",
"NUM_ACTIONS = 20 # @param {type:\"integer\"} Number of actions (movie items) to choose from.\n",
"NUM_ACTIONS = (\n",
" 20 # @param {type:\"integer\"} Number of actions (movie items) to choose from.\n",
")\n",
"PER_ARM = False # Use the non-per-arm version of the MovieLens environment.\n",
"\n",
"# Set agent parameters.\n",
@@ -621,7 +626,8 @@
"source": [
"# Define RL environment.\n",
"env = movielens_py_environment.MovieLensPyEnvironment(\n",
" DATA_PATH, RANK_K, BATCH_SIZE, num_movies=NUM_ACTIONS, csv_delimiter=\"\\t\")\n",
" DATA_PATH, RANK_K, BATCH_SIZE, num_movies=NUM_ACTIONS, csv_delimiter=\"\\t\"\n",
")\n",
"environment = tf_py_environment.TFPyEnvironment(env)\n",
"\n",
"# Define RL agent/algorithm.\n",
@@ -631,7 +637,8 @@
" tikhonov_weight=TIKHONOV_WEIGHT,\n",
" alpha=AGENT_ALPHA,\n",
" dtype=tf.float32,\n",
" accepts_per_arm_features=PER_ARM)\n",
" accepts_per_arm_features=PER_ARM,\n",
")\n",
"print(\"TimeStep Spec (for each batch):\\n\", agent.time_step_spec, \"\\n\")\n",
"print(\"Action Spec (for each batch):\\n\", agent.action_spec, \"\\n\")\n",
"print(\"Reward Spec (for each batch):\\n\", environment.reward_spec(), \"\\n\")\n",
@@ -639,7 +646,8 @@
"# Define RL metric.\n",
"optimal_reward_fn = functools.partial(\n",
" environment_utilities.compute_optimal_reward_with_movielens_environment,\n",
" environment=environment)\n",
" environment=environment,\n",
")\n",
"regret_metric = tf_bandit_metrics.RegretMetric(optimal_reward_fn)\n",
"metrics = [regret_metric]"
]
@@ -704,35 +712,38 @@
" if training_data_spec_transformation_fn is None:\n",
" data_spec = agent.policy.trajectory_spec\n",
" else:\n",
" data_spec = training_data_spec_transformation_fn(\n",
" agent.policy.trajectory_spec)\n",
" replay_buffer = trainer.get_replay_buffer(data_spec, environment.batch_size,\n",
" steps_per_loop)\n",
" data_spec = training_data_spec_transformation_fn(agent.policy.trajectory_spec)\n",
" replay_buffer = trainer.get_replay_buffer(\n",
" data_spec, environment.batch_size, steps_per_loop\n",
" )\n",
"\n",
" # `step_metric` records the number of individual rounds of bandit interaction;\n",
" # that is, (number of trajectories) * batch_size.\n",
" step_metric = tf_metrics.EnvironmentSteps()\n",
" metrics = [\n",
" tf_metrics.NumberOfEpisodes(),\n",
" tf_metrics.AverageEpisodeLengthMetric(batch_size=environment.batch_size)\n",
" tf_metrics.AverageEpisodeLengthMetric(batch_size=environment.batch_size),\n",
" ]\n",
" if additional_metrics:\n",
" metrics += additional_metrics\n",
"\n",
" if isinstance(environment.reward_spec(), dict):\n",
" metrics += [tf_metrics.AverageReturnMultiMetric(\n",
" reward_spec=environment.reward_spec(),\n",
" batch_size=environment.batch_size)]\n",
" else:\n",
" metrics += [\n",
" tf_metrics.AverageReturnMetric(batch_size=environment.batch_size)]\n",
" tf_metrics.AverageReturnMultiMetric(\n",
" reward_spec=environment.reward_spec(), batch_size=environment.batch_size\n",
" )\n",
" ]\n",
" else:\n",
" metrics += [tf_metrics.AverageReturnMetric(batch_size=environment.batch_size)]\n",
"\n",
" # Store intermediate metric results, indexed by metric names.\n",
" metric_results = defaultdict(list)\n",
"\n",
" if training_data_spec_transformation_fn is not None:\n",
" def add_batch_fn(data): return replay_buffer.add_batch(training_data_spec_transformation_fn(data)) \n",
" \n",
"\n",
" def add_batch_fn(data):\n",
" return replay_buffer.add_batch(training_data_spec_transformation_fn(data))\n",
"\n",
" else:\n",
" add_batch_fn = replay_buffer.add_batch\n",
"\n",
@@ -742,10 +753,12 @@
" env=environment,\n",
" policy=agent.collect_policy,\n",
" num_steps=steps_per_loop * environment.batch_size,\n",
" observers=observers)\n",
" observers=observers,\n",
" )\n",
"\n",
" training_loop = trainer.get_training_loop_fn(\n",
" driver, replay_buffer, agent, steps_per_loop)\n",
" driver, replay_buffer, agent, steps_per_loop\n",
" )\n",
" saver = policy_saver.PolicySaver(agent.policy)\n",
"\n",
" for _ in range(training_loops):\n",
@@ -783,7 +796,8 @@
" environment=environment,\n",
" training_loops=TRAINING_LOOPS,\n",
" steps_per_loop=STEPS_PER_LOOP,\n",
" additional_metrics=metrics)\n",
" additional_metrics=metrics,\n",
")\n",
"\n",
"tf.profiler.experimental.stop()"
]
@@ -1092,11 +1106,15 @@
},
"outputs": [],
"source": [
"RUN_HYPERPARAMETER_TUNING = True # Execute hyperparameter tuning instead of regular training.\n",
"RUN_HYPERPARAMETER_TUNING = (\n",
" True # Execute hyperparameter tuning instead of regular training.\n",
")\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = False # Do not train.\n",
"\n",
"HPTUNING_RESULT_DIR = \"hptuning/\" # @param {type: \"string\"} Directory to store the best hyperparameter(s) in `BUCKET_NAME` and locally (temporarily).\n",
"HPTUNING_RESULT_PATH = os.path.join(HPTUNING_RESULT_DIR, \"result.json\") # @param {type: \"string\"} Path to the file containing the best hyperparameter(s)."
"HPTUNING_RESULT_PATH = os.path.join(\n",
" HPTUNING_RESULT_DIR, \"result.json\"\n",
") # @param {type: \"string\"} Path to the file containing the best hyperparameter(s)."
]
},
{
@@ -1124,7 +1142,7 @@
" image_uri: str,\n",
" args: List[str],\n",
" location: str = \"us-central1\",\n",
" api_endpoint: str = \"us-central1-aiplatform.googleapis.com\"\n",
" api_endpoint: str = \"us-central1-aiplatform.googleapis.com\",\n",
") -> None:\n",
" \"\"\"Creates a hyperparameter tuning job using a custom container.\n",
"\n",
@@ -1197,8 +1215,8 @@
"\n",
" # Create job\n",
" response = client.create_hyperparameter_tuning_job(\n",
" parent=parent,\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job)\n",
" parent=parent, hyperparameter_tuning_job=hyperparameter_tuning_job\n",
" )\n",
" job_id = response.name.split(\"/\")[-1]\n",
" print(\"Job ID:\", job_id)\n",
" print(\"Job config:\", response)\n",
@@ -1242,7 +1260,8 @@
" image_uri=f\"gcr.io/{PROJECT_ID}/{HPTUNING_TRAINING_CONTAINER}:latest\",\n",
" args=args,\n",
" location=REGION,\n",
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\")"
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\",\n",
")"
]
},
{
@@ -1292,7 +1311,8 @@
" name = client.hyperparameter_tuning_job_path(\n",
" project=project,\n",
" location=location,\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job_id)\n",
" hyperparameter_tuning_job=hyperparameter_tuning_job_id,\n",
" )\n",
" response = client.get_hyperparameter_tuning_job(name=name)\n",
" return response"
]
@@ -1313,7 +1333,8 @@
" location=REGION,\n",
" api_endpoint=f\"{REGION}-aiplatform.googleapis.com\")\n",
" if response.state.name == 'JOB_STATE_SUCCEEDED':\n",
" print(\"Job succeeded.\\nJob Time:\", response.update_time - response.create_time)\n",
" print(\"Job succeeded.\n",
"Job Time:\", response.update_time - response.create_time)\n",
" trials = response.trials\n",
" print(\"Trials:\", trials)\n",
" break\n",
@@ -1348,8 +1369,8 @@
"if trials:\n",
" # Dict mapping from metric names to the best metric values seen so far\n",
" best_objective_values = dict.fromkeys(\n",
" [metric.metric_id for metric in trials[0].final_measurement.metrics],\n",
" -np.inf)\n",
" [metric.metric_id for metric in trials[0].final_measurement.metrics], -np.inf\n",
" )\n",
" # Dict mapping from metric names to a list of the best combination(s) of\n",
" # hyperparameter(s). Each combination is a dict mapping from hyperparameter\n",
" # names to their values.\n",
@@ -1358,12 +1379,13 @@
" # `final_measurement` and `parameters` are `RepeatedComposite` objects.\n",
" # Reference the structure above to extract the value of your interest.\n",
" for metric in trial.final_measurement.metrics:\n",
" params = {\n",
" param.parameter_id: param.value for param in trial.parameters}\n",
" params = {param.parameter_id: param.value for param in trial.parameters}\n",
" if metric.value > best_objective_values[metric.metric_id]:\n",
" best_params[metric.metric_id] = [params]\n",
" elif metric.value == best_objective_values[metric.metric_id]:\n",
" best_params[param.parameter_id].append(params) # Handle cases where multiple hyperparameter values lead to the same performance.\n",
" best_params[param.parameter_id].append(\n",
" params\n",
" ) # Handle cases where multiple hyperparameter values lead to the same performance.\n",
" print(\"Best hyperparameter value(s):\")\n",
" for metric, params in best_params.items():\n",
" print(f\"Metric={metric}: {sorted(params)}\")\n",
@@ -1443,7 +1465,9 @@
},
"outputs": [],
"source": [
"PREDICTION_CONTAINER = \"prediction-custom-container\" # @param {type:\"string\"} Name of the container image."
"PREDICTION_CONTAINER = (\n",
" \"prediction-custom-container\" # @param {type:\"string\"} Name of the container image.\n",
")"
]
},
{
@@ -1475,7 +1499,7 @@
" machineType: 'E2_HIGHCPU_8'\"\"\".format(\n",
" PROJECT_ID=PROJECT_ID,\n",
" PREDICTION_CONTAINER=PREDICTION_CONTAINER,\n",
" ARTIFACTS_DIR=ARTIFACTS_DIR\n",
" ARTIFACTS_DIR=ARTIFACTS_DIR,\n",
")\n",
"\n",
"with open(\"cloudbuild.yaml\", \"w\") as fp:\n",
@@ -1592,8 +1616,12 @@
},
"outputs": [],
"source": [
"RUN_HYPERPARAMETER_TUNING = False # Execute regular training instead of hyperparameter tuning.\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = True # @param {type:\"bool\"} Whether to use learned hyperparameters in training."
"RUN_HYPERPARAMETER_TUNING = (\n",
" False # Execute regular training instead of hyperparameter tuning.\n",
")\n",
"TRAIN_WITH_BEST_HYPERPARAMETERS = (\n",
" True # @param {type:\"bool\"} Whether to use learned hyperparameters in training.\n",
")"
]
},
{
@@ -1633,10 +1661,12 @@
"job = aiplatform.CustomContainerTrainingJob(\n",
" display_name=\"train-movielens\",\n",
" container_uri=f\"gcr.io/{PROJECT_ID}/{HPTUNING_TRAINING_CONTAINER}:latest\",\n",
" command=[\"python3\", \"-m\", \"src.training.task\"] + args, # Pass in training arguments, including hyperparameters.\n",
" command=[\"python3\", \"-m\", \"src.training.task\"]\n",
" + args, # Pass in training arguments, including hyperparameters.\n",
" model_serving_container_image_uri=f\"gcr.io/{PROJECT_ID}/{PREDICTION_CONTAINER}:latest\",\n",
" model_serving_container_predict_route=\"/predict\",\n",
" model_serving_container_health_route=\"/health\")\n",
" model_serving_container_health_route=\"/health\",\n",
")\n",
"\n",
"print(\"Training Spec:\", job._managed_model)\n",
"\n",
@@ -1645,7 +1675,8 @@
" replica_count=1,\n",
" machine_type=\"n1-standard-4\",\n",
" accelerator_type=\"ACCELERATOR_TYPE_UNSPECIFIED\",\n",
" accelerator_count=0)"
" accelerator_count=0,\n",
")"
]
},
{
@@ -1784,7 +1815,7 @@
"! gcloud ai models delete $model.name --quiet\n",
"\n",
"# Delete Cloud Storage objects that were created\n",
"! gsutil -m rm -r $ARTIFACTS_DIR"
"! gcloud storage rm --recursive $ARTIFACTS_DIR"
]
}
],
@@ -324,7 +324,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -344,7 +344,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -328,7 +328,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -348,7 +348,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -341,7 +341,7 @@
},
"outputs": [],
"source": [
"! gsutil ls $gcs_output_uri_prefix"
"! gcloud storage ls $gcs_output_uri_prefix"
]
},
{
@@ -361,7 +361,7 @@
},
"outputs": [],
"source": [
"! gsutil rm -rf $gcs_output_uri_prefix"
"! gcloud storage rm --recursive --continue-on-error $gcs_output_uri_prefix"
]
}
],
@@ -2,9 +2,9 @@ dllogger@git+https://github.com/NVIDIA/dllogger@v1.0.0
# Fixing these libraries versions to avoid conflicting or broken packages.
immutabledict==4.2.1
protobuf==3.20.3
protobuf==4.25.8
opencv-python-headless==4.11.0.86
docutils==0.16
urllib3==2.0.7
urllib3==2.6.3
google-cloud-storage==3.0.0
retrying
@@ -38,12 +38,12 @@ pydantic==2.11.4
pydantic_core==2.33.2
python-dateutil==2.9.0.post0
pytz==2025.2
requests==2.32.3
requests==2.32.4
rsa==4.9.1
shapely==2.1.0
six==1.17.0
sniffio==1.3.1
typing-inspection==0.4.0
typing_extensions==4.13.2
urllib3==2.4.0
urllib3==2.6.3
websockets==15.0.1
@@ -66,7 +66,7 @@ mkdir -p "$local_folder"
mkdir -p "$output_folder"
# Download the content from the GCS URI
gsutil -m cp -r "$gcs_dataset_path"/* "$local_folder/"
gcloud storage cp --recursive "$gcs_dataset_path"/* "$local_folder/"
# Process files in the local folder
for file in "$local_folder"/*; do
@@ -122,23 +122,23 @@ cp -r "$output_folder" "$images_folder"/images_2
pushd "$images_folder"/images_2
ls | xargs -P 8 -I {} mogrify -resize 50% {}
popd
gsutil -m cp -r "$images_folder"/images_2/* "$gcs_experiment_path"/data/images_2
gcloud storage cp --recursive "$images_folder"/images_2/* "$gcs_experiment_path"/data/images_2
cp -r "$output_folder" "$images_folder"/images_4
pushd "$images_folder"/images_4
ls | xargs -P 8 -I {} mogrify -resize 25% {}
popd
gsutil -m cp -r "$images_folder"/images_4/* "$gcs_experiment_path"/data/images_4
gcloud storage cp --recursive "$images_folder"/images_4/* "$gcs_experiment_path"/data/images_4
cp -r "$output_folder" "$images_folder"/images_8
pushd "$images_folder"/images_8
ls | xargs -P 8 -I {} mogrify -resize 12.5% {}
popd
gsutil -m cp "$images_folder"/images_8/* "$gcs_experiment_path"/data/images_8
gcloud storage cp "$images_folder"/images_8/* "$gcs_experiment_path"/data/images_8
# Copy images and sparse reconstruction files to gcs experiment folder.
gsutil -m cp "$images_folder"/images/* "$gcs_experiment_path"/data/images
gsutil -m cp -r "$local_folder"/sparse "$gcs_experiment_path"/data
gsutil -m cp "$local_folder"/database.db "$gcs_experiment_path"/data
gcloud storage cp "$images_folder"/images/* "$gcs_experiment_path"/data/images
gcloud storage cp --recursive "$local_folder"/sparse "$gcs_experiment_path"/data
gcloud storage cp "$local_folder"/database.db "$gcs_experiment_path"/data
echo "Processing complete."
@@ -99,14 +99,14 @@ create_dir_if_not_exists "$CHECKPOINTS_PATH"
touch "$local_experiment_path/$exp_folder_name/log_render.txt"
# Copy experiment from GCS bucket to local
gsutil -m cp -r "${args[-gcs_experiment_path]}/data" "$local_experiment_path/$exp_folder_name" || exit 1
gsutil -m cp -r "${args[-gcs_experiment_path]}/checkpoints/${training_job_name}/*" "$CHECKPOINTS_PATH" || exit 1
gcloud storage cp --recursive "${args[-gcs_experiment_path]}/data" "$local_experiment_path/$exp_folder_name" || exit 1
gcloud storage cp --recursive "${args[-gcs_experiment_path]}/checkpoints/${training_job_name}/*" "$CHECKPOINTS_PATH" || exit 1
# Check and copy keyframes file.
if [[ -n ${args[-gcs_keyframes_file]} ]]; then
keyframes_file_basename=$(basename "${args[-gcs_keyframes_file]}")
local_keyframes_file="$local_dataset_path/$keyframes_file_basename"
gsutil cp "${args[-gcs_keyframes_file]}" "$local_keyframes_file" || exit 1
gcloud storage cp "${args[-gcs_keyframes_file]}" "$local_keyframes_file" || exit 1
echo "Local keyframe file: $local_keyframes_file"
launch_rendering "$local_keyframes_file"
else
@@ -114,4 +114,4 @@ else
fi
# Copy rendered data back to GCS.
gsutil -m cp -r "$OUTPUT_RENDER_PATH" "${args[-gcs_experiment_path]}/render/${rendering_job_name}"
gcloud storage cp --recursive "$OUTPUT_RENDER_PATH" "${args[-gcs_experiment_path]}/render/${rendering_job_name}"
@@ -74,7 +74,7 @@ create_dir_if_not_exists "$local_experiment_path"
create_dir_if_not_exists "$local_experiment_path/$scene_folder_name"
# Copy experiment from GCS bucket to local.
gsutil -m cp -r "${gcs_experiment_path}/data" "$local_experiment_path/$scene_folder_name" || exit 1
gcloud storage cp --recursive "${gcs_experiment_path}/data" "$local_experiment_path/$scene_folder_name" || exit 1
echo "GCS Experiment: $gcs_experiment_path"
echo "Gin Config File: $gin_config_file"
@@ -89,6 +89,6 @@ accelerate launch train.py --gin_configs="$gin_config_file" \
--gin_bindings="Config.factor = ${factor}" \
--gin_bindings="Config.max_steps = ${max_training_steps}"
gsutil -m rm -r "${gcs_experiment_path}/checkpoints/${training_job_name}"
gsutil -m cp -r "$local_experiment_path/$scene_folder_name/config.gin" "${gcs_experiment_path}/${training_job_name}_config.gin"
gsutil -m cp -r "$local_experiment_path/$scene_folder_name/checkpoints/*/*" "${gcs_experiment_path}/checkpoints/${training_job_name}"
gcloud storage rm --recursive "${gcs_experiment_path}/checkpoints/${training_job_name}"
gcloud storage cp --recursive "$local_experiment_path/$scene_folder_name/config.gin" "${gcs_experiment_path}/${training_job_name}_config.gin"
gcloud storage cp --recursive "$local_experiment_path/$scene_folder_name/checkpoints/*/*" "${gcs_experiment_path}/checkpoints/${training_job_name}"
@@ -102,10 +102,10 @@ def download_gcs_uri_to_local(
if not os.path.exists(destination_dir):
os.mkdir(destination_dir)
subprocess.check_output([
"gsutil",
"-m",
"gcloud",
"storage",
"cp",
"-r",
"--recursive",
gcs_uri,
destination_dir,
])
+2
View File
@@ -26,6 +26,8 @@
/vertex_endpoints/find_ideal_machine_type/find_ideal_machine_type/find_ideal_machine_type.ipynb @entrpn
/vertex_endpoints/nvidia-triton/nvidia-triton-custom-container-prediction.ipynb @RajeshThallam
/vertex_endpoints/optimized_tensorflow_runtime @vlasenkoalexey
/notebooks/community/alphagenome/cloudai_alphagenome_vai_quickstart.ipynb @dpanigra
/notebooks/community/weathernext/weathernext_2_early_access_program.ipynb @dpanigra
/notebooks/community/ml_ops/stage2/get_started_with_visionapi_and_automl.ipynb @mansari
/notebooks/community/neo4j/graph_paysim.ipynb @benofben @laeg
/notebooks/community/ml_ops/stage1/get_started_with_visionapi_and_vertex_datasets.ipynb @mansari
@@ -0,0 +1,552 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f705f4be70e9"
},
"outputs": [],
"source": [
"# Copyright 2025 Google LLC\n",
"#\n",
"# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9a05a17e4e2f"
},
"source": [
"# Deploy ADK Agents to Vertex AI Agent Engine Using Inline Source\n",
"\n",
"<table><tbody><tr>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fvertex-ai-samples%2Fmain%2Fnotebooks%2Fcommunity%2Fagent_engine%2Fadk_inline_source_deployment.ipynb\">\n",
" <img alt=\"Google Cloud Colab Enterprise logo\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" width=\"32px\"><br> Run in Colab Enterprise\n",
" </a>\n",
" </td>\n",
" <td style=\"text-align: center\">\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/agent_engine/adk_inline_source_deployment.ipynb\">\n",
" <img alt=\"GitHub logo\" src=\"https://github.githubassets.com/assets/GitHub-Mark-ea2971cee799.png\" width=\"32px\"><br> View on GitHub\n",
" </a>\n",
" </td>\n",
"</tr></tbody></table>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e6e9014cd53a"
},
"source": [
"## Overview\n",
"\n",
"This notebook demonstrates how to deploy [Google Agent Development Kit (ADK)](https://google.github.io/adk-docs/) agents to [Vertex AI Agent Engine](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview) using **inline source deployment**.\n",
"\n",
"I built this pattern while developing a multi-agent system with ADK. The approach is also featured in Google's [Agent Starter Pack community showcase](https://github.com/GoogleCloudPlatform/agent-starter-pack#community-showcase).\n",
"\n",
"Inline source deployment sends your agent's Python source code directly to Agent Engine, avoiding serialization issues that can occur with the default pickle-based deployment.\n",
"\n",
"### Why Inline Source Deployment?\n",
"\n",
"| Approach | How it works | Best for |\n",
"|----------|--------------|----------|\n",
"| Default (pickle) | Serializes agent object | Simple agents, quick prototypes |\n",
"| **Inline source** | Sends Python source directly | Complex agents, custom dependencies |\n",
"\n",
"Inline source deployment is useful when:\n",
"- Your agent uses custom classes that don't serialize well\n",
"- You need precise control over the deployment environment\n",
"- You want to avoid pickle compatibility issues across Python versions\n",
"\n",
"### Objectives\n",
"\n",
"- Create an ADK agent with tools\n",
"- Deploy to Agent Engine using inline source\n",
"- Test the deployed agent\n",
"- Clean up resources\n",
"\n",
"### Costs\n",
"\n",
"This tutorial uses billable components of Google Cloud:\n",
"\n",
"* Vertex AI Agent Engine\n",
"* Vertex AI (Gemini API)\n",
"\n",
"Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4d998a5140b2"
},
"source": [
"## Get started"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c832028166e3"
},
"source": [
"### Install required packages"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6b2e64d59bf4"
},
"outputs": [],
"source": [
"# Define requirements once for reuse\n",
"REQUIREMENTS = [\n",
" \"google-cloud-aiplatform[adk,agent_engines]>=1.88.0\",\n",
" \"google-adk>=1.18.0\",\n",
"]\n",
"\n",
"!pip install --upgrade --quiet {\" \".join(REQUIREMENTS)}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2f2e1483e7a8"
},
"source": [
"### Restart runtime\n",
"\n",
"After installing packages, restart the runtime to ensure the new packages are loaded."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d76628df8180"
},
"outputs": [],
"source": [
"import IPython\n",
"\n",
"app = IPython.Application.instance()\n",
"app.kernel.do_shutdown(True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "586675a64f57"
},
"source": [
"### Authenticate your notebook environment (Colab only)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2082bbcb771c"
},
"outputs": [],
"source": [
"import sys\n",
"\n",
"if \"google.colab\" in sys.modules:\n",
" from google.colab import auth\n",
"\n",
" auth.authenticate_user()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "296a2345bc1a"
},
"source": [
"### Set Google Cloud project information"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "abafb08d10b1"
},
"outputs": [],
"source": [
"import os\n",
"\n",
"PROJECT_ID = \"[your-project-id]\" # @param {type: \"string\", placeholder: \"[your-project-id]\", isTemplate: true}\n",
"if not PROJECT_ID or PROJECT_ID == \"[your-project-id]\":\n",
" PROJECT_ID = str(os.environ.get(\"GOOGLE_CLOUD_PROJECT\"))\n",
"\n",
"LOCATION = \"us-central1\" # @param {type:\"string\"}\n",
"\n",
"STAGING_BUCKET = f\"gs://{PROJECT_ID}-agent-engine-staging\" # @param {type:\"string\"}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "c7863dcf6a38"
},
"source": [
"### Create staging bucket\n",
"\n",
"Agent Engine requires a staging bucket to store deployment artifacts."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "17ece3f9fb6e"
},
"outputs": [],
"source": [
"!gsutil mb -l {LOCATION} -p {PROJECT_ID} {STAGING_BUCKET} 2>/dev/null || echo \"Bucket already exists\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1029f595561d"
},
"source": [
"### Initialize Vertex AI"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cac3c1b09203"
},
"outputs": [],
"source": [
"import vertexai\n",
"\n",
"vertexai.init(\n",
" project=PROJECT_ID,\n",
" location=LOCATION,\n",
" staging_bucket=STAGING_BUCKET,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "46fb887b5ea3"
},
"source": [
"## Create an ADK Agent\n",
"\n",
"First, let's create a simple ADK agent with a tool. This agent can answer questions and get the current time."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "9d0dd0808c69"
},
"outputs": [],
"source": [
"import datetime\n",
"from zoneinfo import ZoneInfo, ZoneInfoNotFoundError\n",
"\n",
"from google.adk.agents import LlmAgent\n",
"\n",
"\n",
"def get_current_time(timezone: str = \"UTC\") -> str:\n",
" \"\"\"Get the current time in the specified timezone.\n",
"\n",
" Args:\n",
" timezone: IANA timezone name (e.g., 'America/New_York', 'UTC')\n",
"\n",
" Returns:\n",
" Current time as a formatted string\n",
" \"\"\"\n",
" try:\n",
" tz = ZoneInfo(timezone)\n",
" now = datetime.datetime.now(tz)\n",
" return (\n",
" f\"The current time in {timezone} is {now.strftime('%Y-%m-%d %H:%M:%S %Z')}\"\n",
" )\n",
" except ZoneInfoNotFoundError:\n",
" return f\"Error: Timezone '{timezone}' not found. Please use a valid IANA timezone name (e.g., 'America/New_York', 'UTC').\"\n",
" except Exception as e:\n",
" return f\"An unexpected error occurred while getting time for timezone {timezone}: {e}\"\n",
"\n",
"\n",
"# Create the agent\n",
"agent = LlmAgent(\n",
" name=\"time_agent\",\n",
" model=\"gemini-1.5-flash-001\",\n",
" instruction=\"You are a helpful assistant. Use the get_current_time tool when asked about the time.\",\n",
" tools=[get_current_time],\n",
")\n",
"\n",
"print(f\"Created agent: {agent.name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ad0b9cc230ca"
},
"source": [
"### Test the agent locally\n",
"\n",
"Before deploying, let's test the agent locally to make sure it works."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1d3d71bde31e"
},
"outputs": [],
"source": [
"from google.adk import Runner\n",
"from google.adk.sessions import InMemorySessionService\n",
"from google.genai import types\n",
"\n",
"# Create a local runner\n",
"runner = Runner(\n",
" agent=agent,\n",
" app_name=\"test_app\",\n",
" session_service=InMemorySessionService(),\n",
")\n",
"\n",
"# Create a session and test\n",
"session = runner.session_service.create_session(\n",
" app_name=\"test_app\",\n",
" user_id=\"test_user\",\n",
")\n",
"\n",
"# Send a test message\n",
"response = runner.run(\n",
" user_id=\"test_user\",\n",
" session_id=session.id,\n",
" new_message=types.Content(\n",
" role=\"user\",\n",
" parts=[types.Part(text=\"What time is it in New York?\")],\n",
" ),\n",
")\n",
"\n",
"# Print the response\n",
"for event in response:\n",
" if hasattr(event, \"content\") and event.content:\n",
" for part in event.content.parts:\n",
" if hasattr(part, \"text\") and part.text:\n",
" print(part.text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "06bdc3c10ea6"
},
"source": [
"## Deploy Using Inline Source\n",
"\n",
"Now let's deploy the agent to Agent Engine using inline source deployment.\n",
"\n",
"The key difference from default deployment is using `AgentEngineApp.from_agent()` which packages the source code directly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d641a5e29291"
},
"outputs": [],
"source": [
"from vertexai import agent_engines\n",
"from vertexai.agent_engines import AgentEngineApp\n",
"\n",
"# Create the app from the agent (inline source)\n",
"app = AgentEngineApp.from_agent(\n",
" agent=agent,\n",
" enable_tracing=True,\n",
")\n",
"\n",
"print(\"App created successfully\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cfd8fddbf181"
},
"source": [
"### Deploy to Agent Engine"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ebd6642375a8"
},
"outputs": [],
"source": [
"# Deploy the app\n",
"# This may take a few minutes\n",
"remote_app = agent_engines.create(\n",
" app,\n",
" requirements=REQUIREMENTS,\n",
" display_name=\"time-agent-inline\",\n",
")\n",
"\n",
"print(f\"Deployed agent: {remote_app.resource_name}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "43b739d2f980"
},
"source": [
"## Test the Deployed Agent\n",
"\n",
"Now let's test the deployed agent by sending it a query."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8ec0598a1138"
},
"outputs": [],
"source": [
"# Create a session with the remote agent\n",
"remote_session = remote_app.create_session(user_id=\"test_user\")\n",
"\n",
"print(f\"Created session: {remote_session['id']}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "f85603f0e4fd"
},
"outputs": [],
"source": [
"# Query the agent\n",
"response = remote_app.stream_query(\n",
" user_id=\"test_user\",\n",
" session_id=remote_session[\"id\"],\n",
" message=\"What time is it in Tokyo?\",\n",
")\n",
"\n",
"# Print the response\n",
"for chunk in response:\n",
" if hasattr(chunk, \"content\") and chunk.content:\n",
" for part in chunk.content.parts:\n",
" if hasattr(part, \"text\") and part.text:\n",
" print(part.text, end=\"\")\n",
"\n",
"print() # Newline at end"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "e70d97acf3a3"
},
"source": [
"## Cleaning up\n",
"\n",
"Delete the deployed agent to avoid ongoing charges."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6d2d34a6ba7c"
},
"outputs": [],
"source": [
"# Delete the deployed agent\n",
"delete_agent = False # @param {type:\"boolean\"}\n",
"\n",
"if delete_agent:\n",
" remote_app.delete()\n",
" print(\"Agent deleted successfully\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "35870ecf063a"
},
"outputs": [],
"source": [
"# Delete the staging bucket\n",
"delete_bucket = False # @param {type:\"boolean\"}\n",
"\n",
"if delete_bucket:\n",
" !gsutil rm -r {STAGING_BUCKET}\n",
" print(\"Bucket deleted successfully\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "11f2cc08759a"
},
"source": [
"## Summary\n",
"\n",
"In this notebook, you learned how to:\n",
"\n",
"1. Create an ADK agent with tools\n",
"2. Test the agent locally\n",
"3. Deploy to Agent Engine using inline source with `AgentEngineApp.from_agent()`\n",
"4. Query the deployed agent\n",
"\n",
"### Next steps\n",
"\n",
"- Learn more about [ADK](https://google.github.io/adk-docs/)\n",
"- Explore [Agent Engine documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/agent-engine/overview)\n",
"- Try adding more complex tools and multi-agent orchestration"
]
}
],
"metadata": {
"colab": {
"name": "adk_inline_source_deployment.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
+93
View File
@@ -0,0 +1,93 @@
![AlphaGenome header image](https://raw.githubusercontent.com/google-deepmind/alphagenome/refs/heads/main/docs/source/_static/header.png)
# AlphaGenome
[**Overview**](#overview) | [**Use Cases**](#use-cases) | [**Documentation**](#documentation) | [**Pricing**](#pricing) | [**Quick start**](#quick-start)
## Overview
**Disclaimer:** *Experimental*.
*The AlphaGenome Private Preview is a "Pre-GA Offering" subject to the "Pre-GA
Offerings Terms" in the General Service Terms section of the Google Cloud
[Service Specific Terms](https://cloud.google.com/terms/service-terms). It is
also a “Generative AI Preview Product” as defined in and subject to the
[Additional Terms for Generative AI Preview Products](https://cloud.google.com/trustedtester/aitos?e=48754805&hl=en).
Pre-GA products are available "as is" and might have limited support. For more
information, see the [launch stage](https://cloud.google.com/products?e=48754805#product-launch-stages)
descriptions.*
Access to the AlphaGenome model capabilities requires application and approval.
Users must be added to an allowlist to use the service.
If you are interested in applying to the program, **Request Access** above.
&nbsp;
AlphaGenome is Google DeepMind’s unifying model for deciphering the regulatory
code within DNA sequences.
AlphaGenome offers multimodal predictions, encompassing diverse functional
outputs such as gene expression, splicing patterns, chromatin features, and
contact maps (see diagram below). The model analyzes DNA sequences of up to 1
million base pairs in length and can deliver predictions at single base-pair
resolution for most outputs.
Training data was sourced from large public consortia including
[ENCODE](http://encodeproject.org/), [GTEx](https://www.gtexportal.org/),
[4D Nucleome](https://4dnucleome.org/) and
[FANTOM5](https://fantom.gsc.riken.jp/5/), which experimentally measured these
properties covering important modalities of gene regulation across hundreds of
human and mouse cell types and tissues.
![Diagram showing an overview of the AlphaGenome model architecture and its inputs/outputs](https://www.alphagenomedocs.com/_images/model_overview.png)
## Use Cases
* **Sequence-to-function predictions:** Predict multiple functional tracks (such as gene expression, splicing) from DNA sequences across a wide variety of tissues and cell types.
* **Variant effect scoring:** Assess the impact of genetic variants by comparing predictions for the reference and alternative alleles and summarising the differences between them.
* **Identify functional regions:** Use in silico mutagenesis (ISM) to identify functionally important regions in the DNA sequence.
* **Human and mouse capability:** Generate predictions for both human and mouse genomes.
## Documentation
This API provides access to AlphaGenome, Google DeepMind's unifying model for
deciphering the regulatory code within DNA sequences. AlphaGenome offers
multimodal predictions, encompassing diverse functional outputs including gene
expression, splicing patterns, chromatin features, and contact maps (see diagram
below). The model analyzes up to 1 million base pairs of DNA sequence and can
deliver predictions at single base-pair resolution for most modalities.
AlphaGenome achieves state-of-the-art performance across a range of genomic
prediction benchmarks, including diverse variant effect prediction tasks.
The Google Cloud API for AlphaGenome provides a way for Google Cloud customers
to explore the AlphaGenome API for commercial use cases. This API is in private
preview (Request Access above). Once allowlisted, customers can access the API
directly or use the [colab](cloudai_alphagenome_vai_quickstart.ipynb).
### Acknowledgements
*Avsec, Ž., Latysheva, N., Cheng, J., Novati, G., Taylor, K. R., Ward, T., ... Kohli, P. (2025). AlphaGenome: advancing regulatory variant effect prediction with a unified DNA sequence model. bioRxiv.* [https://doi.org/10.1101/2025.06.25.661532](https://doi.org/10.1101/2025.06.25.661532)
### Contact
If you have any questions on using these models on Google Cloud please contact:
[alphagenome-cloud-external@google.com](mailto:alphagenome-cloud-external@google.com) or join the community [Discourse](https://www.alphagenomecommunity.com/) for more generic questions on AlphaGenome.
### Links
* Read our [paper](https://doi.org/10.1101/2025.06.25.661532)
* Read our [blog post](https://deepmind.google/discover/blog/alphagenome-ai-for-better-understanding-the-genome)
* Join the [community](https://www.alphagenomecommunity.com/)
* Check out the [AlphaGenome 101 Video](https://youtu.be/Xbvloe13nak)
## Pricing
Access to AlphaGenome on Vertex AI is currently restricted.
To utilize these models via this service:
* You must **Request Access** using your Google contact.
* Your application will be reviewed, and if approved, you will be **added to
an allowlist**.
* Only allowlisted users can access the API
* **Pricing information** will be shared directly with users upon approval
and placement on the allowlist.
## Quick start
The quickest way to get started with the AlphaGenome in Google Cloud Platform is to run [our example notebook](cloudai_alphagenome_vai_quickstart.ipynb) in [Google Colab](https://colab.research.google.com/).
File diff suppressed because one or more lines are too long
@@ -37,18 +37,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/Anomaly_detection_in_Cloud_Audit_logs_with_BQML.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/Anomaly_detection_in_Cloud_Audit_logs_with_BQML.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/bigquery_ml/Anomaly_detection_in_Cloud_Audit_logs_with_BQML.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -47,18 +47,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/bq_ml_with_vision_translation_nlp.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/bigquery_ml/bq_ml_with_vision_translation_nlp.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/bigquery_ml/bq_ml_with_vision_translation_nlp.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
@@ -35,18 +35,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery_ml/bqml-online-prediction.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/bigquery_ml/bqml-online-prediction.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/official/bigquery_ml/bqml-online-prediction.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -359,7 +359,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l {REGION} -p {PROJECT_ID} {BUCKET_URI}"
"! gcloud storage buckets create --location={REGION} --project={PROJECT_ID} {BUCKET_URI}"
]
},
{
@@ -1098,7 +1098,7 @@
" ! bq rm -r -f $PROJECT_ID:$BQ_DATASET_NAME\n",
"# delete the Cloud Storage bucket\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil -m rm -r $BUCKET_URI"
" ! gcloud storage rm --recursive $BUCKET_URI"
]
}
],
@@ -38,7 +38,7 @@
" </td>\n",
" <td>\n",
" <a href=\"github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/cohere/cohere_embedding_with_matching_engine.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -449,7 +449,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -470,7 +470,7 @@
"outputs": [],
"source": [
"# this will not return anything if the bucket is empty\n",
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -588,7 +588,7 @@
"# NOTE: Everything in this GCS DIR will be DELETED before uploading the data.\n",
"# A CommandException is expected if no data is present\n",
"\n",
"! gsutil rm -rf {BUCKET_NAME}/*"
"! gcloud storage rm --recursive --continue-on-error {BUCKET_NAME}/*"
]
},
{
@@ -599,7 +599,7 @@
},
"outputs": [],
"source": [
"! gsutil cp cohere_embeddings.json {BUCKET_NAME}/cohere_embeddings.json"
"! gcloud storage cp cohere_embeddings.json {BUCKET_NAME}/cohere_embeddings.json"
]
},
{
@@ -610,7 +610,7 @@
},
"outputs": [],
"source": [
"! gsutil ls {BUCKET_NAME}"
"! gcloud storage ls {BUCKET_NAME}"
]
},
{
@@ -36,18 +36,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/vertex_ai_experiments_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/vertex_ai_experiments_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/vertex_ai_experiments_classification.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -537,7 +537,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI"
]
},
{
@@ -557,7 +557,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gcloud storage ls --all-versions --long $BUCKET_URI"
]
},
{
@@ -627,9 +627,9 @@
},
"outputs": [],
"source": [
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectCreator $BUCKET_URI\n",
"! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectCreator\n",
"\n",
"! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.objectViewer $BUCKET_URI"
"! gcloud storage buckets add-iam-policy-binding $BUCKET_URI --member=serviceAccount:{SERVICE_ACCOUNT} --role=roles/storage.objectViewer"
]
},
{
@@ -1736,7 +1736,7 @@
"delete_bucket = False\n",
"\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}"
]
}
],
@@ -33,18 +33,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/explainable_ai/SDK_Custom_Container_XAI.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/explainable_ai/SDK_Custom_Container_XAI.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/explainable_ai/SDK_Custom_Container_XAI.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -405,7 +405,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
"! gcloud storage buckets create --location=$REGION --project=$PROJECT_ID $BUCKET_URI"
]
},
{
@@ -425,7 +425,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
"! gcloud storage ls --all-versions --long $BUCKET_URI"
]
},
{
@@ -698,7 +698,7 @@
},
"outputs": [],
"source": [
"!gsutil cp app/model.joblib {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/"
"!gcloud storage cp app/model.joblib {BUCKET_URI}/{MODEL_ARTIFACT_DIR}/"
]
},
{
@@ -1516,7 +1516,7 @@
"\n",
"delete_bucket = False\n",
"if delete_bucket or os.getenv(\"IS_TESTING\"):\n",
" ! gsutil rm -rf {BUCKET_URI}"
" ! gcloud storage rm --recursive --continue-on-error {BUCKET_URI}"
]
}
],
@@ -637,20 +637,20 @@
},
"outputs": [],
"source": [
"# Upload CSV data to Cloud Storage by passing gsutil commands to system\n",
"# Upload CSV data to Cloud Storage by passing gcloud storage commands to system\n",
"gcs_url <- paste0(\"gs://\", BUCKET_NAME, \"/\")\n",
"\n",
"command <- paste(\"gsutil mb\", gcs_url)\n",
"command <- paste(\"gcloud storage buckets create\", gcs_url)\n",
"\n",
"system(command)\n",
"\n",
"gcs_data_dir <- paste0(\"gs://\", BUCKET_NAME, \"/data\")\n",
"\n",
"command <- paste(\"gsutil cp data/*_data.csv\", gcs_data_dir)\n",
"command <- paste(\"gcloud storage cp data/*_data.csv\", gcs_data_dir)\n",
"\n",
"system(command)\n",
"\n",
"command <- paste(\"gsutil ls -l\", gcs_data_dir)\n",
"command <- paste(\"gcloud storage ls --long\", gcs_data_dir)\n",
"\n",
"system(command, intern = TRUE)"
]
@@ -679,4 +679,4 @@
},
"nbformat": 4,
"nbformat_minor": 4
}
}
@@ -33,12 +33,12 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/feature_store/gapic-feature-store.ipynb\"\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/official/feature_store/gapic-feature-store.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,20 +34,20 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_vertex_feature_store_serving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" \n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/ml_ops/stage6/get_started_vertex_feature_store_serving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" \n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/ml_ops/stage6/get_started_vertex_feature_store_serving.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td>\n",
@@ -471,8 +471,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_URI"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_URI" ]
},
{
"cell_type": "markdown",
@@ -491,8 +490,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_URI"
]
"! gcloud storage ls --all-versions --long $BUCKET_URI" ]
},
{
"cell_type": "markdown",
@@ -511,8 +509,7 @@
},
"outputs": [],
"source": [
"! gsutil uniformbucketlevelaccess set on {BUCKET_URI}"
]
"! gcloud storage buckets update --uniform-bucket-level-access {BUCKET_URI}" ]
},
{
"cell_type": "markdown",
@@ -1910,9 +1907,7 @@
},
"outputs": [],
"source": [
"! gsutil -m rm -r $BUCKET_URI\n",
"! gsutil rb $BUCKET_URI"
]
"! gcloud storage rm --recursive $BUCKET_URI\n", "! gcloud storage buckets delete $BUCKET_URI" ]
}
],
"metadata": {
@@ -33,18 +33,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/vertex-ai-samples/blob/main/notebooks/community/feature_store/mobile_gaming/mobile_gaming_feature_store.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/mobile_gaming/mobile_gaming_feature_store.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/mobile_gaming/mobile_gaming_feature_store.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -32,12 +32,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb\"\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/feature_store/sdk-feature-store.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -886,11 +884,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1329,7 +1327,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n",
"if len(str(test_items[0]).split(\",\")) == 3:\n",
" _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
@@ -1363,8 +1361,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
@@ -1408,7 +1406,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1692,8 +1690,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1711,9 +1709,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1808,7 +1806,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_export_edge.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_export_edge.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,7 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -436,7 +436,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -474,7 +474,6 @@
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -784,11 +783,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1215,7 +1214,6 @@
" }\n",
" response = clients[\"model\"].export_model(name=name, output_config=output_config)\n",
" print(\"Long running operation:\", response.operation.name)\n",
" result = response.result(timeout=1800)\n",
" metadata = response.operation.metadata\n",
" artifact_uri = str(metadata.value).split(\"\\\\\")[-1][4:-1]\n",
" print(\"Artifact Uri\", artifact_uri)\n",
@@ -1244,9 +1242,9 @@
},
"outputs": [],
"source": [
"! gsutil ls $model_package\n",
"! gcloud storage ls $model_package\n",
"# Download the model artifacts\n",
"! gsutil cp -r $model_package tflite\n",
"! gcloud storage cp --recursive $model_package tflite\n",
"\n",
"tflite_path = \"tflite/model.tflite\""
]
@@ -1305,7 +1303,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"test_item = test_items[0].split(\",\")[0]\n",
"\n",
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
@@ -1449,7 +1447,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -721,12 +721,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1342,8 +1340,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"if len(str(test_item[0]).split(\",\")) == 3:\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(str(test_item[0]).split(\",\")) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label = str(test_item[0]).split(\",\")\n",
@@ -1568,8 +1565,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online_proxy.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_classification_online_proxy.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -721,12 +721,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1342,8 +1340,7 @@
},
"outputs": [],
"source": [
"test_item = !gsutil cat $IMPORT_FILE | head -n1\n",
"if len(str(test_item[0]).split(\",\")) == 3:\n",
"test_item = !gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(str(test_item[0]).split(\",\")) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label = str(test_item[0]).split(\",\")\n",
@@ -1747,8 +1744,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -887,11 +885,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1333,7 +1331,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n",
"cols_1 = str(test_items[0]).split(\",\")\n",
"cols_2 = str(test_items[1]).split(\",\")\n",
"if len(cols_1) == 11:\n",
@@ -1373,8 +1371,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
@@ -1418,7 +1416,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1704,8 +1702,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1723,9 +1721,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1820,7 +1818,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_export_edge.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_export_edge.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,7 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -436,7 +436,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -473,9 +473,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format\n"
]
},
{
@@ -785,11 +783,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1218,7 +1216,6 @@
" }\n",
" response = clients[\"model\"].export_model(name=name, output_config=output_config)\n",
" print(\"Long running operation:\", response.operation.name)\n",
" result = response.result(timeout=1800)\n",
" metadata = response.operation.metadata\n",
" artifact_uri = str(metadata.value).split(\"\\\\\")[-1][4:-1]\n",
" print(\"Artifact Uri\", artifact_uri)\n",
@@ -1247,9 +1244,9 @@
},
"outputs": [],
"source": [
"! gsutil ls $model_package\n",
"! gcloud storage ls $model_package\n",
"# Download the model artifacts\n",
"! gsutil cp -r $model_package tflite\n",
"! gcloud storage cp --recursive $model_package tflite\n",
"\n",
"tflite_path = \"tflite/model.tflite\""
]
@@ -1308,7 +1305,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"test_item = test_items[0].split(\",\")[0]\n",
"\n",
"with tf.io.gfile.GFile(test_item, \"rb\") as f:\n",
@@ -1452,7 +1449,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_object_detection_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -396,9 +396,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -722,11 +720,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1345,7 +1343,7 @@
},
"outputs": [],
"source": [
"test_items = !gsutil cat $IMPORT_FILE | head -n1\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n1\n",
"cols = str(test_items[0]).split(\",\")\n",
"if len(cols) == 11:\n",
" test_item = str(cols[1])\n",
@@ -1574,7 +1572,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -891,12 +889,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1325,8 +1321,7 @@
"source": [
"import json\n",
"\n",
"test_items = !gsutil cat $IMPORT_FILE | head -n2\n",
"test_data_1 = test_items[0].replace(\"'\", '\"')\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n2\n", "test_data_1 = test_items[0].replace(\"'\", '\"')\n",
"test_data_1 = json.loads(test_data_1)\n",
"test_data_2 = test_items[0].replace(\"'\", '\"')\n",
"test_data_2 = json.loads(test_data_2)\n",
@@ -1367,9 +1362,8 @@
"file_1 = test_item_1.split(\"/\")[-1]\n",
"file_2 = test_item_2.split(\"/\")[-1]\n",
"\n",
"! gsutil cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gsutil cp $test_item_2 $BUCKET_NAME/$file_2\n",
"\n",
"! gcloud storage cp $test_item_1 $BUCKET_NAME/$file_1\n",
"! gcloud storage cp $test_item_2 $BUCKET_NAME/$file_2\n", "\n",
"test_item_1 = BUCKET_NAME + \"/\" + file_1\n",
"test_item_2 = BUCKET_NAME + \"/\" + file_2"
]
@@ -1412,8 +1406,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1681,8 +1674,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1699,10 +1691,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1796,8 +1786,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_image_segmentation_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -726,12 +726,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1338,8 +1336,7 @@
"source": [
"import json\n",
"\n",
"test_items = !gsutil cat $IMPORT_FILE | head -n1\n",
"test_data = test_items[0].replace(\"'\", '\"')\n",
"test_items = !gcloud storage cat $IMPORT_FILE | head -n1\n", "test_data = test_items[0].replace(\"'\", '\"')\n",
"test_data = json.loads(test_data)\n",
"try:\n",
" test_item = test_data[\"image_gcs_uri\"]\n",
@@ -1555,8 +1552,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -809,14 +807,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1377,8 +1372,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1646,8 +1640,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1664,10 +1657,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1761,8 +1752,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_binary_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -735,14 +735,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1668,8 +1665,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -810,14 +808,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1363,8 +1358,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1632,8 +1626,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1650,10 +1643,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1747,8 +1738,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -810,14 +808,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1363,8 +1358,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1631,8 +1625,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1649,10 +1642,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/explanation*.csv\n",
"\n",
" ! gsutil cat $folder/explanation*.csv\n",
" break\n",
" ! gcloud storage ls $folder/explanation*.csv\n", "\n",
" ! gcloud storage cat $folder/explanation*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1746,8 +1737,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_export_cloud.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_export_cloud.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -416,8 +416,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -436,8 +435,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -749,14 +747,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1283,15 +1278,12 @@
"source": [
"print(\"Model Package:\", model_package)\n",
"print(\"Contents:\")\n",
"! gsutil ls $model_package\n",
"\n",
"! gcloud storage ls $model_package\n", "\n",
"print(\"\\nTF Saved Model\")\n",
"path = model_package + \"/predict\"\n",
"files = ! gsutil ls $path\n",
"saved_dir = files[1]\n",
"files = ! gcloud storage ls $path\n", "saved_dir = files[1]\n",
"print(saved_dir)\n",
"! gsutil ls $saved_dir"
]
"! gcloud storage ls $saved_dir" ]
},
{
"cell_type": "markdown",
@@ -1312,8 +1304,7 @@
},
"outputs": [],
"source": [
"! gsutil cp -r $model_package ."
]
"! gcloud storage cp --recursive $model_package ." ]
},
{
"cell_type": "markdown",
@@ -1599,8 +1590,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -735,14 +735,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
" raise Exception(\"label column missing\")"
@@ -1645,8 +1642,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_classification_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -397,7 +397,6 @@
"\n",
"import google.cloud.aiplatform_v1beta1 as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -735,13 +734,13 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"! gcloud storage cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"label_column = str(heading).split(\",\")[-1].split(\"'\")[0]\n",
"print(\"Label Column Name\", label_column)\n",
"if label_column is None:\n",
@@ -1820,7 +1819,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_forecasting_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_forecasting_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -779,14 +777,11 @@
},
"outputs": [],
"source": [
"count = ! gsutil cat $IMPORT_FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $IMPORT_FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $IMPORT_FILE | head\n",
"\n",
"heading = ! gsutil cat $IMPORT_FILE | head -n1\n",
"label_column = \"deaths\" # @param {type:\"string\"}\n",
"! gcloud storage cat $IMPORT_FILE | head\n", "\n",
"heading = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "label_column = \"deaths\" # @param {type:\"string\"}\n",
"time_column = \"date\" # @param {type:\"string\"}\n",
"time_series_identifier_column = \"county\" # @param {type:\"string\"}\n",
"print(\"Label Column Name\", label_column)\n",
@@ -1358,8 +1353,7 @@
" f.write(str(INSTANCE_2) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1627,8 +1621,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1645,10 +1638,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.csv\n",
"\n",
" ! gsutil cat $folder/prediction*.csv\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.csv\n", "\n",
" ! gcloud storage cat $folder/prediction*.csv\n", " break\n",
" time.sleep(60)"
]
},
@@ -1742,8 +1733,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_regression_online_bq.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_tabular_regression_online_bq.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -397,7 +397,6 @@
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
@@ -1614,7 +1613,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -881,12 +879,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1314,8 +1310,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"if len(test_items[0]) == 3:\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "if len(test_items[0]) == 3:\n",
" _, test_item_1, test_label_1 = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2 = str(test_items[1]).split(\",\")\n",
"else:\n",
@@ -1371,8 +1366,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1644,8 +1638,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1662,10 +1655,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1759,8 +1750,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -716,12 +716,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1327,7 +1325,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n",
"if len(test_item[0]) == 3:\n",
" _, test_item, test_label = str(test_item[0]).split(\",\")\n",
"else:\n",
@@ -1533,8 +1531,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,7 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
"! gcloud storage buckets create --location $REGION $BUCKET_NAME"
]
},
{
@@ -441,7 +441,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
"! gcloud storage ls --all-versions --long $BUCKET_NAME"
]
},
{
@@ -478,9 +478,7 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
"from google.protobuf import json_format"
]
},
{
@@ -888,11 +886,11 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"count = ! gcloud storage cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
"! gcloud storage cat $FILE | head"
]
},
{
@@ -1377,7 +1375,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
"! gcloud storage cat $gcs_input_uri"
]
},
{
@@ -1652,8 +1650,8 @@
"outputs": [],
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" \"\"\"Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n",
" latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
@@ -1671,9 +1669,9 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n",
" break\n",
" time.sleep(60)"
]
@@ -1768,7 +1766,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
" ! gcloud storage rm --recursive $BUCKET_NAME"
]
}
],
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_entity_extraction_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -723,12 +723,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1544,8 +1542,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -881,12 +879,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1314,8 +1310,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"cols_1 = str(test_items[0]).split(\",\")\n",
"cols_2 = str(test_items[1]).split(\",\")\n",
"test_item_1 = cols_1[0]\n",
@@ -1372,8 +1367,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1645,8 +1639,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1663,10 +1656,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1760,8 +1751,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_multi-label_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -716,12 +716,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1327,8 +1325,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "\n",
"cols = str(test_item[0]).split(\",\")\n",
"test_item = cols[0]\n",
"test_label = cols[1:]\n",
@@ -1533,8 +1530,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -421,8 +421,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -441,8 +440,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -883,12 +881,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1316,8 +1312,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"if len(test_items[0]) == 4:\n",
" _, test_item_1, test_label_1, _ = str(test_items[0]).split(\",\")\n",
" _, test_item_2, test_label_2, _ = str(test_items[1]).split(\",\")\n",
@@ -1375,8 +1370,7 @@
" f.write(json.dumps(data) + \"\\n\")\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1643,8 +1637,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" \"\"\" Get the latest prediction subfolder using the timestamp in the subfolder name\"\"\"\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split(\"/\")[-2]\n",
" if subfolder.startswith(\"prediction-\"):\n",
@@ -1661,10 +1654,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1758,8 +1749,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_text_sentiment_analysis_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -718,12 +718,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1329,8 +1327,7 @@
},
"outputs": [],
"source": [
"test_item = ! gsutil cat $IMPORT_FILE | head -n1\n",
"if len(test_item[0]) == 3:\n",
"test_item = ! gcloud storage cat $IMPORT_FILE | head -n1\n", "if len(test_item[0]) == 3:\n",
" _, test_item, test_label, max = str(test_item[0]).split(\",\")\n",
"else:\n",
" test_item, test_label, max = str(test_item[0]).split(\",\")\n",
@@ -1534,8 +1531,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_action_recognition_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_action_recognition_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -888,12 +886,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1310,8 +1306,7 @@
"import json\n",
"\n",
"import_file = IMPORT_FILES[0]\n",
"test_items = ! gsutil cat $import_file | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $import_file | head -n2\n", "\n",
"cols = str(test_items[0]).split(',')\n",
"test_item_1 = str(cols[0])\n",
"test_label_1 = str(cols[-1])\n",
@@ -1360,8 +1355,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1637,8 +1631,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1655,10 +1648,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1752,8 +1743,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -885,12 +883,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1301,8 +1297,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"if len(test_items[0]) == 5:\n",
" _, test_item_1, test_label_1, _, _ = str(test_items[0]).split(',')\n",
" _, test_item_2, test_label_2, _, _ = str(test_items[1]).split(',')\n",
@@ -1351,8 +1346,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1630,8 +1624,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1648,10 +1641,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
"! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1745,8 +1736,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_object_tracking_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/automl/showcase_automl_video_object_tracking_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -420,8 +420,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -440,8 +439,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -890,12 +888,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1309,8 +1305,7 @@
},
"outputs": [],
"source": [
"test_items = ! gsutil cat $IMPORT_FILE | head -n2\n",
"\n",
"test_items = ! gcloud storage cat $IMPORT_FILE | head -n2\n", "\n",
"cols_1 = test_items[0].split(',')\n",
"cols_2 = test_items[1].split(',')\n",
"if len(cols_1) > 12:\n",
@@ -1365,8 +1360,7 @@
" f.write(json.dumps(data) + '\\n')\n",
"\n",
"print(gcs_input_uri)\n",
"! gsutil cat $gcs_input_uri"
]
"! gcloud storage cat $gcs_input_uri" ]
},
{
"cell_type": "markdown",
@@ -1638,8 +1632,7 @@
"source": [
"def get_latest_predictions(gcs_out_dir):\n",
" ''' Get the latest prediction subfolder using the timestamp in the subfolder name'''\n",
" folders = !gsutil ls $gcs_out_dir\n",
" latest = \"\"\n",
" folders = !gcloud storage ls $gcs_out_dir\n", " latest = \"\"\n",
" for folder in folders:\n",
" subfolder = folder.split('/')[-2]\n",
" if subfolder.startswith('prediction-'):\n",
@@ -1656,10 +1649,8 @@
" raise Exception(\"Batch Job Failed\")\n",
" else:\n",
" folder = get_latest_predictions(predictions)\n",
" ! gsutil ls $folder/prediction*.jsonl\n",
"\n",
" ! gsutil cat $folder/prediction*.jsonl\n",
" break\n",
" ! gcloud storage ls $folder/prediction*.jsonl\n", "\n",
" ! gcloud storage cat $folder/prediction*.jsonl\n", " break\n",
" time.sleep(60)"
]
},
@@ -1753,8 +1744,7 @@
" print(e)\n",
"\n",
"if delete_bucket and 'BUCKET_NAME' in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_ab_testing.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_ab_testing.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -35,18 +35,18 @@
"\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain_example_based_api.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain_example_based_api.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/notebooks/community/gapic/custom/showcase_custom_image_classification_online_explain_example_based_api.ipynb\">\n",
" <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
" <img src=\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\" alt=\"Vertex AI logo\">\n",
" Open in Vertex AI Workbench\n",
" </a>\n",
" </td> \n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_raw.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_raw.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_classification_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_super_resolution_online_post.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_image_super_resolution_online_post.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_batch_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_explain.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_tabular_regression_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_batch.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_container.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_pipeline.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_binary_classification_online_tfserving.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_custom_text_classification_online_exported_ds.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -426,8 +426,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -446,8 +445,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -483,9 +481,6 @@
"import time\n",
"\n",
"from google.cloud.aiplatform import gapic as aip\n",
"from google.protobuf import json_format\n",
"from google.protobuf.json_format import MessageToJson, ParseDict\n",
"from google.protobuf.struct_pb2 import Struct, Value"
]
},
{
@@ -991,12 +986,10 @@
"else:\n",
" FILE = IMPORT_FILE\n",
"\n",
"count = ! gsutil cat $FILE | wc -l\n",
"print(\"Number of Examples\", int(count[0]))\n",
"count = ! gcloud storage cat $FILE | wc -l\n", "print(\"Number of Examples\", int(count[0]))\n",
"\n",
"print(\"First 10 rows\")\n",
"! gsutil cat $FILE | head"
]
"! gcloud storage cat $FILE | head" ]
},
{
"cell_type": "markdown",
@@ -1153,8 +1146,7 @@
"source": [
"jsonl_index = result.exported_files[0]\n",
"\n",
"! gsutil cat $jsonl_index | head"
]
"! gcloud storage cat $jsonl_index | head" ]
},
{
"cell_type": "markdown",
@@ -1286,8 +1278,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_image_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_image_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1119,8 +1117,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1617,8 +1614,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_cifar10.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2062,8 +2058,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_tabular_regression.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_tabular_regression.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location $REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1116,8 +1114,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1626,8 +1623,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_boston.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2085,8 +2081,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {
@@ -34,12 +34,12 @@
"<table align=\"left\">\n",
" <td>\n",
" <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_text_binary_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
" <img src=\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\" alt=\"Colab logo\"> Run in Colab\n",
" </a>\n",
" </td>\n",
" <td>\n",
" <a href=\"https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/master/notebooks/community/gapic/custom/showcase_hyperparmeter_tuning_text_binary_classification.ipynb\">\n",
" <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
" <img width=\"32px\" src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-24.svg\" alt=\"GitHub logo\">\n",
" View on GitHub\n",
" </a>\n",
" </td>\n",
@@ -419,8 +419,7 @@
},
"outputs": [],
"source": [
"! gsutil mb -l $REGION $BUCKET_NAME"
]
"! gcloud storage buckets create --location=$REGION $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -439,8 +438,7 @@
},
"outputs": [],
"source": [
"! gsutil ls -al $BUCKET_NAME"
]
"! gcloud storage ls --all-versions --long $BUCKET_NAME" ]
},
{
"cell_type": "markdown",
@@ -1116,8 +1114,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -1618,8 +1615,7 @@
"! rm -f custom.tar custom.tar.gz\n",
"! tar cvf custom.tar custom\n",
"! gzip custom.tar\n",
"! gsutil cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz"
]
"! gcloud storage cp custom.tar.gz $BUCKET_NAME/trainer_imdb.tar.gz" ]
},
{
"cell_type": "markdown",
@@ -2060,8 +2056,7 @@
" print(e)\n",
"\n",
"if delete_bucket and \"BUCKET_NAME\" in globals():\n",
" ! gsutil rm -r $BUCKET_NAME"
]
" ! gcloud storage rm --recursive $BUCKET_NAME" ]
}
],
"metadata": {

Some files were not shown because too many files have changed in this diff Show More